Python Data Engineer

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.

Work extensively with Databricks and Delta Lake, including Delta tables, MERGE operations, schema evolution, Time Travel, and data optimization.

Develop, schedule, and monitor Apache Airflow DAGs for workflow orchestration and dependency management.

Work with Kubernetes for containerized application deployment, orchestration, monitoring, and troubleshooting.

Develop reusable and optimized Python code following software engineering best practices.

Optimize Spark jobs for performance, scalability, and cost efficiency.

Implement data quality checks, error handling, logging, and monitoring mechanisms.

Use Git for version control, branching, pull requests, code reviews, and collaborative development.

Troubleshoot pipeline failures, investigate data issues, and implement reliable solutions.

Collaborate with cross-functional teams to understand business requirements and deliver efficient data solutions.

Required Technical Skills

Python

Strong programming skills, including data manipulation, exception handling, modular programming, and debugging.

PySpark

Hands-on experience with Spark DataFrames, Spark SQL, joins, transformations, partitioning, and performance tuning.

Databricks

Strong understanding of Databricks architecture, notebooks, workflows, compute management, and job execution.

Delta Lake: Experience with Delta tables, ACID transactions, MERGE/UPSERT, schema enforcement, schema evolution, VACUUM, and Time Travel.

Apache Airflow: Experience developing DAGs, managing task dependencies, configuring retries, scheduling workflows, and troubleshooting failures.

Kubernetes: Practical knowledge of pods, deployments, services, ConfigMaps, Secrets, resource management, and troubleshooting containerized workloads.

Git

Proficiency in branching strategies, pull requests, merging, and resolving conflicts.

SQL

Strong SQL skills, including complex joins, CTEs, window functions, and query optimization.

Good to Have

Experience with Docker and containerization best practices.

Experience with Databricks Unity Catalog and data governance.

Knowledge of CI/CD pipelines and automated deployments.

Experience deploying and managing Airflow workloads on Kubernetes.

Familiarity with Kubernetes monitoring, autoscaling, and resource optimization.

Experience with cloud platforms such as Azure, AWS, or GCP.

Understanding of data lakehouse architecture and data modeling.

Experience with batch and incremental data processing.

Familiarity with data quality frameworks and automated testing.

Preferred Candidate Profile

Strong analytical and problem-solving skills.

Ability to independently design, develop, deploy, and troubleshoot data pipelines.

Good understanding of distributed data processing and container orchestration.

Experience working in an Agile development environment.

Strong communication and collaboration skills.