Data Engineer – AWS, PySpark & Airflow

🚀 Job Title : Data Engineer – AWS, PySpark & Airflow

Experience

5+ Years

Industry

Health Insurance / BFSI

Location

Sector 43, Gurugram, Haryana

Work Mode

Work From Office (5 Days a Week)

Employment Type

Full-Time

Role Overview

We are looking for an experienced Data Engineer with strong hands-on expertise in AWS Data Engineering, Data Lake Architecture, Python, SQL, PySpark, and Apache Airflow.

The candidate will be responsible for designing, developing, and maintaining scalable, reliable, and high-performance data pipelines using AWS cloud technologies. The role involves building end-to-end ETL / ELT workflows, implementing data transformation and processing solutions, optimizing pipeline performance, and ensuring the reliability of production data systems.

Mandatory Skills

AWS Data Engineering | Data Lake Architecture | Amazon S3 | AWS Glue | Amazon EMR | Athena | Redshift | Python | SQL | PySpark | Apache Spark | Apache Airflow | ETL/ELT | PostgreSQL | Data Modelling | Pipeline Optimization | Production Support | Root Cause Analysis

Key Responsibilities

Design, develop, and maintain end-to-end data pipelines for data ingestion, staging, transformation, curation, and consumption.

Build and manage AWS-based data lake solutions using Amazon S3, AWS Glue, Amazon EMR, Athena, and Redshift.

Develop, maintain, and optimize ETL / ELT pipelines using Python, SQL, Apache Spark, and PySpark.

Implement and manage workflow orchestration using Apache Airflow, including scheduling, dependency management, retries, monitoring, and failure handling.

Integrate data from multiple structured and semi-structured sources.

Design and maintain data models, dimensional models, and analytical data marts.

Work with relational databases, including PostgreSQL and Amazon RDS.

Optimize Spark jobs, SQL queries, data transformations, and cloud resource utilization.

Implement data quality checks, validation rules, and error-handling mechanisms.

Monitor production data pipelines, troubleshoot failures, and perform root cause analysis (RCA).

Ensure pipeline reliability, scalability, performance, and compliance with service-level agreements (SLAs).

Follow data security, access management, governance, and sensitive-data protection best practices.

Collaborate with business, analytics, and application teams to deliver effective data solutions.

Maintain technical documentation and follow established data engineering standards.

Required Technical Skills

AWS Data Engineering

Strong hands-on experience with AWS-based data engineering and data lake architecture.

AWS Services

Amazon S3, AWS Glue, Glue Data Catalog, Amazon EMR, Athena, Redshift, AWS Lambda, and IAM.

Programming

Strong proficiency in Python and SQL.

Big Data Processing

Hands-on experience with Apache Spark and PySpark.

Workflow Orchestration

Experience developing and managing workflows using Apache Airflow.

ETL/ELT Development

Experience building scalable data ingestion, transformation, and processing pipelines.

Database Technologies

Good understanding of relational databases and PostgreSQL concepts.

Data Modelling

Experience with data modelling, data transformation, and analytical data structures.

Production Support

Ability to monitor pipelines, troubleshoot failures, perform RCA, and resolve production issues.

Performance Optimization

Experience optimizing SQL queries, Spark jobs, and data processing workflows.

Good-to-Have Skills

Experience with AWS Database Migration Service (DMS) and Amazon RDS.

Familiarity with dbt for data transformation and modelling.

Knowledge of Delta Lake, Apache Iceberg, or Apache Hudi.

Exposure to Apache Kafka and real-time data streaming.

Familiarity with GraphQL, Hasura, or similar data-serving technologies.

Understanding of data governance and data quality frameworks.

Experience in the BFSI, healthcare, or health insurance domain.

Candidate Requirements

5+ years of relevant experience in Data Engineering.

Proven hands-on experience building and maintaining production-grade data pipelines on AWS.

Strong understanding of data lake architecture and large-scale data processing.

Practical experience with Apache Airflow and Spark / PySpark-based transformations.

Strong programming, analytical, debugging, and problem-solving skills.

Experience handling production incidents, pipeline failures, and performance bottlenecks.

Ability to take ownership of deliverables and work effectively with cross-functional teams.

Good communication, collaboration, and technical documentation skills.

Willingness to work from the Gurugram office five days a week.

Ideal Candidate

The ideal candidate is a hands-on Data Engineer who can independently build, optimize, and maintain AWS-based data pipelines, work with large datasets using PySpark, orchestrate workflows through Airflow, and resolve production issues in a business-critical environment.