Senior Data Engineer (AWS & Confluent Data/AI Projects) | Remote

Work Set-up

Remote

Schedule

10am-6pm SGT

Responsibilities

Architect and Design Data Solutions: Lead the design and architecture of scalable,

secure, and efficient data pipelines for both batch and real-time data processing on

AWS. This includes data ingestion, transformation, storage, and consumption layers.

Confluent Kafka Expertise

Design, implement, and optimize highly performant and

reliable data streaming solutions using Confluent Platform (Kafka, ksqlDB, Kafka

Connect, Schema Registry). Ensure efficient data flow for real-time analytics and AI

applications.

AWS Cloud Native Development

Develop and deploy data solutions leveraging a wide

range of AWS services, including but not limited to

Data Storage

S3 (Data Lake), RDS, DynamoDB, Redshift, Lake Formation.

Data Processing

Glue, EMR (Spark), Lambda, Kinesis, MSK (for Kafka integration).

Orchestration

AWS Step Functions, Airflow (on EC2 or MWAA)

Analytics & ML

Athena, QuickSight, SageMaker (for MLOps integration).

Required Skills and Qualifications

Bachelor's or Master's degree in Computer Science, Software Engineering, or a related

quantitative field.

3 to 5 years of experience in data engineering, with a significant focus on cloud-based

solutions.

Strong expertise in AWS data services (S3, Glue, EMR, Redshift, Kinesis, Lambda, etc.).

Extensive hands-on experience with Confluent Platform/Apache Kafka for building

real-time data streaming applications.

Proficiency in programming languages such as Python, PySpark, Scala, or Java.

Expertise in SQL and experience with various database systems (relational and NoSQL).

Solid understanding of data warehousing, data lakes, and data modeling concepts (star

schema, snowflake schema, etc.).

Experience with CI/CD pipelines and DevOps practices (Git, Terraform, Jenkins, Azure

DevOps, or similar).

AWS Certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified

Preferred Qualifications (Nice to Have)

Solutions Architect - Associate/Professional).

Experience with other streaming technologies (e.g., Flink).

Knowledge of containerization technologies (Docker, Kubernetes).

Familiarity with Data Mesh or Data Fabric concepts.

Experience with data visualization tools (e.g., Tableau, Power BI, QuickSight).

Understanding of MLOps principles and tools.

Candidate must have a working laptop

Originally posted on Himalayas