Senior Data Engineer / Databricks Expert

Stermedia.ai is looking for an experienced Senior Data Engineer with strong Databricks expertise to join a data platform team building a global analytics environment for a leading pharmaceutical organization.

The platform integrates data from more than 80 facilities worldwide, including laboratories, manufacturing systems, logistics platforms, and operational environments. Its goal is to transform distributed datasets into reliable, analytics-ready data that supports reporting, operational monitoring, and data-driven decision-making.

You’ll contribute to the full delivery lifecycle — from direct client discussions and business requirements analysis to solution design, implementation, and production deployment. Working alongside international data engineering, analytics, and machine learning teams, you’ll focus on building scalable data pipelines and transformation layers using Databricks, Apache Spark, Python, SQL, and Delta Lake.

About the project

The project focuses on building a scalable data platform for operational data from pharmaceutical laboratories, production systems, and global supply chains.

Data from multiple regions is consolidated into a centralized analytics environment. The role focuses on using Databricks and a lakehouse architecture to ingest, process, and organize this data into reliable datasets for downstream analytics.

As a Databricks expert, you’ll design and maintain data pipelines, optimize distributed processing, and support data quality, governance, and production reliability. DBT is a complementary skill for SQL-based transformations and analytics engineering workflows.

Requirements

Design, develop, and maintain data pipelines in Databricks using Python, PySpark, and SQL

Build and maintain Delta Lake tables and transformation layers following bronze, silver, and gold architecture

Integrate data from multiple source systems into consistent, analytics-ready datasets

Implement incremental processing, change data capture, and schema evolution where required

Develop and orchestrate production workflows, including dependencies, scheduling, retries, and monitoring

Optimize Spark workloads, SQL queries, and compute usage to improve performance and manage costs

Implement automated data quality checks, validation, and pipeline tests

Support data access management, governance, and lineage using Unity Catalog

Maintain reusable code, technical documentation, and Git-based development workflows

Collaborate with business stakeholders, data engineers, analysts, and machine learning specialists to translate requirements into reliable data solutions

Required skills

Strong commercial experience with Databricks, including developing and operating production data pipelines

Advanced knowledge of Python, PySpark, and SQL

Hands-on experience with Apache Spark, distributed data processing, and performance troubleshooting

Practical experience with Delta Lake, including incremental loads, merge operations, and schema management

Strong understanding of lakehouse architecture, ETL/ELT patterns, and data modeling

Experience orchestrating and monitoring workflows in Databricks

Practical knowledge of Unity Catalog, including permissions, data organization, and lineage

Experience optimizing pipeline performance and compute resource usage

Familiarity with cloud storage and services in at least one major cloud environment: Azure, AWS, or GCP

Experience with Git, code reviews, automated testing, and CI/CD workflows

Strong analytical and communication skills, with the ability to work directly with international stakeholders

Good command of English

Nice to have

Experience with DBT, including models, tests, macros, and integration with Databricks

Experience with Structured Streaming and Databricks Auto Loader

Familiarity with Terraform and infrastructure as code

Experience supporting datasets and workflows used by machine learning teams

Experience building global analytics platforms or integrating data from multiple facilities

Familiarity with pharmaceutical, manufacturing, laboratory, or supply chain data

Databricks certifications in data engineering

Technology focus

Data platform

Databricks

Processing

Apache Spark, PySpark

Languages

Python, SQL

Storage and table format

Delta Lake, cloud object storage

Architecture

Lakehouse, bronze / silver / gold layers

Governance

Unity Catalog

Orchestration

Databricks jobs and workflows

Development and delivery

Git, CI/CD, automated testing

Complementary analytics tooling: DBT

We offer you

Opportunities to work with modern data engineering and machine learning technologies

An annual self-development budget

The opportunity to contribute to a variety of interesting projects

Internal workshops and knowledge-sharing sessions

Support for personal branding through articles, conference talks, and leading internal workshops

Flexible working hours

The possibility of remote work

A chillout room, free beverages, and team and company events

A friendly atmosphere

MultiSport

LuxMed

Salary

20,000–30,000 PLN + VAT (B2B)

Highlights

Originally posted on Himalayas