
Applications Development Senior Programmer Analyst
Job Description
The Applications Development Senior Programmer Analyst - Databricks is a senior-level technical role responsible for the design, development, optimization, and governance of enterprise-scale data pipelines, Lakehouse solutions, and analytics/reporting platforms. Operating at the intersection of Big Data engineering and compliance analytics, this role will architect robust ETL/ELT data pipelines using Databricks, Apache Spark, Iceberg, Delta Lake, and PySpark/SQL to power high-throughput, low-latency reporting and regulatory solutions.
The ideal candidate will have 8+ years of professional experience in distributed data systems, demonstrating deep hands-on expertise in Lakehouse architecture (Medallion pattern), data modeling, query optimization, and enterprise data governance. You will work closely with cross-functional global teams—including Data Architects, Business Analysts, Compliance Officers, and Operations teams—to deliver mission-critical reporting solutions that adhere to stringent regulatory, performance, and security standards.
Job Background / Context
In an evolving regulatory landscape, managing high-volume data feeds, complex data transformations, and timely compliance/regulatory reporting is critical. The Applications Development team is responsible for engineering the next-generation data and reporting ecosystem supporting key regulatory functions, including Anti-Money Laundering (AML), Know Your Customer (KYC), Customer Due Diligence (CDD), and Enhanced Due Diligence (EDD) .
This position sits within the core Applications Development organization and focuses on modernizing and scaling data platforms by leveraging Databricks Lakehouse architecture , federated querying capabilities, and advanced analytics pipelines. The role requires a seasoned engineer capable of delivering reliable data pipelines, optimizing complex query execution paths, driving legacy report modernization, and enforcing data quality and security across multi-tiered data repositories.
Key Responsibilities
Requirements & Strategy
- Collaborate closely with Business Analysts, KYC Operations leads, Compliance Officers, and Data Architects to translate complex business and regulatory requirements into scalable reporting solutions and robust data pipelines.
- Participate in technical design sessions, architectural reviews, and data modeling discussions to ensure alignment with enterprise Lakehouse standards and compliance frameworks.
Databricks & Data Pipeline Engineering
- Design, build, and maintain high-performance batch and real-time data pipelines using Databricks, Apache Spark (Core, SQL, DataFrames), PySpark, Iceberg and Delta Lake (Bronze, Silver, and Gold Medallion layers).
- Implement enterprise governance, data lineage, and fine-grained access control policies utilizing Databricks Unity Catalog .
- Develop automated, resilient workflow orchestrations using Databricks Workflows, Delta Live Tables (DLT) , and enterprise scheduling tools.
- Ingest and integrate data across diverse transactional engines, relational databases, NoSQL datastores, and federated query layers.
Maintenance & Performance Tuning
- Troubleshoot report failures, resolve data discrepancies, and optimize report execution times and database access paths for peak operational efficiency.
- Profile and fine-tune Spark jobs, SQL queries, and cluster resource allocations using optimization techniques such as Z-Ordering, Liquid Clustering, partition pruning, and broadcast join optimizations.
- Conduct root-cause analysis (RCA) on production incidents, data pipeline anomalies, and performance bottlenecks to ensure high availability and data consistency.
Mentorship & Support
- Provide technical guidance, code/model reviews, and mentorship to junior developers to foster engineering excellence and adherence to coding standards.
- Deliver training and tier-3 support to end-users and analytical stakeholders on reporting tools, semantic models, and data exploration layers.
- Champion modern DataOps practices, including CI/CD automated deployments, comprehensive unit testing, and structured release management.
Knowledge & Experience
Core Experience & Skills
- 6+ years of professional experience in Data Engineering, Big Data Application Development, or Distributed Systems Analytics.
- Databricks & Spark: 6+ years of hands-on experience designing and operating enterprise solutions on Databricks utilizing Apache Spark, PySpark, Spark SQL, and Delta Lake .
- Data Modeling & SQL: Expert-level SQL skills with strong proficiency in data warehousing concepts, dimensional modeling (Star and Snowflake schemas), and data transformation design.
- Communication & Analytical Skills: Strong communication, analytical, and problem-solving skills with the ability to articulate technical solutions clearly to both technical and non-technical stakeholders.
- Data Governance & Security: Practical experience with data governance, RBAC, access policies, and audit logging using Unity Catalog .
- Airflow Orchestration tool
- DevOps & Engineering Practices: Solid understanding of version control ( Git ), branching workflows, automated testing, and CI/CD pipelines.
Desirable / Good to Have
- Starburst / Presto: Experience with Starburst Presto for federated query execution, distributed data integration, and access governance.
- NoSQL / Alternate Data Sources: Exposure to MongoDB or distributed big data repositories.
- Domain Knowledge: Functional understanding of AML / KYC, Customer Due Diligence (CDD), Enhanced Due Diligence (EDD) , and regulatory compliance metrics and workflows.
- Automation & Scripting: Familiarity with workload automation and scheduling tools ( Autosys, Cron ), Unix/Linux shell scripting, or Cognos Java SDK.
- Cloud Infrastructure: Exposure to major cloud platforms (AWS / Azure / GCP) and cloud data storage services (ADLS Gen2, AWS S3).
- Inhouse OnPrem to data bricks migrati
काम पाने के लिए कभी पैसे मत दो। Fee माँगे तो scam है। दस निशान →