Senior Databricks Engineer

Job Description – Senior Databricks Engineer

Job Title:

Senior Databricks Engineer

Experience:

8–10 Years

Location:

Remote

Employment Type:

Full-Time

Job Summary
We are seeking a highly skilled

Senior Databricks Engineer

with 8–10 years of experience in Data Engineering, Big Data technologies, and cloud-based data platforms. The ideal candidate will have strong hands-on expertise in

Azure Databricks ,

Apache Spark ,

Python , and modern data architecture. The role involves designing, developing, and optimizing scalable data pipelines and analytics solutions that support business intelligence and advanced analytics initiatives.

Key Responsibilities
Design, develop, and maintain scalable data pipelines using Azure Databricks and Apache Spark.
Build and optimize ETL/ELT processes for large-scale structured and unstructured data.
Develop data ingestion frameworks from various sources including databases, APIs, files, and streaming platforms.
Implement data transformation, cleansing, and enrichment processes.
Collaborate with business stakeholders, data architects, and analytics teams to understand requirements and deliver solutions.
Optimize Databricks workloads for performance, scalability, and cost efficiency.
Develop and maintain Delta Lake architectures and data models.
Ensure data quality, governance, security, and compliance standards are met.
Troubleshoot production issues and provide ongoing support for data platforms.
Participate in architecture discussions and recommend best practices for cloud-based data solutions.
Mentor junior engineers and contribute to technical knowledge sharing.
Required Skills
8–10 years of experience in Data Engineering and Big Data technologies.
Strong expertise in

Azure Databricks .
Hands-on experience with

Apache Spark (PySpark/Spark SQL) .
Proficiency in

Python

and

SQL .

Experience with

Azure Data Factory (ADF) .
Strong understanding of

Delta Lake , Lakehouse architecture, and data modeling.
Experience working with cloud platforms, preferably

Microsoft Azure .
Knowledge of CI/CD processes and Dev Ops practices.

Experience with data warehousing concepts and dimensional modeling.
Strong analytical, troubleshooting, and problem-solving skills.
Preferred Skills

Experience with Azure Synapse Analytics.
Knowledge of streaming technologies such as Kafka or Event Hubs.

Experience with Terraform, Git, and Azure Dev Ops.
Understanding of Data Governance and Data Quality frameworks.
Exposure to AI/ML data pipelines is an added advantage.
Educational Qualification
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
Why Join Us?
Work on large-scale cloud and data transformation projects.
Opportunity to collaborate with global teams and modern data technologies.
Continuous learning and career growth opportunities.
Competitive compensation and benefits package.