Databricks Data Engineer - LTM Limited

We are looking for a highly motivated Data Engineer with strong expertise in Databricks and modern data engineering practices

The ideal candidate should have handson experience in building scalable data pipelines notebooks implementing CICD processes

and working with Delta Lake technologies The candidate must possess excellent Python programming skills

a solid understanding of Unity Catalog and the ability to collaborate effectively within large cross functional teams

• Design develop and maintain scalable ETLELT pipelines Notebooks using Databricks
• Build and optimize data processing solutions using PySpark and Python
• Implement and manage data governance security and access controls using Unity Catalog
• Work with Delta Lake features including Delta Sharing Time Travel Change Data Feed CDF and data optimization techniques
• Develop and maintain CICD pipelines for Databricks deployments using industry standard Dev Ops practices
• Perform performance tuning and optimization of Databricks workloads
• Collaborate with architects business analysts and stakeholders to understand data requirements and deliver solutions
• Support production deployments troubleshooting monitoring and performance improvements
• Follow coding standards documentation and best practices for software development lifecycle SDLC

• Databricks Strong
• Strong understanding of Databricks architecture and ecosystem
• Handson experience with
• Delta Lake
• Delta Sharing
• Unity Catalog
• Structured Streaming
• Workflow orchestration
• Performance tuning and optimization
• Data governance and access management
• Experience with Medallion Architecture Bronze Silver Gold layers
• Programming
• Strong handson coding experience in Python
• Experience with PySpark for large scale data processing
• Good understanding of SQL and query optimization
• Dev Ops CICD
• Experience implementing CICD processes for Databricks workloads
• Familiarity with Gitbased source control and branching strategies
• Experience with deployment automation tools and Dev Ops pipelines
• Data Engineering Concepts
• ETLELT design patterns
• Data modeling and data warehousing concepts
• Data quality monitoring and observability frameworks

• AWS Cloud
• AWS Glue
• S3
• IAM
• Lambda
• Azure Cloud
• Azure Data Factory ADF
• Azure Storage
• Azure Databricks
• Key Vault

• Excellent verbal and written communication skills
• Strong problem solving and analytical abilities
• Ability to work effectively in a large global team environment
• Strong stakeholder management and collaboration skills
• Self driven proactive and able to work independently with minimal supervision