DataBricks Data Engineer

Overview

Prodapt is the largest specialized player in the Connectedness industry. As an AI-first strategic technology partner, Prodapt provides consulting, business reengineering, and managed services for the largest telecom and tech enterprises building networks and digital experiences of tomorrow. A Service Now-invested company, Prodapt has been recognized by Gartner as a Large, Telecom-Native, Regional IT Service Provider. A Great Place To Work Certified company, Prodapt employs over 5,000 technology and domain experts across the Americas, Europe, India, Africa, & Japan.

Prodapt is part of the 130-year-old business conglomerate The Jhaver Group, which employs over 32,000 people across 80+ locations globally.

We are seeking an experienced Azure Databricks Engineer to join our team in Dallas, TX. The ideal candidate has strong hands-on experience in designing and building scalable data engineering solutions using Azure Databricks, PySpark, Python, SQL, and Azure Data Services including leading the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks.

• Design, develop, and implement scalable data pipelines using Azure Databricks.
• Build and optimize ETL/ELT pipelines using PySpark, Python, and SQL.
• Develop data solutions using the Medallion Architecture (Bronze, Silver, and Gold layers).
• Work with Delta Lake, Delta tables, and advanced Databricks optimization techniques.
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault.
• Develop and manage Databricks Workflows, Jobs, and Notebooks.
• Implement data quality, monitoring, error handling, and performance optimization.
• Collaborate with Data Architects, Data Engineers, Data Scientists, and business stakeholders.
• Establish best practices for CI/CD, Git/version control, code reviews, and automated deployments.
• Lead technical discussions and provide guidance to junior engineers.
• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze, Silver, Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.

• 10+ years of overall Data Engineering experience, including designing and implementing ETL/ELT pipelines with Azure Data Factory and other Azure services.
• Bachelor’s degree in Computer Science, Engineering, or a related field; OR equivalent combination of education and relevant experience.
• Strong hands-on experience with Azure Databricks; 4+ years of experience across Azure services and Databricks (ADLS, ADF, Azure Dev Ops, etc.).
• 7+ years of Python development experience, with the ability to design and build reusable libraries.
• 4+ years of experience with Snowflake or SQL (No-SQL experience is a plus).
• Expert-level knowledge of PySpark and Spark SQL.
• Strong programming experience in Python and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Strong experience with Azure Data Lake Storage (ADLS Gen2).
• Experience with Azure Data Factory (ADF) and other Azure data services.
• Strong understanding of ETL/ELT, data modeling, and large-scale data pipelines.
• Experience with performance tuning and optimization in Databricks/Spark.
• Experience with Git, CI/CD, and Dev Ops practices.

• Databricks certifications.
• Experience with Unity Catalog and Databricks governance/security.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of Azure Dev Ops.
• Experience designing APIs and integrating with React JS within a cloud platform