Databricks advanced developer ; tuning expertise

Databricks advanced developer with (performance tuning expertise)

Senior Databricks Developer/Data Engineer should be an expert responsible for leading the design, development, optimization, and deployment of large-scale, robust data solutions and data pipelines within the Databricks Lakehouse platform. This role requires technical leadership, advanced performance tuning expertise, and a strong understanding of cloud architecture and Dev Ops practices.

• Design & Architecture: Lead the end-to-end design and implementation of scalable data pipelines and a medallion architecture (bronze, silver, gold layers) for structured and unstructured data.
• ETL/ELT Development: Architect and implement complex ETL/ELT processes to process and transform large-scale datasets from diverse sources using Databricks notebooks, Delta Lake, and Apache Spark (PySpark, Scala, or SQL).
• Performance Optimization: Optimize Spark jobs for performance and cost-efficiency using advanced techniques such as partitioning, caching, cluster configuration tuning, and troubleshooting bottlenecks.
• Data Governance & Security: Implement data security strategies, compliance standards, and governance frameworks like Unity Catalog, including managing access controls, encryption, and data lineage.
• Dev Ops & Automation: Champion code quality and implement Dev Ops (CI/CD) pipelines using tools like Azure Dev Ops, Git Hub Actions, or Jenkins to automate deployment and management of data assets and infrastructure.
• Troubleshooting & Monitoring: Develop and implement robust monitoring, logging, and alerting systems for production-grade pipelines to proactively identify and resolve issues.

• Extensive
  
  Experience:
  
  10+ years of experience in data engineering, with 5+ years specifically in Databricks development and leadership roles.
• Programming Mastery:

Should have at least 5 years of experience in optimizing Spark jobs for performance and cost-efficiency using advanced techniques such as partitioning, caching, cluster configuration tuning, and troubleshooting bottlenecks.

Advanced deep knowledge in Scala, Python, and SQL for optimizing code and building high performing data pipelines on Databricks.

Minimum 5 years of experience in Scala, Spark and Databricks programming.

Must have 5-10 years of experience implementing complex ETL/ELT processes to process and transform large-scale datasets from diverse sources using Databricks notebooks, Delta Lake, and Apache Spark (PySpark, Scala and SQL).

Must be able to design and develop CI/CD pipelines that can push the code to different development and production environments, preferably to Azure using Git actions.

Must have 5 years of experience in logging, monitoring and alerting systems.

• Big Data Technologies: Deep knowledge of the Apache Spark ecosystem, Delta Lake, and Lakehouse architecture.

Should have 2-3 years experience converting requirements to architecture and design in Spark and Databricks.

5 years of experience in debugging and identifying performance in Databricks.

• Cloud Platform Expertise: Hands-on experience with Azure and their relevant data services (e.g., Azure Data Lake Storage, Azure Data Factory).
• Certifications: Professional certifications in Databricks (e.g., Databricks Certified Data Engineer Professional) or cloud data platforms are highly beneficial.

We strive to provide flexibility wherever possible. Based on this role’s business requirements, this is a hybrid position at Fort Worth TX (3 days a week), open to qualified applicants in the United States. Regardless of your working arrangement, we are here to support a healthy work-life balance through our various wellbeing programs.

The working arrangements for this role are accurate as of the date of posting. This may change based on the project you’re engaged in, as well as business and client requirements. Rest assured; we will always be clear about role expectations.

Please note, this role is not able to offer visa transfer or sponsorship now or in the future.

The annual salary for this position is depending on experience and other qualifications of the successful candidate.

• Medical/Dental/Vision/Life Insurance
• Paid holidays plus Paid Time Off
• 401(k) plan and contributions
• Long-term/Short-term Disability
• Paid Parental Leave
• Employee Stock Purchase Plan

Disclaimer:
The salary, other compensation, and benefits information is accurate as of the date of this posting. Cognizant reserves the right to modify this information at any time, subject to applicable law.