Databricks Architect​/Senior Data Engineer

Overview

Note:
Only applications from U.S. citizens or lawful permanent residents (Green Card Holders) will be considered.

We believe work should be innately rewarding and a team-building venture. Working with our teammates and clients should be an enjoyable journey where we can learn, grow as professionals, and achieve amazing results. Our core values revolve around this philosophy. We are relentlessly committed to helping our clients achieve their business goals, leapfrog the competition, and become leaders in their industry.

What drives us forward is the culture of creativity combined with a disciplined approach, passion for learning & innovation, and a ‘can-do’ attitude!

We are seeking a highly experienced Databricks professional with deep expertise in data engineering, distributed computing, and cloud-based data platforms. The ideal candidate is both an architect and a hands-on engineer who can design scalable data solutions while actively contributing to development, optimization, and deployment.

This role requires strong technical leadership, a deep understanding of modern data architectures, and the ability to implement best practices in Data Ops, performance optimization, and data governance.

Experience with modern AI/GenAI-enabled data platforms and real-time data processing environments is highly desirable.

The Databricks Senior Data Engineer will play a critical role in designing, implementing, and optimizing enterprise-scale data platforms using the Databricks Lakehouse architecture. This role combines architecture leadership with hands-on engineering, focusing on building scalable, secure, and high-performance data pipelines and platforms. The ideal candidate will establish coding standards, define data architecture frameworks such as the Medallion Architecture, and guide the end-to-end development lifecycle of modern data solutions.

This individual will collaborate with cross-functional stakeholders, including data engineers, BI developers, analysts, and business leaders, to deliver robust data platforms that enable advanced analytics, reporting, and AI-driven decision-making.

• Architecture & Design: Architect and design scalable, reliable data platforms and complex ETL/ELT and streaming workflows for the Databricks Lakehouse Platform (Delta Lake, Spark).
• Hands-On Development: Write, test, and optimize code in Python, PySpark, and SQL for data ingestion, transformation, and processing.
• Data Ops & Automation: Implement CI/CD, monitoring, and automation (e.g., with Azure Dev Ops, DBX) for data pipelines.
• Stakeholder
  
  Collaboration:
  
  Work with BI developers, analysts, and business users to define requirements and deliver data-driven solutions.
• Performance Optimization: Tune delta tables, Spark jobs, and SQL queries for maximum efficiency and scalability.
• GenAI Applications Development: It is a big plus to have experience in GenAI application development

• 8+ years of experience in data engineering, with strong hands-on expertise in Databricks and Apache Spark.
• Proven experience designing and implementing scalable ETL/ELT pipelines in cloud environments.
• Strong programming skills in Python and SQL; experience with PySpark required.
• Hands-on experience with Databricks Lakehouse, Delta Lake, and distributed data processing.
• Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP (Azure preferred).
• Experience with CI/CD pipelines, Git, and Dev Ops practices for data engineering.
• Strong understanding of data architecture, data modeling, and performance optimization.
• Experience working with cross-functional teams to deliver enterprise data solutions.
• Tackles complex data challenges, ensuring data quality and reliable delivery.

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
• Experience with data integration tools such as Azure Data Factory or similar platforms.
• Familiarity with cloud data warehouses such as Databricks, Snowflake, or Azure Fabric.
• Experience supporting analytics, reporting, or AI/ML workloads is highly desirable.
• Databricks, Azure, or cloud…