DataBricks Data Engineer

Due to client requirements, applicants must be able to work on a w2 basis

We are seeking an experienced Databricks Data Engineer to join a growing data and analytics team responsible for building scalable, high-performance data solutions. This role is ideal for a hands‑on engineer with deep expertise in Databricks, modern data engineering practices, and Sales Incentive Compensation (SIC) processes. Candidates with a proven track record of designing enterprise data pipelines, collaborating with cross‑functional stakeholders, and delivering reliable, business‑critical data solutions will be prioritized for interviews.

• 7+ years of professional experience in data engineering, including at least 3 years of hands‑on Databricks development.
• Strong expertise with Databricks components, including Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
• Advanced proficiency in SQL and Python or Scala for data engineering and pipeline development.
• Extensive experience building scalable ETL/ELT pipelines, data models, and enterprise data architectures.
• Functional and technical knowledge of Sales Incentive Compensation (SIC), including quota management, attainment calculations, commission processing, and payout reporting.
• Experience integrating cloud‑based data platforms and services within Azure, AWS, or Google Cloud environments.
• Strong analytical, troubleshooting, and performance optimization skills.
• Excellent communication skills with the ability to collaborate effectively across technical and business teams.

In this role, you will design and optimize enterprise data solutions that power reporting, analytics, and sales compensation processes. You will work closely with business stakeholders, finance teams, sales operations, and technology partners to transform complex business requirements into scalable, secure, and maintainable data architectures. Success in this position requires a proactive mindset, strong ownership of data quality, and the ability to continuously improve platform performance through automation and best practices.

• Design, develop, and maintain scalable data pipelines using the Databricks platform.
• Build and optimize Delta Lake tables, notebooks, workflows, and supporting data models for enterprise analytics.
• Architect integrated data solutions that support long‑term scalability, maintainability, and performance.
• Enhance existing Databricks work spaces by improving automation, reliability, and operational efficiency.
• Develop data pipelines supporting Sales Incentive Compensation processes, including quota allocation, commission calculations, attainment tracking, and payout reporting.
• Partner with Sales Operations, Finance, Business Intelligence, and technical teams to translate business rules into accurate and auditable data transformations.
• Perform testing, validation, debugging, and performance tuning to ensure data accuracy and pipeline reliability.
• Integrate Databricks with cloud platforms and enterprise data sources, including modern streaming and ingestion technologies.
• Evaluate and implement new Databricks capabilities such as Unity Catalog, Delta Live Tables, and Photon to improve governance, performance, and data quality.
• Produce clear technical documentation covering architecture, data models, pipeline configurations, and development standards.
• Provide technical guidance and production support while troubleshooting complex data issues across the platform.
• Promote data engineering best practices, governance standards, and continuous improvement across the data ecosystem.

• This position works closely with cross‑functional teams including Data Engineering, Finance, Sales Operations, Information Technology, and Business Intelligence.
• Experience with Sales Incentive Compensation platforms such as Anaplan, Oracle Incentive Compensation Management (ICM), Varicent, or comparable solutions is highly desirable.
• Candidates should demonstrate the ability to interpret complex business logic and translate it into scalable, maintainable data engineering solutions.
• A collaborative mindset combined with the ability to work independently and manage multiple priorities is essential.
• Experience with cloud‑native data services such as Azure Data Factory, Event Hubs, Kafka, or equivalent technologies is preferred.
• A passion for building reliable, high‑quality data platforms and delivering exceptional user experiences is highly valued.
• Databricks Certified Data Engineer Associate or Professional certification is considered an advantage but is not required.

W2 employees of Overture Partners who work 30 or more hours per week are eligible for the following benefits: medical (choice of 3 plans), 401(k) starting on day one, a variety of voluntary benefits including life and disability insurance, and sick time if required by law in the worked‑in state/locality.