Senior Data Engineer; Databricks

Responsibilities

• The Sugar Predict platform powers revenue intelligence for mid-market enterprises by fusing ERP and CRM data into actionable insights.
• As a Senior Data Engineer, you will own the Databricks pipelines that make this possible, driving production reliability, cost efficiency, and platform growth through customer onboarding and legacy modernization.
• You will work closely with ML engineers, product teams, and the Enterprise Architecture team to ensure the data backbone behind Sugar Predict is always fast, clean, and ready to deliver at a global scale.
• Own Databricks production support for the Sugar Predict data platform, including monitoring, alerting, and incident response across all production data flows.
• Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability across internal and external stakeholders.
• Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows while tracking impacts through measurable KPIs.
• Migrate legacy ETL/ELT pipelines to Databricks, building automation tooling to reduce manual intervention and ensure uninterrupted data delivery during transitions.
• Support new customers onboarding by provisioning, validating, and hardening tenant data pipelines that deliver reliable, isolated data from day one.
• Design and build high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments.
• Own the Delta Lake architecture including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns.
• Enforce data security best practices across Databricks environments, including role-based access control, secrets management, and compliance requirements for enterprise CRM and ERP data.
• Implement data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly supports Sugar Predict prediction accuracy.
• Apply and enforce multi-tenant data isolation patterns ensuring reliable, secure data delivery across Sugar Predict enterprise customers.
• Partner with the Enterprise Architecture team to ensure Sugar Predict data pipelines integrate seamlessly with the broader SugarAI product ecosystem.
• Support a globally distributed operation through on-call rotation and after-hours incident response, meeting SLAs across multiple time zones.
• Maintain technical documentation, runbooks, and architectural decision records, contributing to team knowledge sharing and operational readiness across on-call and incident response scenarios.
• Apply CI/CD best practices to data pipeline development, including version control, automated testing, and deployment tooling to ensure reliable and repeatable pipeline delivery.

• Financial Planning Services (Origin)
• Online Legal Services (Rocket Lawyer)
• Health and Wellness Reimbursement Program
• Excellent healthcare package for you and your family
• Unlimited Paid Time Off
• Savings and Investment – 401(k) match
• Corporate Benefit Program (Working Advantage). This benefit offers you exclusive travel and entertainment offers and special discounts that are not available to the general public
• Paid Parental Leave
• Discounted Pet Insurance (Embrace Pet Insurance)
• We are a merit-based company - many opportunities to learn, excel and grow your career!
• Educational Resources - Career & Personal Development Program
• Travel Discounts
• Employee Referral Bonus Program

• Hands‑on experience with Delta Lake including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns.
• Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production.
• Hands‑on experience with pipeline performance tuning and compute optimization in production Databricks environments.
• 4+ years of data engineering experience.
• Experience supporting large‑scale multi‑tenant architectures with a focus on tenant isolation, per‑tenant performance, and data privacy, including navigating tools and…