Data Engineer

About The Role

The data engineer role is changing. The traditional pattern - build a pipeline, hand it off, wait for someone else to figure out if the data is right - doesn't work anymore. The engineers who create the most value now are the ones who go deep into the business domain, understand the problem firsthand, and use AI tools to move from question to working solution in hours instead of sprints.

• Design, build, and operate the data platform on Azure Databricks: ingestion, transformation, storage, and serving layers that power analytics, AI models, and operational reporting.
• Build and maintain data pipelines across the ecosystem:
  Salesforce, SQL Server, Snowflake, third‑party sources, and the new cloud‑native payments platform.
• Engineer for quality and trust with validation checks, anomaly detection, lineage tracking, and documentation to ensure every downstream consumer can rely on the data.
• Write clean, version‑controlled, production‑grade code. Think like a software engineer building a product, not a script runner maintaining jobs.

• Partner directly with business stakeholders across physician growth, member services, finance, and operations to understand how data drives decisions and then build for those decisions, not for abstract requirements.
• Act as a technical product owner for your domain areas: own the backlog, prioritize based on business impact, and ship iteratively without waiting for a PM to sequence your work.
• Translate ambiguous business questions into data models, feature tables, and curated datasets that analysts and data scientists can build on immediately.
• Close the loop: follow your data through to the dashboard, the model, or the operational workflow and validate that it's actually driving the outcome.

• Use Claude Code and agentic development as your primary workflow: AI‑driven pipeline generation, automated testing, rapid prototyping to ship at a pace that would be impossible with traditional approaches.
• Build data infrastructure that is AI‑ready: well‑documented, semantically clear, and structured so that AI tools and agents can reason over it effectively.
• Scout, evaluate, and adopt emerging AI tools and platforms that make the data team faster by separating real value from hype with hands‑on testing.
• Share what you learn. Document patterns, run demos, and help the broader team adopt AI‑first workflows with confidence.

• Sales team tracks every doctor engagement (email, phone, cold calls, martech tools)
• Pull engagement data from multiple source systems into Databricks via APIs
• Model and layer the data, serve to Power BI dashboards
• Expose dataset to Claude so analytics team can ask questions directly
• Build engagement scoring model:
  Claude generates logic (vibe coding), Python validates the scoring, scores written back into Databricks

• A data engineer who refuses to stay in the technical silo. You go find the business problem, not wait for it to arrive as a Jira ticket.
• Someone who thinks like a product owner: you prioritize by impact, ship incrementally, and own the outcome, not just the pipeline.
• The kind of engineer who's already using AI tools to write, test, and deploy code before anyone asked you to and who knows when the output needs a human eye.
• Equally comfortable writing a Spark transformation, debugging a Salesforce data sync, presenting findings to leadership, and pushing back on a vague requirement until it's sharp enough to build against.
• Pragmatic over perfectionist. You optimize for business impact and speed to value, not theoretical elegance.
• A strong collaborator who elevates the people around them through clear communication, reusable patterns, and generous knowledge sharing.

• BS in Computer Science, Data Science, or related field; 6+ years in data engineering or a hybrid data engineering/analytics role.
• Deep hands‑on experience with Azure Databricks: notebooks, Delta Lake, Unity Catalog, and production‑scale pipelines. Databricks 3+ years is non‑negotiable.
• Strong Python and…