Data Engineer Lead

• Location 2859 Paces Ferry Road Southeast,Atlanta, GA, 30339,United States
• Employee Type Full Time Exempt

Description

Integrity. Care. Trust. Compassion. Expertise.
Do these words resonate with you?
These values of Builders culture create success in all we do. We strive to provide deeply supportive partnerships to our customers, agents, and each other.

Builders is proud to be named among the Great Places to Work. Our award-winning culture has earned top marks in Company Direction, Employee Appreciation, Work-Life Balance, Leadership, and Compensation and Benefits. Our strong culture keeps us Built Strong in a forever-changing world, and our AM Best A Rating is evidence of our financial strength.

Position Summary

The Data Engineering Lead serves as the technical owner of the company’s Azure Databricks lakehouse platform supporting its property and casualty insurance business. This role drives the design, development, and evolution of the platform, leveraging Databricks technologies including Delta Lake, Unity Catalog, Lakeflow pipelines and Databricks SQL. The Data Engineering Lead establishes and promotes engineering standards, best practices and architectural principles to ensure a scalable, secure, reliable and high-performing data platform.

A key responsibility is to lead the phased migration and modernization of data from Guidewire and legacy policy, claims and billing systems into the Azure Databricks lakehouse. The role champions effective data governance, security and lineage through Unity Catalog, partnering closely with the Data Governance Lead to translate governance policies into practical, consistent engineering practices. In addition the Data Engineering Lead provides technical direction, mentorship and leadership to a team of engineers and contractors, fostering collaboration, engineering excellence and the successful delivery of strategic data initiatives.

• Design the Databricks lakehouse architecture (bronze/silver/gold medallion on Delta Lake) for property and casualty policy, rating, claims, billing, reinsurance and customer data.
• Plan and lead the phased data migration from Guidewire (Policy Center, Claim Center, Billing Center) and legacy systems into the lakehouse on Azure Databricks, including historical loads, incremental/CDC (change data capture) loads, reconciliation and decommissioning support.
• Model property and casualty insurance data: accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments and recoveries.
• Deliver data for Actuarial and Finance: written/earned/unearned premium, loss triangles, IBNR (incurred but not reported) inputs, loss ratios, Schedule P and NAIC (National Association of Insurance Commissioners) statutory reporting.
• Integrate third-party data sources such as geospatial and weather data.
• Define the Unity Catalog design: metastore, catalogs, schemas, environments (development, test and production), naming conventions, permissions, row/column-level security and lineage.
• Manage pipeline orchestration and operations in Databricks: design job scheduling and dependencies with Databricks Workflows (Lakeflow Jobs), set up retries, alerting and SLA (Service Level Agreement) monitoring and establish failure handling and runbooks for production pipelines.
• Manage performance and cost:
  Spark tuning, partitioning and liquid clustering, Photon, cluster and SQL warehouse sizing.
• Shape the data infrastructure with IT (Information Technology) and the Platform Engineer: storage, networking, security, identity, IaC (Infrastructure as Code) and CI/CD (Continuous Integration/Continuous Deployment).
• Set engineering standards for pipelines, testing, code review and documentation; lead, mentor and review the work of Data Engineers and junior contractors.
• Partner with the data governance lead to implement access controls, PII (personally identifiable information) masking, lineage and data quality rules.
• Perform other duties as assigned.

• Associate degree or an equivalent combination of education and experience
• Ten or more years of experience in data engineering, including six or more years of hands-on experience with Databricks (PySpark, Spark SQL, Delta Lake); experience leading or mentoring engineering teams
• Certification as a Databricks Data Engineer Professional; certification as an Azure data or solutions architect is preferred
• Knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage and system…