Data Engineer Lead

About The Role

The Data Engineer Lead is a hands-on engineering role within AIG's Data Engineering organization. You will design, build, and operate production-grade data pipelines and platforms - and bring sound engineering judgment to every solution you deliver.

AIG | Data Office

• Data Engineering

The Data Engineer Lead is a hands-on engineering role within AIG's Data Engineering organization. You will design, build, and operate production-grade data pipelines and platforms - and bring sound engineering judgment to every solution you deliver.

We actively leverage GenAI tooling - including Claude Code and Snowflake Cortex - as a genuine productivity multiplier across the development lifecycle. We want someone who is motivated to learn, keeps up with a fast-moving space, and continuously finds new ways to put these tools to work.

You will collaborate closely with the product owner, business stakeholders, and fellow engineers. Strong communication and analytical skills matter here as much as technical depth - the ability to ask the right questions, think through trade-offs clearly, and explain your reasoning is part of the job.

• Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3 with integrated data quality checks and observability.
• Own end-to-end delivery of data products from requirements through production, working within an Agile/Scrum framework - including sprint planning, CI/CD, release management, and production readiness
• Translate business and product requirements into concrete technical designs, defining pipeline structure, data models, and SLAs.
• Demonstrate a sound understanding of architecture and design patterns, evaluate trade-offs of design decisions, and articulate clear rationale to peers and stakeholders.
• Set and enforce engineering standards - coding conventions, data quality frameworks, reusable pipeline patterns, and observability hooks - across the team.
• Conduct thorough code reviews and pair with engineers on hard problems, raising the technical quality of the whole team.
• Leverage GenAI tooling
  - Claude Code CLI, Snowflake Cortex - throughout the development lifecycle to write better code faster and improve solution quality.
• Collaborate with product owners to refine requirements, surface trade-offs, and push back constructively when scope is technically unworkable.
• Communicate design decisions and delivery updates clearly to both engineering peers and non-technical stakeholders.
• Mentor junior engineers through review, pairing, and knowledge sharing, building team capability alongside product delivery.
• Produce clear technical documentation
  - Architecture Decision Record (ADR), runbooks, data dictionaries, and pipeline lineage notes - as a standard part of delivery, not an afterthought.
• Evaluate and drive adoption of productivity tooling and best practices, setting team norms for safe and effective use.

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical field.
• 7+ years of hands-on data engineering experience with a consistent track record delivering production data pipelines end-to-end.
• Strong command of PySpark, Python, and SQL - including Spark optimization, partition management, and query performance tuning in Snowflake.
• Hands-on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, and Cloud Watch.
• Working knowledge of architecture and design principles
  - Lakehouse patterns, data modeling, fault-tolerant ingestion - with demonstrated ability to apply them in practice.
• Strong analytical and problem-solving skills: able to decompose complex requirements, reason through design trade-offs, and arrive at practical, well-justified solutions.
• Clear communicator - able to explain technical decisions and design rationale in plain language to both engineering peers and business stakeholders.
• Self-motivated learner with a genuine interest in staying current with evolving tools, frameworks, and engineering best practices.

• AWS certification
  - Solutions Architect Associate or equivalent.
• Domain familiarity with insurance or financial services data - claims, policy, risk, or actuarial structures.
• Experience writing ADRs, design notes, or technical documentation in a collaborative engineering team.
• Demonstrated use of GenAI tooling (e.g. Claude Code CLI or Snowflake Cortex) to improve productivity or solution quality.

AIG's Data Office is a team of…