Lead Data Engineer, Associate Director

Requisition

Location:

Chicago, IL, US

• Lead the design and architecture of end-to-end data pipelines and solutions on modern cloud-based platforms, including Snowflake, Databricks, and AWS.
• Design and implement data solutions using PostgreSQL for relational data and MongoDB for No
  
  SQL requirements, ensuring optimal performance and scalability.
• Architect and deploy containerized data applications using Docker, Kubernetes, and AWS EKS, incorporating Git Hub Actions for automated deployments.
• Design and implement CI/CD pipelines using Git Hub Actions, establish branching strategies, and ensure automated testing, code quality checks, and security scanning.
• Collaborate with cross-functional teams—including Data Scientists, Analytics teams, and business stakeholders—to translate requirements into scalable technical solutions.
• Mentor and guide data engineers by promoting technical excellence, establishing coding standards, and conducting architecture reviews.
• Drive data platform modernization initiatives and ensure data quality, reliability, and governance across all data systems.
• Design and implement AI‑enhanced data pipelines that leverage LLMs and Agentic AI frameworks to automate data quality checks, anomaly detection, and intelligent data transformation workflows.
• Architect data infrastructure to support AI/ML workloads, including feature stores, vector databases, and real‑time inference pipelines integrated with cloud‑native services.
• Leverage established standards and best practices to integrate AI agents into data engineering workflows, including context management protocols (MCP) for seamless AI‑to‑data‑platform communication.

• 8+ years of data engineering experience, including 3+ years in a lead role architecting large‑scale data platforms.
• Expert‑level proficiency in Java, Springboot for building cloud‑native data processing solutions running on Docker/Kubernetes.
• Deep hands‑on experience with Apache Airflow, Snowflake, and Databricks.
• Production database experience with PostgreSQL (design, optimization, replication) and MongoDB (document modeling, sharding, replica sets).
• Proven CI/CD and Git Ops experience using Git Hub, Git Hub Actions, and ArgoCD for automated deployments and multi‑environment management.
• Proficiency with agile tools such as JIRA for sprint management and Confluence for technical documentation and knowledge sharing.
• Excellent analytical, problem‑solving, and communication skills, with the ability to explain complex concepts to non‑technical stakeholders and drive initiatives in complex environments.
• Working knowledge of AI/ML frameworks (Lang Chain, Llama Index, Auto Gen, etc.) and an understanding of how Agentic AI can enhance data engineering workflows through automated data validation, intelligent orchestration, and self‑healing pipelines.
• Practical understanding of AI integration patterns in data platforms, including prompt engineering, RAG architectures, and vector database implementations.
• Familiarity with Model Context Protocol (MCP) or similar frameworks for enabling AI agents to interact securely and efficiently with data sources, APIs, and tools.
• Experience with AI‑powered development tools such as Git Hub Copilot and Amazon Q.

• Experience with code quality metrics and shift‑left principles.
• Experience testing container resiliency (Docker/Kubernetes).
• Experience building large and high‑performing data pipelines.
• Exposure to Playwright and BDD for automated testing.
• Exposure to the financial industry and data platforms (data warehouses, data lakes).
• Experience with modern data stack tools, data mesh/fabric architectures, and streaming platforms (Kafka, Kinesis).
• Proficiency with observability tools (Datadog) and data quality/governance frameworks.
• Understanding of data security and compliance standards (GDPR, SOC 2, CCPA) and contributions to open‑source data projects.
• Relevant certifications (AWS Data Analytics/Solutions Architect, Databricks/Snowflake Data Engineer, CKA).
• Hands‑on experience building production Agentic AI systems that operate on data platforms, including multi‑agent orchestration and intelligent pipeline…