ETL Lead

Job Tittle: ETL Lead | Insurance Domain

Job Location:

Green Ville , SC (Onsite)

Duration : 6+ Months contract

Visa: USC and GC

Years of Exp: 12+

• Production Support, Operations & Reliability
• The ETL Team Lead owns end-to-end operational support for Canal’s existing data stack:
• Ensure success of day-to-day production Operations.
• Monitor daily ETL loads across SQL jobs, DHIC (GW Data Hub and Info Center) and legacy SSIS packages.
• Work with AMS team where necessary to troubleshoot pipeline failures, performance issues, schema mismatches, permissions issues, and cloud resource failures.
• Work with AMS team where necessary to perform root-cause analysis and to implement permanent fixes.
• Ensure SLA adherence and on-time delivery of critical reporting datasets for scheduled ETL jobs .
• Provide direction for both AMS and ETL Developers for Legacy & Current ETL maintenance.
• Refactor or retire outdated or redundant ETL processes.
• Maintain and improve existing pipelines that utilize the following technologies:
• Microsoft SQL Server Database programming
• T-SQL Scripting
• SQL Server Integration Services
• Microsoft Power Shell
• Guidewire Data Hub and Info Center
• Oracle Database programming
• Oracle PL-SQL Scripting
• SAP BODS (SAP Business Objects Data Services)
• PostgreSQL Scripting
• Operational Excellence
• Work with AMS team to assist with the creation and/or enhancement of operational runbooks, SOPs, monitoring dashboards, and incident response workflows.
• Partner with other IT operational segments, business SMEs, and AMS team to minimize downtime and to ensure that business SLAs are met.
• Improve upon existing, as well as implement new proactive for daily processing.
• Business Continuity
• Work with AMS team to ensure development support coverage for critical data pipelines (rotation-based).
• Support month-end and quarter-end financial reporting cycles.
• Coordinate production releases and validate deployments.
• The ETL Tech Lead will become the steady-state technical owner of the entire data operations layer during the Canal modernization journey.
• Technical Leadership & Collaboration
• Serve as technical lead guiding onshore/offshore developers.
• Review code, enforce best practices, and mentor junior engineers.
• Partner with Scrum Masters, Project Mangers, Enterprise Architecture, QA Automation, Change Management, and AMS support teams.
• Data Ingestion, ETL/ELT Development & Optimization
• Develop reusable ingestion patterns for Guidewire Data Hub and Info Center, Hub Spot, Telematics, and other facets of the business.
• Work with Canal Architects to modernize existing ETL workloads using Delta Lake, Medallion Architecture, and Fabric Lakehouse.
• Build scalable data ingestion pipelines using potential upcoming technology (Azure Data Factory, MS Fabric , Databricks, Synapse Pipelines, etc.).
• Work to bring in Internal and External integrations data into the platform.
• Real-Time, Streaming & Event-Driven Engineering
• Proven experience designing and implementing real-time data pipelines using Event Hub, Fabric Real-Time Analytics, Databricks Structured Streaming, and KQL-based event processing.
• Develop and enable real-time operational insights and automation capabilities—including Telematics alerting, FNOL automation, and fraud/VNOS/VNOP detection—through event-driven architectures and streaming analytics.
• Modern Azure Data Stack Leadership
• Lead the strategy, design, and engineering of Canal’s modern Azure data ecosystem using next-generation tools and Medallion Architecture, including:
• Implementing Medallion Architecture (Bronze/Silver/Gold) across Fabric Lakehouse, Warehouse, Eventhouse, and KQL Database.
• Leveraging Delta tables with schema enforcement, ACID compliance, and versioning.
• Building ingestion and transformation pipelines using Dataflows Gen2, Fabric Pipelines, and Notebook-based ETL.
• Enabling real-time analytics using RTA, Event Streams, and Event Hub.
• Driving notebook-based transformations and advanced engineering using Databricks and Synapse runtimes.
• Data Modeling, Curation & Governance
• Develop curated, analytics-ready datasets to support Power BI, operational reporting, and advanced analytics use cases.
• Assist Canal architect with implementation of Data Governance tools.
• Establish robust data quality, validation, alerting, and observability frameworks to maintain accuracy, consistency, and trust in enterprise data.
• AI/ML Data Enablement (Optional)
• Prepare ML-ready datasets for pricing, risk, fraud detection, underwriting, claims leakage, and predictive insights