Manager - Data Engineering; TPA)

Role

Purpose:

To ensure the organization has reliable, well-governed, and analytics-ready data by building and maintaining robust data pipelines and models. The role exists to make trusted data available securely and cost-effectively to support enterprise reporting, BI, and data-driven decision-making.

• Build reusable, parameterized mappings and task flows in Informatica IDMC to standardize data ingestion.
• Implement change data capture (CDC), idempotent loads, schema evolution, and data-quality gates.
• Optimize Big Query loads using partitioning, clustering, and the appropriate load-versus-stream approach.
• Set up version control and CI/CD pipelines for data engineering assets.

• Profile source systems and define field-level mappings and transformation rules.
• Specify business logic, including joins, lookups, and derivations.
• Define data-quality rules, exception handling, and reject criteria.

• Parameterize task flows and configure schedules and dependencies.
• Implement retries, backoff, and checkpointing to ensure reliable processing.
• Integrate monitoring and alerting through the Ops console and Chat Ops.

• Define least-privilege IAM roles and service accounts for data access.
• Apply dataset, table, row, and column-level security and data masking.
• Enable audit logging and retention policies, and classify PHI/PII data.

• Implement Medallion Architecture across the Bronze, Silver, and Gold layers.
• Develop scalable, curated data models for analytical and reporting needs.
• Define data quality, lineage, and governance standards for curated data.
• Collaborate with business and analytics teams to create trusted datasets.

• Track data freshness, job health, volumes, and anomalies.
• Monitor SLAs across job duration, errors, cost per TB, and slot usage.
• Tune Big Query performance through query optimisation and resource management.
• Optimize cost through storage lifecycle management, query tuning, and caching.