Data & AI Engineer - Microsoft Fabric, Experimentation Data & Agent Development

Job Description

Must Have Technical/Functional

Skills:

• Core Data Engineering Competencies;
  Data Concepts & Data Modelling;
  Digital:
  Big Data Platforms; data pipelines design;
  Microsoft fabric data agents;
  Azure AI Services; AI and ML Integration;
  Analytical and problem solving skills;
  Performance tuning and monitoring;
  Digital:
  PySpark; ecommerce domain knowledge;
  Digital:
  Adobe Analytics;
  Digital:
  Customer Analytics;
  Adobe customer journey analytics; clickstream data

• Experimentation data enablement (Silver layer ownership) – Own the design, build, and maintenance of curated Silver-layer datasets in Microsoft Fabric to support experimentation reporting and analysis.
• Partner with the Data Reporting/BI team to identify required dimensions, metrics, and joins (visitor/session, variant, campaign/flight, geo, device, channel, funnel steps, conversion events) and ensure these are available in Silver.
• Translate experimentation team needs into standardized, reusable data products (tables/views) that can be consumed consistently for scorecards, dashboards, and ad hoc analysis.
• Ensure Silver-layer outputs are analysis-ready (cleaned, conformed, deduplicated, and aligned to agreed definitions).
• Data gap analysis and assessment:
  Conduct regular gap assessments between experimentation requirements, existing Silver layer availability, and upstream telemetry/source systems. Identify missing/incorrect fields, inconsistent definitions, data latency issues, or join-key problems; document business impact, severity/priority, remediation approach, timelines and dependencies. Provide recommendations on data model improvements to reduce recurring data quality issues.
• Gold layer requirements and stakeholder requirement gathering:
  Lead requirement workshops with stakeholders (experimentation, measurement, BI/reporting, engineering) to define Gold layer outputs: KPI definitions and calculation logic, experiment attribution rules, scorecard structure, segmentation needs and slicing dimensions, governance and refresh SLAs. Produce clear functional + technical specifications: source-to-target mappings, data dictionary, metric definitions, validation rules, and acceptance criteria. Drive alignment on single source of truth definitions to avoid mismatch across CJA/Power BI/scorecards.
• Data pipeline engineering (1DS + Fabric pipelines/ADF):
  Build and operate robust pipelines using Microsoft Fabric Pipelines and/or ADF to ingest and transform data into Silver and Gold layers. Understand and work with 1DS (telemetry) pipelines to ensure required events and attributes flow correctly into Fabric. Implement reliable orchestration, incremental loads, error handling, and monitoring to meet experimentation reporting timelines.
• Data validation and reconciliation (CJA included):
  Perform data validation and reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA) – event counts, session/user logic, conversions, experiment/variant attribution consistency, time window alignment and filtering rules. Create validation checks and automated routines for missing data detection, duplicate events, schema drift, metric anomalies, SRM-supporting signals (where applicable from data). Document issues and coordinate fixes with upstream owners.
• Experimentation lifecycle and scorecard readiness:
  Support the experimentation lifecycle by ensuring datasets are ready for pre-launch readiness checks, launch measurement, scorecard generation, ongoing health checks, post-test learnings/archives.
• Enable consistent scorecard outputs by curating experiment metadata (test IDs, start/end dates, allocations), KPI metrics (primary/secondary), and slicing dimensions required by experimentation stakeholders.
• AI agent design & build for experimentation team:
  Design and build AI-powered agents (Fabric Data Agents / Copilot / Azure OpenAI) to accelerate experimentation workflows – automated scorecard creation and narrative summaries, self-serve Q&A over experimentation datasets, anomaly explanations and investigation guidance, metric definition assistant / data dictionary lookup, pipeline health and data quality assistant. Define…