Data Engineer

Job Overview

Location: Norwell, MA Day 1 Onsite

Duration: 1 Year

Position: Data Engineer

• Design and implement Silver and Gold layer data models following medallion architecture best practices.
• Perform data cleansing, standardization, enrichment, and aggregation to support analytics and reporting.
• Build optimized PySpark-based transformations for large-scale data processing.
• Ensure data consistency, performance, and scalability across datasets.
• Build and maintain CI/CD pipelines using Git-based workflows (Azure Dev Ops / Git Hub).
• Use ARM templates (or IaC equivalents) for automated infrastructure provisioning.
• Enable automated deployment of data pipelines, notebooks, and configurations.
• Follow Dev Ops best practices for version control, branching, and release management.
• Create modular, maintainable, and testable Python code.
• Support automation of metadata, logging, alerting, and operational tasks.
• Implement data quality libraries such as Great Expectations.
• Define and automate data quality rules (completeness, accuracy, freshness, consistency).
• Monitor, log, and troubleshoot data quality issues proactively.
• Work closely with data architects, analysts, QA, and business stakeholders.
• Translate business and analytical requirements into robust data engineering solutions.
• Participate in Agile ceremonies and support sprint-based delivery.

• Can you explain your experience with building data pipelines using PySpark?
• How do you ensure data consistency and scalability across datasets?
• Have you worked with ARM templates for automated infrastructure provisioning?
• How do you integrate data quality checks into CI/CD workflows?
• Can you provide an example of a data quality rule you have defined and automated?

• Strong hands-on experience with Silver & Gold layer development, Python (automation and data processing) and PySpark.
• Experience with Great Expectations (data quality framework).
• Experience with CI/CD pipelines using Git-based tools.
• Hands-on experience with ARM templates or infrastructure-as-code concepts.
• Strong understanding of data modeling and medallion architecture.
• Experience working with large datasets in distributed environments.

• Microsoft Certified:
  Azure Data Engineer Associate or Azure Enterprise Data Analyst Associate.
• Waste management or oil and gas domain knowledge.