Data Engineer

As a Data Engineer, you will be responsible for:

Data Engineering & Development

* Design, build, and maintain high-quality, scalable, and tested data pipelines.

* Develop and manage Databricks structured streaming pipelines.

* Build and optimize event-driven and real-time data processing solutions.

* Implement and maintain Unity Catalog-based Lakehouse architecture.

* Develop analytics-ready datasets to support business insights and reporting.

Platform & Automation

* Build and manage CI/CD pipelines using Azure Dev Ops.

* Identify and implement automation opportunities across workflows.

* Ensure reliable and stable data platform operations.

* Apply governance, security, and documentation standards.

Data Quality & Reliability

* Establish the Data Lakehouse as a trusted and reliable source of truth.

* Monitor, troubleshoot, and resolve data incidents.

* Support business users and technical teams with data-related queries.

* Continuously improve platform performance and reliability.

Collaboration & Support

* Work closely with data science, analytics, platform, and business teams.

* Champion data engineering best practices.

* Provide technical guidance and mentorship where required.

* Contribute to a culture of learning, quality, and continuous improvement.

Essential Skills

* Strong experience with Azure Databricks and cloud data platforms.

* Advanced proficiency in Python, PySpark, and SQL.

* Experience developing Spark/Databricks pipelines.

* Hands-on experience with structured streaming and event-driven systems.

* Strong understanding of Lakehouse architecture and best practices.

* Experience with Unity Catalog.

* Expertise in Azure Dev Ops and CI/CD pipelines.

* Knowledge of data modelling (dimensional/star schemas).

* Experience working in Agile environments.

Desirable Skills

* Exposure to multiple data technology stacks.

* Experience in large-scale enterprise environments.

* Knowledge of security, governance, and compliance frameworks