Azure Data Lead

We are looking for Azure Data Lead – Guindy, Chennai. This isa fulltime permanent position.

Note:

Looking for someone who can join in a week.

What you’ll do:

- Lead the design, development, and maintenance of scalable data pipelines and lakehouse architectures using Azure Databricks (Notebooks, PySpark, SQL).
- Architect and implement Medallion Architecture (Bronze, Silver, Gold layers) to drive optimal data transformation and organization.
- Experience in Integrating ETL process from external API’s, Data Dumps, Public Calls and other data warehouse sources.
- Oversee data governance, access control, and lineage using Databricks Unity Catalog; champion best practices for metadata management.
- Manage and mentor a team of data engineers; provide technical guidance and career development opportunities.
- Direct the management of data storage solutions using Azure Data Lake Storage (ADLS), ensuring robust access policies.
- Establish and maintain Delta Lake tables for reliable, scalable ACID transactions and incremental processing across projects.
- Optimize Spark jobs and Databricks clusters for performance, scalability, and cost efficiency at enterprise scale.
- Lead ingestion and transformation strategies from diverse sources (APIs, files, databases, cloud platforms).
- Partner with stakeholders and cross-functional teams to deliver well-governed, reusable data models and analytics solutions.
- Drive implementation of data quality frameworks, error handling mechanisms, and robust monitoring practices.
- Enforce data security, governance, and compliance requirements across all Azure and cloud data assets.
- Utilize Azure Synapse Analytics, Key Vault, and Azure Monitor to build resilient, secure, and observable data platforms.
- Own CI/CD and deployment best practices using Git and Azure Dev Ops; establish automation standards and code review processes.
- Stay ahead of industry trends in big data, ETL, and lakehouse technologies; proactively recommend improvements.
- Collaborate with data science and AI teams to support machine learning (ML), generative AI, and large language model (LLM) initiatives using enterprise data.
- Guide data consultation efforts for enterprise digital transformation, advising stakeholders on data architecture modernization and strategic priorities.
- Develop and articulate an “AI-ready” data roadmap, enabling the business to leverage advanced analytics, ML, and LLM solutions.
- Ensure data pipelines and platforms are optimized for ML and AI workloads, including integration and preparation of datasets for model training and inference.

What we seek in you:

- Bachelor’s or Master’s in Computer Science, Engineering, or related field.
- 8+ years of experience in data engineering, with 2+ years in a technical lead or supervisory role.
- Advanced expertise in Databricks, PySpark, SQL, Azure cloud data engineering, and related tools.
- Proven track record architecting and leading end-to-end pipeline solutions leveraging Medallion architecture, Unity Catalog, Iceberg-Databricks and Delta Lake.
- Experience mentoring and scaling data engineering teams; excellent leadership and communication skills.
- Deep understanding of big data principles, data lakehouse architectures, and cloud ETL orchestration.
- Hands-on with Azure Data Lake Storage (ADLS), Data Factory (ADF), Synapse Analytics, Key Vault, Sql Server and Azure Monitor.
- Skilled in CI/CD pipelines, automation with Git and Azure Dev Ops, and code quality enforcement.
- Strong analytical mindset with a strategic approach to problem-solving and stakeholder management.
- Certifications in Azure/Data Engineering (Databricks, Azure) are highly desired.
- Proven ability to consult on data strategy and architecture for large-scale digital transformation initiatives.
- Strong knowledge of data preparation, feature engineering, and optimization for ML and AI workloads.
- Experience collaborating cross-functionally with data science, analytics, and business teams on ML/AI use cases.

Life at Next:

At our core, we're driven by the mission of tailoring growth for our customers by enabling them to transform their aspirations into tangible outcomes. We're dedicated to…