Principal Data Platform Engineer - Azure

Role Value Proposition

We are seeking a skilled Principal Data Platform Engineer with 5+ years of experience in designing, building, implementing, and managing data, analytics, and MLOps platform on Azure Cloud. Will be responsible for automating infrastructure, ensuring platform reliability, optimizing costs, and supporting data engineering, analytics, and machine learning workloads.

Platform Architecture & Design

• Design and architect scalable, secure, and cost-effective data, analytics, and MLOps platforms on Azure

• Define cloud architecture patterns and best practices for data workloads

• Create technical specifications and solution designs for complex cloud infrastructure

• Evaluate and recommend new technologies and tools to enhance platform capabilities

Infrastructure Automation & Management

• Implement Infrastructure as Code (IaC) using Azure Resource Manager (ARM), Bicep, or Terraform

• Develop and maintain CI/CD pipelines for infrastructure deployment and application delivery

• Automate provisioning and configuration management using tools like Azure Dev Ops, Git Hub Actions, or Jenkins

• Design and implement automated backup, disaster recovery, and high availability solutions

• Deploy and manage Azure data services including Azure Data Factory, Azure Synapse Analytics, Azure Databricks, and Azure Data Lake

• Implement data governance, security, and compliance frameworks

• Monitor and optimize data pipeline performance and resource utilization

• Ensure data quality, lineage, and metadata management across the platform

• Build and maintain MLOps infrastructure using Azure Machine Learning, MLflow, or similar platforms

• Implement model deployment, versioning, and monitoring solutions

• Develop automated ML pipeline orchestration and model lifecycle management

• Ensure reproducibility and scalability of machine learning workloads

• Implement security best practices including identity and access management (IAM)

• Configure network security, encryption, and data protection measures

• Ensure compliance with regulatory requirements (GDPR, HIPAA, SOX, etc.)

• Conduct security assessments and vulnerability management

• Monitor and optimize cloud resource costs through rightsizing and automation

• Implement cost allocation and chargeback mechanisms

• Performance tuning of data processing and analytics workloads

• Capacity planning and resource forecasting

• Work closely with data engineers, data scientists, and analytics teams

• Provide technical guidance and mentoring to junior team members

• Participate in on-call rotation for platform support and incident response

• Document processes, procedures, and system architecture

Required

• Bachelor's degree in Computer Science, Engineering, or related technical field, or 7+ equivalent professional experience will be considered

• Cloud Platforms: 5+ years of experience with Microsoft Azure, including deep knowledge of Azure services (Azure SQL, Cosmos DB, Azure Data Factory, Synapse, Databricks, Data Lake, Machine Learning, etc.)

• Infrastructure as Code:
  Proficiency in ARM templates, Bicep, Terraform, or similar tools

• Containerization:
  Experience with Docker, Azure Container Instances, and Azure Kubernetes Service (AKS)

• Dev Ops Skills

• CI/CD:
  Strong experience with Azure Dev Ops, Git Hub Actions, Jenkins, or similar platforms

• Automation of infrastructure, application deployments using Azure Dev Ops Pipeline / Github actions.

• Programming:
  Proficiency in Python, Power Shell, Bash, and/or other scripting languages

• Monitoring & Logging:
  Experience with Azure Monitor, Log Analytics, Application Insights, or similar tools

• Data Engineering:
  Knowledge of data pipeline orchestration tools (Databricks, Azure Data Factory)

• Soft Skills

• Excellent problem-solving and analytical skills

• Strong communication and collaboration abilities

• Ability to work in fast-paced, agile environments

• Leadership and mentoring capabilities

• Customer Service Orientation

Preferred

• Certifications

• Microsoft Azure Solutions Architect Expert (AZ-305)

• Microsoft Azure Dev Ops Engineer Expert (AZ-400)

• Terraform Associate or Professional certification

• Experience with multi-cloud environments (AWS, GCP)

• MLOps

• Experience with MLOps frameworks and tools (MLflow, Azure ML, Kubeflow

• Understanding of machine learning lifecycle and model deployment strategies

• GenAI exp. is preferred

• Knowledge of data mesh and modern data architecture patterns

• Data Technologies:
  
  Experience with SQL, No
  
  SQL databases, data warehousing, and big data technologies

• Security & Compliance:
  Strong understanding of cloud security principles and best practices

• Experience with Azure Active Directory, RBAC, and identity management

• Knowledge of data privacy regulations and compliance requirements

• Experience with security scanning and vulnerability assessment tools

• Experience with streaming data technologies (Apache Kafka, Azure Event Hubs)

• Background in financial services,…