Sr. Data Operations Engineer

Summary

The Data Ops Engineer is responsible for improving, modernizing, and supporting data ingestion operations with a focus on building reliable, scalable, and resilient data pipelines. This role drives enhancements to existing ingestion processes through automation, monitoring, containerization, data quality controls, and operational best practices to improve reliability, reduce manual intervention, and ensure continuity of data operations. The Data Ops Engineer monitors pipeline health, troubleshoots production data issues, supports workflow orchestration and deployment, and maintains operational documentation to ensure trusted, timely delivery of data for analytics, reporting, business operations, and machine learning initiatives.

Essential Duties and Responsibilities

This list of duties and responsibilities is not all inclusive and may be expanded to include other duties and responsibilities as management may deem necessary from time to time.

• Design, operate, and maintain reliable batch and streaming data pipelines that support analytics, reporting, business operations, and machine learning.
• Modernize existing ingestion workflows by identifying opportunities to containerize solutions for scalability and resiliency.
• Monitor pipeline health, job performance, data freshness, data availability, and SLA adherence using observability, logging, and alerting tools.
• Troubleshoot production data incidents, identify root causes, coordinate remediation, and document preventive actions through runbooks and post-incident reviews.
• Implement data quality checks, schema validation, anomaly detection, and reconciliation processes to improve downstream dataset and dashboard reliability.
• Support orchestration, scheduling, deployment, and lifecycle management for data workflows using tools such as Azure Data Factory, Azure Synapse, Azure Databricks, cloud-native data services, or comparable platforms.
• Collaborate with data engineers, business stakeholders, product owners, data analysts, and data architects to understand requirements and resolve operational issues.
• Improve reliability, scalability, performance, and cost efficiency of data platforms, warehouses, lakes, and integration processes.
• Maintain documentation for data workflows, operational procedures, dependencies, lineage, ownership, escalation paths, and support processes.
• Establish or contribute to CI/CD, infrastructure-as-code, environment management, access controls, and governance practices for production data systems.
• Participate in on-call or production support rotations as needed to support timely response to critical data issues.
• Support AI and machine learning initiatives by maintaining reliable, secure, and well-governed data pipelines and datasets used for model development, testing, deployment, monitoring, and ongoing performance evaluation.
• Maintain confidentiality of information processed & follow company policies and procedures.

Qualifications

Requires five (5) to seven (7) years of data engineering, data operations, data platform operations, or related experience. Bachelor’s degree in Computer Science, Engineering, Information Systems, or related field, or equivalent professional experience preferred. Experience leading teams and developing supervisory staff preferred.

To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodation may be made to enable individuals with disabilities to perform the essential functions.

Other qualifications include:

• Strong SQL skills and experience working with relational databases, data warehouses, and lakehouse environments.
• Hands-on experience with ETL/ELT pipelines, workflow orchestration, data transformation, and production data operations.
• Working knowledge of Microsoft Azure or other cloud platforms, Linux or command-line tools, scripting languages such as Python or Bash, and version control systems such as Git.
• Familiarity with Azure tools and platforms such as Azure Databricks, Azure Data Factory, Azure Container Apps, Azure Synapse, Azure Event Hubs, and Service Bus.
• Experience supporting data pipelines, datasets, or platform operations used for AI and machine learning workloads, with an understanding of data quality, observability, governance, privacy, and responsible AI considerations.
• Understanding of monitoring, alerting, logging, incident response, root cause analysis, and operational runbook development.
• Experience with data quality, schema management, metadata, lineage, governance, security, and access-control concepts.
• Ability to analyze pipeline performance, optimize queries, manage dependencies, and improve cost-performance of data workloads.
• Clear communication skills for translating technical issues into business impact and coordinating with cross-functional teams.
• Strong problem-solving mindset, attention to detail, ownership, and comfort working in…