Senior DataOps Engineer

Akkodis is proud to partner with an innovative automotive client who is seeking a Senior Data Ops Engineer for a contractual opportunity in the Charlotte, NC area.

Pay Range: $55/hr.

- $65/hr. (W2) - The rate may be negotiable based on experience, education, geographic location, and other factors.

The Senior Data Ops Engineer will lead the design, implementation, automation, and operational support of enterprise-scale data platforms primarily leveraging Databricks and AWS cloud services. The role will focus on building highly scalable, reliable, and governed data pipelines, optimizing Databricks platform operations, and implementing Infrastructure as Code (IaC) and Data Ops best practices across the enterprise data ecosystem.

The engineer will serve as a senior technical resource responsible for Databricks platform engineering, operational excellence, workload optimization, governance implementation, automation, and CI/CD enablement supporting analytics, AI/ML, and enterprise data initiatives.

• Lead the design, development, optimization, and operational management of enterprise-scale ETL/ELT pipelines within Databricks.
• Build and maintain scalable batch and streaming data pipelines using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
• Configure, optimize, and manage Databricks clusters for performance, scalability, reliability, and cost efficiency.
• Implement and enforce Delta Lake best practices including partitioning, schema evolution, compaction, optimization, and performance tuning.
• Administer and manage Unity Catalog, including governance policies, access controls, lineage, auditing, and data security standards.
• Design and support medallion/lakehouse architecture patterns across Bronze, Silver, and Gold data layers.
• Implement operational monitoring, observability, alerting, and troubleshooting processes for Databricks jobs, workflows, clusters, and platform services.
• Support enterprise AI/ML and analytics workloads running on Databricks.
• Develop and maintain scalable data ingestion and transformation pipelines using Python, PySpark, SQL, AWS Glue, and cloud-native AWS services.
• Integrate structured, semi-structured, unstructured, and streaming data from enterprise and cloud-based data sources.
• Implement real-time and event-driven data processing using AWS Kinesis, Firehose, and related streaming technologies.
• Collaborate with architecture, analytics, AI/ML, and platform teams to deliver enterprise-grade data solutions.

• Lead Infrastructure as Code (IaC) implementation using Terraform for provisioning and managing Databricks work spaces, clusters, jobs, permissions, and related cloud infrastructure.
• Automate environment provisioning, deployment processes, configuration management, and operational workflows.
• Implement and maintain CI/CD pipelines supporting Databricks code deployments, infrastructure automation, and platform operations.
• Maintain version-controlled repositories and Dev Ops processes supporting enterprise Data Ops practices.
• Drive platform standardization, operational governance, and deployment consistency across environments.

• Ensure compliance with enterprise data governance, privacy, security, and regulatory standards.
• Implement data quality validation, lineage tracking, auditability, and operational controls.
• Establish operational best practices, platform standards, monitoring frameworks, and support procedures.
• Provide technical leadership, mentorship, and guidance for Data Ops and Databricks engineering practices.
• Production-ready Databricks ETL/ELT pipelines and workflows.
• Optimized and governed Databricks platform environments.
• Terraform modules and Infrastructure as Code automation templates.
• Monitoring, observability, and operational dashboards for Databricks workloads and pipelines.
• Enterprise data models, lineage documentation, and operational runbooks.
• CI/CD pipelines and deployment automation frameworks.
• Weekly status reports and participation in Agile sprint ceremonies.

• 8+ years of experience in Data Engineering, Platform Engineering, or Data Ops roles.
• 5+ years of hands-on experience with Databricks in enterprise-scale environments.
• Strong expertise in PySpark, Spark SQL, Python, SQL, and distributed data processing.
• Deep hands-on experience with Delta Lake, Databricks Workflows, Unity Catalog, cluster optimization, and platform administration.
• Strong experience implementing medallion/lakehouse architectures in Databricks.
• Proven expertise with Terraform and Infrastructure as Code (IaC) automation.
• Experience implementing CI/CD pipelines and Dev Ops/Data Ops operational practices.
• Strong knowledge of AWS cloud services including AWS Glue, Kinesis, Firehose, S3, and IAM.
• Strong understanding of data governance, security, observability, and operational monitoring frameworks.
• Excellent communication, leadership, troubleshooting, and collaboration skills.

If you are interested in this Senior Data Ops…