Senior DataOps Engineer

Job Title: Senior Data Ops Engineer

Duration: 12+ Months

Rate: $80-85/hr on Vendor W2- MAX

Location: Remote or Hybrid (Charlotte, NC)

Department: Integration, Data & AI Engineering

Seeking a Senior Data Ops Engineer to design, build, automate, and support enterprise-scale data platforms leveraging Databricks and AWS. This role is responsible for developing scalable, high-performing, and governed data pipelines while driving platform automation, Infrastructure as Code (IaC), and Data Ops best practices.

The ideal candidate will have extensive experience administering Databricks environments, optimizing platform performance, implementing CI/CD pipelines, and supporting enterprise analytics and AI/ML initiatives.

• Databricks Platform Engineering & Data Ops
• Design, build, optimize, and support enterprise-scale ETL/ELT pipelines within Databricks.
• Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
• Configure and optimize Databricks clusters for performance, scalability, reliability, and cost efficiency.
• Implement Delta Lake best practices, including partitioning, schema evolution, compaction, optimization, and performance tuning.
• Administer Unity Catalog, including governance, access controls, auditing, lineage, and security.
• Design and support Medallion (Bronze, Silver, Gold) Lakehouse architectures.
• Monitor, troubleshoot, and optimize Databricks jobs, workflows, and platform services.
• Support enterprise AI/ML and analytics workloads running within Databricks.
• Cloud Data Engineering
• Build and maintain scalable data ingestion and transformation pipelines using Python, PySpark, SQL, AWS Glue, and other AWS services.
• Integrate structured, semi-structured, unstructured, and streaming data from enterprise systems.
• Develop real-time data processing solutions using AWS Kinesis and Firehose.
• Partner with architecture, analytics, AI/ML, and engineering teams to deliver enterprise data solutions.
• Infrastructure Automation & Dev Ops
• Implement Infrastructure as Code (IaC) using Terraform to provision and manage Databricks environments and AWS infrastructure.
• Automate deployments, environment provisioning, configuration management, and operational workflows.
• Design and maintain CI/CD pipelines supporting Databricks deployments and infrastructure automation.
• Manage version control repositories and Data Ops best practices.
• Drive platform standardization and deployment consistency across development, test, and production environments.
• Governance & Operational Excellence
• Ensure compliance with enterprise security, governance, privacy, and regulatory standards.
• Implement data quality controls, lineage tracking, auditing, and operational monitoring.
• Develop operational standards, monitoring frameworks, and support procedures.
• Provide technical leadership and mentor engineers on Data Ops and Databricks best practices.

• 8+ years of experience in Data Engineering, Platform Engineering, or Data Ops.
• 5+ years of hands‑on experience with Databricks in enterprise environments.
• Strong experience with Python, PySpark, Spark SQL, and SQL.
• Deep expertise with Delta Lake, Databricks Workflows, Unity Catalog, cluster administration, and performance optimization.
• Experience designing and supporting Lakehouse/Medallion architectures.
• Proven experience with Terraform and Infrastructure as Code (IaC).
• Strong knowledge of CI/CD pipelines and Dev Ops/Data Ops methodologies.
• Experience with AWS services including AWS Glue, Kinesis, Firehose, S3, and IAM.
• Strong understanding of data governance, security, observability, and monitoring.
• Excellent communication, leadership, problem‑solving, and collaboration skills.

• Production‑ready Databricks ETL/ELT pipelines and workflows.
• Optimized and governed Databricks platform environments.
• Terraform modules and Infrastructure as Code automation.
• Monitoring and observability dashboards for Databricks workloads.
• Enterprise data models, lineage documentation, and operational runbooks.
• CI/CD pipelines and deployment automation.
• Weekly status updates and participation in Agile ceremonies.