Databricks Data Operations Specialist

Scicom Infrastructure Services is seeking a Databricks Data Operations Specialist to support the operational stability, reliability, performance, and security of a large-scale Databricks data platform.

This role will work as part of a multidisciplinary Databricks team that includes data engineers, architects, business analysts, governance specialists, and technical managers. The Data Operations Specialist will be responsible for monitoring production data pipelines, responding to incidents, troubleshooting platform and data-processing issues, coordinating deployments, and ensuring that data products meet established availability, quality, and service-level requirements.

The ideal candidate will have hands‑on experience operating Databricks environments and supporting production data workloads using Apache Spark, Delta Lake, SQL, Python, cloud infrastructure, monitoring tools, and automated deployment practices.

• Monitor the health, availability, performance, and capacity of Databricks work spaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services.
• Provide day‑to‑day operational support for production Databricks environments.
• Monitor scheduled and event‑driven data pipelines for failures, delays, data‑quality issues, and performance degradation.
• Troubleshoot Databricks jobs, Spark workloads, cluster failures, notebook errors, connectivity issues, and data‑processing exceptions.
• Restart, rerun, recover, or coordinate remediation of failed workflows according to documented procedures.
• Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch‑processing workloads.
• Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to identify root causes.
• Support configuration and maintenance of Databricks clusters, serverless compute, instance pools, cluster policies, and SQL warehouses.
• Monitor resource utilization and recommend changes to cluster sizing, autoscaling, workload scheduling, and compute policies.
• Help maintain consistent configurations across development, testing, staging, and production environments.
• Support platform upgrades, runtime‑version changes, library updates, and configuration changes.
• Coordinate with Databricks support and cloud‑service providers when vendor assistance is required.

• Monitor ETL, ELT, streaming, API, and file‑based data‑integration processes.
• Validate that scheduled data loads complete within required processing windows.
• Investigate missing, late, duplicated, incomplete, or inconsistent data.
• Perform controlled data reprocessing, backfills, reconciliation, and recovery activities.
• Support ingestion and transformation processes involving JSON, CSV, XML, Parquet, APIs, databases, message queues, and cloud storage.
• Monitor Bronze, Silver, and Gold data layers within a medallion architecture.
• Validate source‑to‑target record counts, control totals, schema consistency, and business‑rule compliance.
• Coordinate with data engineers to resolve recurring pipeline or transformation defects.
• Maintain operational runbooks for common failures, recovery procedures, escalation paths, and recurring support tasks.
• Ensure production interventions are documented and performed in accordance with change‑control procedures.

• Serve as a first or second level of support for production Databricks and data‑pipeline incidents.
• Triage incidents based on severity, business impact, affected systems, and service‑level commitments.
• Coordinate incident response among engineering, cloud, security, governance, and business teams.
• Provide timely status updates during active…