Databricks Architect
Listed on 2026-10-22
-
IT/Tech
Data Engineering, Cloud Computing: Infrastructure & Operations
Welcome to Sky Lambda — Powering Scalable Innovation in the Cloud
ResponsibilitiesResponsibilities:
Design and implement scalable data architectures using the Databricks Lakehouse Platform.
Develop and optimize data pipelines using Apache Spark, PySpark, SQL, and Delta Lake.
Architect and implement ETL/ELT workflows for batch and real-time data processing.
Design and manage data lakehouse solutions, including data ingestion, transformation, storage, and consumption layers.
Implement data governance, security, access controls, data quality, and metadata management using Unity Catalog and related tools.
Optimize Spark jobs, cluster configurations, query performance, and compute costs.
Integrate Databricks with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
Design and implement workflow orchestration and scheduling using Databricks Workflows and tools such as Apache Airflow.
Establish CI/CD pipelines, infrastructure automation, and deployment best practices using tools such as Git, Terraform, and cloud-native Dev Ops services.
Implement monitoring, logging, error handling, and performance optimization for production data pipelines.
Collaborate with data engineers, analysts, data scientists, and business teams to define technical requirements and deliver data solutions.
Provide technical leadership, conduct code reviews, establish architecture standards, and mentor engineering teams.
Ensure compliance with enterprise data security, privacy, governance, and regulatory requirements.
Evaluate emerging data technologies and recommend improvements to the existing data platform.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).