Lead Databricks Data Engineer

Toronto (Vaughan) - Hybrid 3 days per week in client office

Lead Data Engineer – Databricks

Toronto (Vaughan) - Hybrid 3 days per week in client office

KData is a data and AI consulting and technology company helping organizations design, build, and scale modern data platforms. We partner with our clients to deliver high-performance analytics, cloud data platforms, and AI-ready architectures using best-in-class technologies such as Databricks and Azure. At KData, you will work on impactful projects in a collaborative, engineering-first culture that values quality, innovation, and continuous learning.

As a Data Engineer – Databricks, you will lead a team and drive the approach that uses intelligent code assistants for efficiencies. You will be responsible to design, build, and operate scalable and reliable data pipelines and data products on cloud platforms. You will work closely with data architects, analytics teams, and business stakeholders to deliver high-quality datasets that support analytics, BI, and advanced data use cases.

This role requires leadership, strong hands‑on Databricks experience, and the ability to work on‑site in Vaughan.

• Design, develop, and maintain scalable data pipelines and data products using Databricks on cloud platforms (AWS and/or Azure).
• Implement data ingestion, transformation, and curation pipelines using Apache Spark (PySpark preferred).
• Build and manage Delta Lake architectures (Bronze, Silver, Gold layers).
• Ensure data quality through automated controls, validation checks, and robust error handling.
• Optimize workloads for performance, reliability, and cost efficiency (Spark tuning, cluster configuration).
• Implement monitoring, logging, and alerting to ensure stable and predictable operations.
• Apply software engineering best practices: version control, CI/CD pipelines, automated testing, and documentation.
• Collaborate with data architects, analytics, BI, and business teams to translate requirements into technical solutions.
• Participate in production support, incident analysis, and continuous improvement initiatives.
• Contribute to data platform evolution, standards, and best practices within KData.

• 8+ years of experience in data engineering roles delivering production‑grade data solutions.
• Team leadership
• Strong hands‑on experience with Databricks in real‑world projects.
• Solid expertise in Apache Spark, with PySpark as a strong preference.
• Strong SQL skills with a focus on performance and data correctness.
• Good understanding of Delta Lake concepts (ACID transactions, schema evolution, time travel).
• Experience with cloud platforms (AWS and/or Azure), including storage and IAM concepts.
• Familiarity with CI/CD, automated testing, and version‑controlled deployments.
• Experience working with data orchestration and monitoring tools is a plus.

• Fluent English (mandatory) – professional working proficiency required

• Experience with data architecture patterns (Lakehouse, Medallion architecture).
• Exposure to data governance, data quality frameworks, and security best practices.
• Experience using AI‑assisted development tools (e.g., Git Hub Copilot, Databricks Assistant).
• Consulting or client‑facing experience.