Databricks Engineer

Recruitment Manager @ Ventures Unlimited Inc | IT Recruitment, Team Management

Client :

- TCS

Duration:

- Full Time Position

Job Description :

-

Experience Required - 8+ Years

Must Have Technical/Functional Skills

• Tenured in the fields of Computer Science/Engineering or Software Engineering.
• Bachelor's degree in computer science, or a related technical field including programming with 15+ years of demonstrated experience in software development and delivery.
• Deep hands‑on experience with Databricks and Python.
• Spark and SQL Expertise:
  Writing performant Spark SQL and PySpark transformations. Optimizing joins, window functions, and aggregations. Handling large-scale data with partitioning and caching strategies
• Experience with real‑time data processing and streaming pipelines.
• Proficient in both Delta Live Tables (DLT) and Workflows – Understanding of Lakeflow Declarative Pipelines is an asset.
• Dev Ops and CI/CD
  - Using Databricks Asset Bundles (DABs) for deployment
  - Integrating with Git, Bitbucket, and Jenkins for version control and automation
• Proven track record using dbt for building robust, testable data transformation workflows following TDD.
• Strong grasp of cloud‑based database infrastructure (AWS, Azure, or GCP).
• Skilled in developing insightful dashboards and scalable data models using Power BI.
• Expert in SQL development and performance optimization.
• Demonstrated success in building and maintaining data observability tools and frameworks.
• Proven ability to plan and execute deployments, upgrades, and migrations with minimal disruption to operations.
• Focus on standing up/adopting tools that will enable both internal and external customers. Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
• Work efficiently within a high security, PII and PCI‑DSS environments
• Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
• Support architects and engineers as they design and build effective, agile applications and platforms
• Design, develop, and optimize batch and real‑time data pipelines using Medallion Architecture, preferably on Databricks cloud data platforms
• Build data transformation workflows using dbt, with a strong focus on Test‑Driven Development (TDD) and modular design.
• Implement and manage CI/CD pipelines using Git Lab and Jenkins, enabling automated testing, deployment, monitoring of data workflows.
• Drive Data Ops practices by integrating testing, monitoring, versioning, and collaboration into every phase of the data pipeline lifecycle.
• Build scalable and reusable data models that support business analytics and dashboarding in Power BI.
• Develop and support real‑time data streaming pipelines (e.g., using Kafka, Spark Structured Streaming) for near‑instant data availability.
• Establish and implement data observability practices, including monitoring data quality, freshness, lineage, and anomaly detection across the platform.
• Plan and own deployments, migrations, and upgrades across data platforms and pipelines to minimize service impacts, including developing and executing mitigation plans.
• Collaborate with stakeholders to understand data requirements and deliver reliable, high‑impact data solutions.
• Document pipeline architecture, processes, and standards, promoting consistency and transparency across the team.
• Apply exceptional problem‑solving and analytical skills to troubleshoot complex data and system issues.
• Demonstrate excellent written and verbal communication skills when collaborating across technical and non‑technical teams.

Mid‑Senior level

Full‑time

Consulting