Databricks Engineer

Join to apply for the Databricks Engineer role at Tata Consultancy Services

Join to apply for the Databricks Engineer role at Tata Consultancy Services

Databricks Engineer

8+ Years

• Tenured in the fields of Computer Science/Engineering or Software Engineering.
• Bachelor's degree in computer science, or a related technical field including programming with 15+ years of demonstrated experience in software development and delivery.
• Deep hands‑on experience with Databricks and Python.
• Spark and SQL Expertise:
  Writing performant Spark SQL and PySpark transformations, optimizing joins, window functions, and aggregations, and handling large‑scale data with partitioning and caching strategies.
• Experience with real‑time data processing and streaming pipelines.
• Proficient in both Delta Live Tables (DLT) and Workflows; understanding of Lakeflow Declarative Pipelines is an asset.
• Dev Ops and CI/CD – using Databricks Asset Bundles (DABs) for deployment and integrating with Git, Bitbucket, and Jenkins for version control and automation.
• Proven track record using dbt for building robust, testable data transformation workflows following TDD.
• Strong grasp of cloud‑based database infrastructure (AWS, Azure, or GCP).
• Skilled in developing insightful dashboards and scalable data models using Power BI.
• Expert in SQL development and performance optimization.
• Demonstrated success in building and maintaining data observability tools and frameworks.
• Proven ability to plan and execute deployments, upgrades, and migrations with minimal disruption to operations.

• Focus on standing up/adopting tools that enable internal and external customers; key leader/contributor in the data and software, platform, and data warehouse development life cycle (design, development, documentation, testing, deployment, support).
• Work efficiently within a high‑security, PII and PCI‑DSS environment.
• Support architects and engineers as they design and build effective, agile applications and platforms.
• Design, develop, and optimize batch and real‑time data pipelines using Medallion Architecture on Databricks cloud data platforms.
• Build data transformation workflows using dbt with a strong focus on TDD and modular design.
• Implement and manage CI/CD pipelines using Git Lab and Jenkins, enabling automated testing, deployment, and monitoring of data workflows.
• Drive Data Ops practices by integrating testing, monitoring, versioning, and collaboration into every phase of the data pipeline lifecycle.
• Build scalable and reusable data models that support business analytics and dashboarding in Power BI.
• Develop and support real‑time data streaming pipelines (e.g., using Kafka, Spark Structured Streaming) for near‑instant data availability.
• Establish and implement data observability practices, including monitoring data quality, freshness, lineage, and anomaly detection across the platform.
• Plan and own deployments, migrations, and upgrades across data platforms and pipelines to minimize service impacts, including developing and executing mitigation plans.
• Collaborate with stakeholders to understand data requirements and deliver reliable, high‑impact data solutions.
• Document pipeline architecture, processes, and standards, promoting consistency and transparency across the team.
• Apply exceptional problem‑solving and analytical skills to troubleshoot complex data and system issues.
• Demonstrate excellent written and verbal communication skills when collaborating across technical and non‑technical teams.

$100,000 to $120,000 per year

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage:
  Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support:
  Maternal & Parental Leaves.
• Insurance Options:
  Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth:
  Commuter Benefits & Certification & Training Reimbursement.
• Time Off:
  Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance:
  Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Mid‑Senior level

Full‑time

Engineering and Information Technology

IT Services and IT Consulting

Referrals increase your chances of interviewing at Tata Consultancy Services by 2x

Apply BELOW