Lead Databricks Data Engineer

Perficient is a global AI-first consultancy. Our team of strategists, designers, technologists, and engineers partners with the world's most innovative enterprises and admired brands to deliver real business results through the transformative power of AI. As part of our AI-First strategy, we empower every employee to build AI fluency and actively engage with AI tools to drive innovation and efficiency.

We break boundaries, obsess over outcomes, and shape the future for our clients. Join a company where bold ideas and brilliant minds converge to redefine what's possible - while building a career filled with growth, balance, and purpose.

Perficient currently has a career opportunity for a Lead Databricks Data Engineer to join our team located in the United States. This role is primarily office-based at a local Perficient location, with travel and on-site client engagements expected as needed.

Job Overview

As a Lead Databricks Data Engineer, you are expected to lead the development team and have strong development and programming skills in Python, SQL, Pyspark, Databricks, Notebook Repositories, Medallion Architecture, CI/CD, and ETL platforms associated with Data Lakehouse solutions. You are expected to be experienced and fluent in agile development and agile tools.

You will work with architects and infrastructure teams to develop, test, deploy and troubleshoot your code as well as provide input into solutions and design of the system. You will collaborate with some of the best talent in the industry to create and implement innovative high-quality solutions focused on our clients' business needs.

About Us

The Perficient Data & Analytics Solutions organization supports a wide variety of clients, technologies, and applications. Our Team members drive innovation in the arena of data and analytics solutions. We work on innovative digital transformation deployment for various clients across the globe. We had explosive growth in the last few years to meet the demands of a massive transition to digital with an AI First approach.

Our practice groups include Artificial Intelligence, Analytics, Content Management, Data Governance, Data Management, Data Strategy and Search.

Perficient is always looking for the best and brightest talent and we need you! We're a quickly growing, global digital consulting leader, and we're transforming the world's largest enterprises and biggest brands.

You'll work with the latest technologies, expand your skills, and become a part of our global community of talented, diverse, and knowledgeable colleagues.

RESPONSIBILITIES

• Lead the technical planning and requirements gathering.
• Serve as a technical lead and mentor. Provide technical support and leadership in the development and continual improvement of service.
• Develop and maintain effective working relationships with team members.
• Demonstrate the ability to adapt and work with team members of various experience levels.
• Work with the data engineering team to define and develop data ingestion, validation, transformation, and data engineering code.
• Lead the design, development, testing and deployment of scalable batch/streaming data pipelines using Databricks. Data jobs include data ingestion, cleansing, transformation, aggregation, governance, and sharing.
• Define and enforce best practices for Databricks usage, including notebook development, job orchestration, metadata-driven frameworks, data quality checks, and optimization of spark job performance.
• Develop Databricks platform components using Python, SQL, Pyspark, Notebook Repositories (medallion architecture L2, L3, L4), and CI/CD.
• Integrate Databricks solutions with cloud platforms like Azure (ADF, ADLS, Event Hub, Azure Dev Ops) or AWS (S3, Lamda, Glue).
• Design and implement analytical data models and star schemas for BI and AI usage.
• Collaborate with data scientists and analysts to meet data requirements.
• Implement logging, monitoring and exception handling for data workflows.
• Participate in code reviews and CI/CD pipeline development.
• Document code artifacts and participate in developing user documentation and run books
• Troubleshoot deployment to various environments and provide test support.
• Participate in design sessions, demos, and prototype sessions, testing and training workshops with business users and IT associates.

QUALIFICATIONS

• Passionate developer with 8+ years of data engineering experience, with at least 3 years in a lead or senior data engineer role.
• Minimum 5 years of hands-on Databricks experience using Python, SQL, Pyspark, Notebook Repositories (medallion architecture L2, L3, L4), CI/CD.
• Minimum 5 years of Databricks experience in developing large scale data processing/data storage/data distribution systems.
• Experience with AWS and/or Azure cloud environment.
• Experience with various Notebook Repositories (medallion architecture L2, L3, L4) as well as unit testing frameworks.
• Dev Ops experience with Databricks CI/CD (Databricks Asset Bundles) and Git
• Minimum 5 years of working with large…