Senior Data Engineer, Databricks

• Design, build, and test end-to-end data pipelines, including streaming, events, and batch data ingestion, on cloud-based infrastructure using Azure and Cosmos DB
• Develop and implement ETL processes for data ingestion, transformation, and loading into data lakes
• Work extensively on Databricks and data warehousing concepts
• Design and develop custom high-throughput and configurable frameworks and libraries
• Drive change through collaboration, influence, and demonstration of POCs
• Manage all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation
• Work collaboratively within an agile project team
• Follow best practices and coding standards
• Grow personal skillset

• Extensive technical, people and process skillsets
• Strong background in application architecture and development
• Very good understanding of leading IT technologies
• Experience in micro-services architecture and understanding of Cloud Computing;
  Azure preferred
• Proficient working knowledge of Databricks, Delta Lake, and Apache Spark
• Hands-on development experience in Databricks SQL and Scala
• Strong understanding of the Databricks platform, including clusters, jobs, and other resources
• Ability to monitor and troubleshoot data pipelines
• Experience implementing data quality checks and validations
• Experience performing data analysis and data exploration
• Strong hands-on experience troubleshooting Dev Ops pipelines, including ADO and Harness
• Experience in a multi-developer environment using version control such as Git
• Experience handling semi-structured data including Avro, JSON, and XML
• Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer
• Bachelor’s Degree level qualification in a computer or IT-related subject
• 10+ years of overall Big Data data pipeline experience
• 5+ years of hands-on Databricks experience
• 5+ years of cloud-based development experience, including Azure Services, Azure Dev Ops, Kubernetes, and Docker
• Experience with Snowflake is a plus
• Effective professional written and oral communication
• Team player with a positive attitude, enthusiasm, initiative, and self-motivation
• Ability to multitask; energetic, fast learner, and problem solver
• Experience working in the financial industry
• Experience with agile development methodology

Demonstrates expertise in designing and implementing end-to-end data pipelines using Azure, Databricks, and cloud-based infrastructure, with a strong focus on ETL processes and data quality. Proficient in application architecture, micro-services, and agile methodologies, with a proven ability to collaborate effectively within teams.

• Azure Cloud Computing
• Databricks SQL
• ETL Processes
• Data Pipeline Monitoring
• Agile Development Methodology

• Data Pipeline Development
• Databricks
• Apache Spark
• Scala
• SQL
• Data Quality Checks
• Shell Scripting
• UNIX
• Version Control (Git)
• Data Analysis

• Effective Communication
• Team Collaboration
• Problem Solving
• Self-Motivation
• Positive Attitude

• Bachelor’s Degree in Computer or IT-related Subject

• Big Data
• Financial Industry
• Agile Methodology
• Micro-Services Architecture

• Azure Dev Ops
• Kubernetes
• Docker
• Cosmos DB
• Snowflake
• TOAD
• Delta Lake
• ADO
• Harness