Senior Data Engineer

Requisition

Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture.

The Wealth Data engineering team within the Global Wealth Engineering (GWE) is the key team in meeting the operational data needs of the various stake holders within Wealth Management. The Lead Data Engineer will play a key role in designing and implementing data solutions using Big Data, Hadoop (Cloudera) and Google cloud working closely with the enterprise data team and data architects, solution architects, business systems analyst and data engineers

Is this role right for you? In this role you will:

• Leading development efforts in ingesting and transforming data from various sources. Working in the Big Data/ Hadoop environment, should be hands on in writing code, building scripts, writing specifications and responsible for end to end delivery of data in the Enterprise Data Lake environment
• Build distributed, reliable and scalable data pipelines to ingest and process data from multiple data sources
• Designing, building, operationalizing the data platform using Google Cloud Platform (GCP) data services such as Data Proc, Dataflow, CloudSQL, Big Query, Cloud Spanner in combination with third parties such as Spark, Apache Beam/ composer, DBT, Cloud Pub Sub, Confluent Kafka, Cloud storage Cloud Functions & Github
• Designing and implementing data ingestion patterns that will support batch, streaming and API interface on both the Ingress and Egress.
• Guide a team of data engineers and work hands on in developing framework and custom code using best practices that will meet the demanding performance requirements
• Take a lead in designing and building production data pipelines from data ingestion to consumption using GCP services, Java, Python, Scala, Big Query, DBT, SQL etc.
• Experience using Cloud Dataflow using Java/Python for deploying streaming jobs in GCP as well as batch jobs using text/JSON files and writing them to Big Query
• Building and managing data pipelines with a deep understanding of workflow orchestration, task scheduling and dependency management
• Provide end-to-end technical guidance and expertise on how to effectively use Google Cloud to build solutions; creatively applying cloud infrastructure and platform services to help solve business problems; and communicating these approaches to different business users
• Provide guidance on Implementing application logging, notification, jobs monitoring and performance monitoring

Do you have the skills that will enable you to succeed in this role? We’d love to work with you if you have:

• 10+ years of experience in data engineering, performance optimization for large OLTP applications
• Expertise knowledge of Hadoop HDFS, Hive, Pig, Flume and Sqoop.
• Experience with the primary managed data services within GCP, including Data Proc, Dataflow, Big Query/DBT, Cloud Spanner, Cloud SQL, Cloud Pub/Sub etc.
• Experience with Google Cloud Platform Databases (SQL, Spanner, Postgre
  
  SQL)
• Working experience in HQL
• Good knowledge of the concepts of Hadoop.
• Experience working with relational/No
  
  SQL databases
• Experience with data streaming and technologies such as Kafka, Spark-streaming etc.
• Experience with Infrastructure as Code (IaC) practices and frameworks like Terraform
• Knowledge of Java microservices and Spring Boot
• Strong architecture knowledge with experience in providing technical solutions for cloud infrastructure.
• Working knowledge of developing and scaling JAVA REST services, using frameworks such as Spring
• Good communication and problem-solving skills. Ability to effectively convey ideas to business and technical teams

Nice-To-Have Skills:

• Understanding of Wealth business line and the various data domains required for building an end to end solution

What's in it for you?

• Diversity, Equity, Inclusion & Allyship - We strive to create an inclusive culture where every employee is empowered to reach their fullest potential, respected for who they are, and are embraced through bias-free practices and inclusive values across Scotiabank. We embrace diversity and provide opportunities for all employee to learn, grow & participate through our various Employee Resource Groups…