Dataops​/Cloud Data Engineer - Senior - Softline Technology

Description

• Designing, developing, and optimizing Azure Data Factory and Databricks pipelines from Oracle databases to Lake houses
• Designing and optimizing data models using Medallion Architecture principles
• Designing, developing and optimizing data connections from Databricks Medallion Architecture to on-premises data sources for downstream consumers.
• Designing and optimizing relational data models
• Translating Informatica ETL to Azure Data Factory and Databricks ELT

• Transfer From Data Ops/Cloud Data Engineer to Designated CSC Resource

• Knowledge transfer must be completed one week prior to the end of the project or one week prior to the consultant leaving the ministry.

• All deliverables, including design/supporting/release/training documents must be checked into designated version control repositories (for example, git repositories or SharePoint). All final documents and working drafts related to project requirements or solution design must be stored on designated project repositories (for example, SharePoint site, HPQC, TFS, etc.)
• Project manager and designated ministry staff must be regularly informed in writing (by email) of where documentation has been stored and must be provided a minimum of one walk-through of all documentation as part of the final knowledge transfer activities.

• Knowledge will be transferred through 1 on 1 sessions, emails, document updates and document review with the team.

?

Note:

This position will require the consultant to work from the office location 5 days per week.

• Experience developing ETL processes with Informatica, Azure Data Factory and Databricks (Python, Lakeflow, and SQL Optimization)
• Experience with solution development with Databricks in Medallion Architecture.
• Experience with Cloud data platforms, data management and data exchange tools and technologies
• Experience with Commercial and opensource data and database development and management and specializing in data storage setting (Delta Lake) and managing cloud Data As a Service (DaaS). application Database as a Service (DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms (both in the cloud and on-premise).
• Experience with Data pipeline and workflow development, orchestration, deployment, and automation and specializing in programming and pipelines to create and managing the dataflow, Delta Lake indexing, parallelism and movement of data.