Senior Data Engineer

Main Purpose

Design, implement, test, monitor, operate and orchestrate data pipeline solutions in order to flow data from a variety of production systems into OLAP-enabled data lakes/warehouses that are both on-premises and in the cloud, in collaboration with all relevant stakeholders, ensuring data availability and reliability to enable business to analyze and discover insights.

• Define the current integration patterns.
• Harvest new pipeline patterns.
• Design, develop, test and deploy required:
• data ingestion between ODS and OLAP (Warehouse or Lake).
• pipeline between the ODS and OLAP.
• Investigate and resolve problems in the OLAP layer.

• Design and implement metadata harvesting in the OLAP layer.
• Support relevant toolsets, where required.

• Define and document pipeline patterns.
• Document and manage pipeline technical debt.

• Design, develop and test data quality rules as per the business requirements.
• Support data quality tools where required.

• Design, develop and test data controls to ensure:
• Integrity and completeness as per the non-functional and business requirements.
• Timeliness of data as per the agreed OLA and/or SLA.
• Escalation and notification procedures and solutions where control processes are not met.

• Identify technological skills weaknesses within business users on cloud and on-prem data warehouse.
• Identify and define approaches to address shortcomings for users on cloud and on-prem data warehouse.
• Plan the execution of knowledge transfer and knowledge creation initiatives on cloud and on-prem data warehouse.

• Identify topics of potential interest to the organization such as industry trends and state-of-the-art technology.
• Identify topics of interest to the organization relating to existing technology and processes
• Prepare a proposal for research work.
• Research topics, document the results and commit the output to a body of knowledge.
• Apply research findings to the analysis phase of upcoming initiatives.
• Apply research findings to ongoing implementation efforts.
• Apply research findings to planning for future initiatives.

• Analyze and interpret regulatory and compliance requirements in the form of specifications and regulation.
• Conduct impact analyses.
• Advise on impact and approach to mitigate risk Continuous improvement of internal processes.
• Timeliness of data as per the agreed OLA and/or SLA.
• Escalation and notification procedures and solutions where control processes are not met.

• Bachelor of Science in Information Systems and Computer Science
• 10 Years Relevant Experience
• Data Quality
• Data Quality Monitoring
• Data warehouse cube development
• ETL development (APS)
• Extensive experience of SQL services (SQL Server 2012 ~ 2016, Reporting services, Integration services, Analysis Services, T-SQL, D-SQL and APS)
• Power BI and BI development
• SSRS Development