IT Specialist (Data Ingestion & Transformation Engineer

Job Title

Data Engineer

The following are the duties of this position at the full working level. If this vacancy includes more than one grade and you are selected at a lower grade level, you will have the opportunity to learn to perform these duties and receive training to help you grow in this position.

Design and build data pipelines:
Create scalable and reliable data pipelines using tools like Apache Kafka, Apache NiFi, and cloud-based services (e.g., AWS Glue). Implement and maintain the Extract, Transform, and Load (or Load and Transform) processes to move data from source to target systems. Develops ETL pipelines in Databricks and Python. Optimizes data ingestion from multiple source systems. Design, develop and operate systems and features relating to data ingestion and transformation.

Data transformation and cleansing:
Apply advanced techniques like parsing, cleansing, normalization, and transformation to ensure data quality and consistency. Monitor and troubleshoot data pipelines to identify and fix inefficiencies, optimize performance, and ensure scalability. Designs transformation logic to ensure compliance with data model standards.

Data validation and quality:
Conduct data validation and testing to ensure accuracy, consistency, and compliance with data governance policies. Monitors data flow performance and scalability across cloud environments. Stay up to date with emerging technologies and methodologies to enhance data processing capabilities.

Cross-functional collaboration:

Working with data analysts, engineers, and scientists to understand data requirements and translate them into technical solutions; as well as to ensure the best experience for our customers. Document data processing workflows, best practices, and technical specifications for knowledge sharing and maintainability. Provide considerate and timely review of your peers' design proposals and pull requests.