Senior Data Engineer

Job Description

Designs and builds proper data pipelines required for optimal data collection and processing from a variety of data sources. Implements data governance policies and procedures for data handling to manage data consistency, integrity, accuracy, security, and reliability. Implements rigorous data validation and integrity checks to support data pipeline and model performance. Designs, develops, and optimizes automated and scalable data pipeline architectures using Extract, Transform, and Load (ETL) processes to build reusable data products, independently.

Collaborates in an agile development environment to develop, maintain, and debug data solutions that are scalable, efficient, and reliable.

Designs and builds proper data pipelines required for optimal data collection and processing from a variety of data sources. Implements data governance policies and procedures for data handling to manage data consistency, integrity, accuracy, security, and reliability. Implements rigorous data validation and integrity checks to support data pipeline and model performance. Designs, develops, and optimizes automated and scalable data pipeline architectures using Extract, Transform, and Load (ETL) processes to build reusable data products, independently.

Collaborates in an agile development environment to develop, maintain, and debug data solutions that are scalable, efficient, and reliable.

• Identifies data requirements and business objectives of a project or initiative in collaboration with cross-functional teams.
• Designs and builds proper data pipelines required for optimal data processing from a variety of data sources.
• Independently analyzes, designs, and troubleshoots data flows based on business needs.
• Analyzes business requirements and translates that into a technical specification.
• Adjusts data collection processes that involve indexing and query optimizations, for optimal performance.
• Builds Extract, Transform, and Load (ETL) pipelines to support efficient data collection and extraction, independently.
• Analyzes data sources for profiling the data to ensure pipeline build success.
• Defines success and failure thresholds for data collection pipelines.

• Implements data governance policies and procedures for data handling (e.g., data retention) to manage data consistency, integrity, accuracy, and reliability throughout the data lifecycle.
• Redacts Personally Identifiable Information (PII) and Protected Health Information (PHI) data to ensure compliance with data privacy and security standards.
• Follows data security measures to protect data from unauthorized access, use, disclosure, alteration, or destruction.
• Ensures data compliance with relevant laws, regulation, and industry standards.

• Implements rigorous data validation and integrity checks, identifying and addressing any data quality issues that could impact data pipeline and model performance.
• Defines data annotation and labeling processes to ensure data quality, independently.
• Designs and implements automation of data validation and governance.

• Independently designs, develops, and optimizes automated and scalable data pipeline architectures using ETL processes to build reusable data products.
• Implements appropriate data storage solutions to store the processed data to be used in a scalable and optimized way for access and analysis.
• Manages the flow of data pipeline and storage day-to-day operations.

• Works independently and collaboratively in an agile development environment with other engineers to develop, maintain, and debug data solutions that are scalable, efficient, cost effective, and reliable.
• Writes runnable code and performs testing and debugging of data solutions, independently.

• Independently manages work, monitoring timelines and deliverables to ensure…