Principal Data Engineer Job Nashville area,Tennessee USA,IT/Tech

Does this position require a security clearance? No
Years 10+ years
Applicants are required to read, write, and speak the following languages English

Job Description

Provides expertise on the design and participates in building of data infrastructure to optimize data processing from a variety of data sources. Independently designs and implements data governance policies and procedures for data handling to manage data consistency, integrity, accuracy, and reliability. Provides expertise on and participates in the design and implementation of rigorous data validation and integrity checks, proactively mitigating data quality issues that could impact data pipeline and model performance.

Leverages advanced knowledge of Extract, Transform, and Load (ETL) processes to design, develop, and optimize automated, scalable, and efficient data pipeline architectures to build reusable data products. Works independently and collaboratively in an agile development environment with other engineers to develop, maintain, and debug advanced data solutions that are scalable, efficient, cost effective, and reliable.

Responsibilities

Key Responsibilities Data Processing & Pipelining – Data Requirements, Collection, and

Infrastructure:

Mentors less experienced team members to identify data requirements and business objectives of a project or initiative.

Provides expertise on the design and participates in building of data infrastructure to optimize data processing from a variety of data sources.

Independently analyzes, designs, and troubleshoots data flows based on business needs.

Participates and designs architecture, performance, and security reviews of the technical solution.

Adjusts data collection processes that involve indexing and query optimizations for optimal performance.

Builds Extract, Transform, and Load (ETL) pipelines to support efficient and scalable data collection and extraction.

Engages with and holds the upstream and downstream teams accountable for the predefined service level agreements (SLAs).

Manages relationships with the data providers.

Independently designs and implements data governance policies and procedures for data handling (e.g., data retention) to manage data consistency, integrity, accuracy, and reliability throughout the data lifecycle.

Leads the execution of redaction processes for Personally Identifiable Information (PII) and Protected Health Information (PHI) data, ensuring compliance with data privacy and security standards.

Ensures minimal data collection and usage in accordance with data minimization principles.

Follows data security measures to protect data from unauthorized access, use, disclosure, alteration, or destruction, proactively identifying and escalating potential issues.

Ensures data compliance with relevant laws, regulations, and industry standards.

Data Processing & Pipelining – Data Validation & Quality Assurance:

Provides expertise on and participates in the design and implementation of rigorous data validation and integrity checks, proactively mitigating data quality issues that could impact data pipeline and model performance.

Mentors less experienced team members to define data annotation and labeling processes to ensure data quality.

Identifies opportunities for automation of data validation and governance, and implements them.

Independently corrects deviations and non-conformance when identified.

Data Pipeline and Solutions Engineering – Pipeline Design:

Leverages advanced knowledge of ETL processes to design, develop, and optimize automated, scalable, and efficient data pipeline architectures to build reusable data products.

Implements advanced data storage solutions to store the processed data to be used in a scalable, optimized, and efficient way for access and analysis.

Mentors less experienced team members to manage the flow of data pipeline and storage day-to-day operations.

Data Pipeline and Solutions Engineering – Data Solutions Engineering:

Works independently and collaboratively in an agile development environment with other engineers to develop, maintain, and debug advanced data solutions that are scalable, efficient, cost effective, and reliable.

Reviews runnable…