Specialist Data Engineer

Description

Job Title:

Specialist Data Engineer

Salary Range: $114,070 - $134,641

Dept/DIV:
Information Technology

Supervisor:
Director, Product Development

Location:

Various

Hours of Work: 9:00 am - 5:30 pm (7.5 hours/day) or as required

This position is eligible for teleworking, which is currently 2 days per week. New hires are eligible to apply 30 days after their effective date of hire.

The Metropolitan Transportation Authority is North America's largest transportation network, serving a population of 15.3 million people across a 5,000-square-mile travel area surrounding New York City, Long Island, southeastern New York State, and Connecticut. The MTA network comprises the nation’s largest bus fleet and more subway and commuter rail cars than all other U.S. transit systems combined. MTA strives to provide a safe and reliable commute, excellent customer service, and rewarding opportunities.

This role is responsible for the design, deployment, development and maintenance of data solutions. The Data Engineer will participate in a variety of data related projects and work closely with the Data analysts, data scientists, Business users and other stakeholders to gather requirements and build data pipelines that meet the organizations' needs. The Data Engineer will deploy and support Data platforms that process and store data.

They will contribute to software development methods, tools, and techniques and applies agreed standards and tools to achieve well-engineered outcomes.

• Design, develop, and implement data solutions to meet business requirements and data ingestion needs facilitating accurate and timely data availability for analysis and decision-making.
• Extract, load and Transform (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files.
• Write well designed, efficient code, that adheres to security standards.
• Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner to minimize disruptions in data processing for on premise and cloud environments.
• Implement data quality checks and validation processes to ensure accuracy and completeness of data. collaborate with the data team to resolve them.
• Writes complex queries and scripts to efficiently manipulate, transform, and process raw data.
• Creates and executes data validation processes to ensure the reliability and consistency of incoming data. Build processes to monitor data quality.
• Continuously optimize data pipelines for performance, scalability, and reliability.
• Create and maintain technical documentation. Contribute to building and maintaining data catalog and lineage.
• Design and develop CI/CD processes that ensure high availability and agility.
• Develop cloud data services provisioning automation with integrated capabilities of IAM, network, security policies as code, and observability.
• Build tools and services to support data discovery, lineage, resiliency, and privacy compliance across the data platform.
• Stay updated with the latest trends and best practices in data engineering, cloud computing, and Azure services to suggest innovative solutions to continually improve the organization’s data intelligence capabilities.
• Monitors and reports on supplier performance, customer satisfaction, adherence to security requirements and market intelligence.
• May mentor less experienced staff

• Bachelor’s Degree in Arts/Sciences (BA/BS). An equivalent combination of education and experience may be considered in lieu of a degree.
• Minimum 3 years of relevant experience.
• Microsoft Certified:
  Azure Data Engineer Associate is preferred.
• Microsoft Certified:
  Azure Data Fundamentals is preferred.

• Strong knowledge of Big Data architectures, large data warehouse and Data Lake solutions.
• Experience designing and implementing Modern Data and Analytics solutions including Lakehouse architecture, Medallion architecture.
• Proficient in cloud services including Azure Databricks. Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview and Power BI.
• Experience deploying and running cloud-based data…