Data Engineer II

Data Engineer II Job Details

Data Engineer II is an experienced data professional responsible for designing, building, and maintaining robust data pipelines and infrastructure that enable the collection, storage, and processing of large datasets. This role expands upon the Data Engineer I position by handling more complex data projects and working with greater independence. A Data Engineer II ensures data is accurate, secure, and compliant with data governance standards.

A Data Engineer II collaborates with cross‑functional teams (e.g., business stakeholders, IT, and subject‑matter experts) to deliver solutions that meet business and research needs.

• Maintains and optimizes data pipeline architecture by designing, building, and managing ETL processes that extract, transform, and load data from diverse sources. Assembles large, complex data sets to meet both functional and non-functional requirements, and develops scalable architectures for structured and unstructured data.
• Integrates and consolidates data from multiple systems—such as disparate databases and electronic health records—into unified repositories like data warehouses or data lakes. Develops and enhances the underlying data infrastructure using SQL and cloud technologies to ensure scalability and reliability.
• Creates and supports analytics tools that empower analysts and data scientists to access and analyze data efficiently. Builds custom queries, scripts, and dashboards that enable insight generation and data product optimization. Collaborates with analytics experts to organize, query, and visualize data for reporting and research.
• Identifies and implements process improvements to enhance data operations. Automates manual workflows, optimize data delivery pipelines, and redesign system architecture to support scalability and performance. Continuously evaluates workflows and technologies to recommend improvements that accommodate growing data complexity.
• Ensures data governance and security by validating data for accuracy and consistency, and maintaining secure, compliant data environments. Follows best practices and regulatory standards (e.g., HIPAA) to protect sensitive information and uphold data integrity.
• Collaborates with stakeholders across departments—including executives, product managers, researchers, and designers—to address data infrastructure needs and resolve technical issues. Translates non-technical requirements into effective data solutions and advises on best practices for data architecture.
• Manages and executes data projects from planning through deployment. Applies light project management techniques to coordinate tasks, communicates with team members, and ensures timely delivery. Exercises independent judgment to overcome obstacles and align project outcomes with organizational goals.

• Adheres to internal controls and reporting structure.
• Performs related duties as required.

• Systems Knowledge:
  Broad understanding of system-level concepts in computing. Includes knowledge of programming and scripting, operating systems, database query languages (SQL) and data mining techniques, as well as familiarity with IT infrastructure (servers, networking, cloud services). Such knowledge enables the Data Engineer II to troubleshoot and optimize across the technology stack.
• Big Data Processing:
  Proficiency with big data frameworks such as Apache Spark for distributed data processing and large‑scale computations. Experience optimizing Spark jobs for performance is often required.
• Workflow Orchestration:
  Experience with workflow orchestration tools like Apache Airflow (or similar platforms) to schedule and manage complex data pipelines. Ability to design reliable job workflows and handle dependencies between tasks.
• Programming & Databases:
  Strong programming skills in Python (especially using PySpark) and solid knowledge of SQL for querying and manipulating data. Familiarity with working in both relational databases (SQL) and No
  
  SQL databases, with the ability to design and optimize database schemas and queries for each.
• Version Control:
  Experience using Git or…