Data Engineer

Job Details

Data Engineer - Hybrid (Various locations in USA)

Location:

San Francisco, CA

Job

Category:
Information Technology

Position Type:
Contract

Duration:
Long Term

Remaining Positions: 1

The Common Data Platform (CDP) is an exciting new, multi-district program to create a cloud-based, end-to-end data management platform to reduce data cost and improve user experience. The CDP program team uses the Scaled Agile Framework (SAFe) to deliver incremental business value. As a Data Engineer, this contingent worker will collect, parse, manage, analyze, and visualize large sets of data to transform information into actionable insights.

They will work across multiple platforms to ensure that data pipelines are scalable, repeatable, and secure, capable of serving multiple users.

• Design, develop, and maintain robust and efficient data pipelines to ingest, transform, catalog, and deliver curated, trusted, and quality data from disparate sources into our Common Data Platform.
• Actively participate in Agile rituals and follow Scaled Agile processes as set forth by the CDP Program team.
• Deliver high-quality data products and services following SAFe Agile Practices.
• Proactively identify and resolve issues with data pipelines and analytical data stores.
• Deploy monitoring and alerting for data pipelines and data stores, implementing auto‑remediation where possible to ensure system availability and reliability.
• Employ a security‑first, testing, and automation strategy, adhering to data engineering best practices.
• Collaborate with cross‑functional teams, including product management, data scientists, analysts, and business stakeholders, to understand their data requirements and provide them with the necessary infrastructure and tools.
• Keep up with the latest trends and technologies, evaluating and recommending new tools, frameworks, and technologies to improve data engineering processes and efficiencies.

• Bachelor's degree in Computer Science, Information Systems, or a related field, or equivalent experience.
• 2+ years’ experience with tools such as Databricks, Collibra, and Starburst.
• 3+ years’ experience with Python and PySpark.
• Experience using Jupyter notebooks, including coding and unit testing.
• Recent accomplishments working with relational and No
  
  SQL data stores, methods, and approaches (STAR, Dimensional Modeling).
• 2+ years of experience with a modern data stack (object stores like S3, Spark, Airflow, lakehouse architectures, real‑time databases) and cloud data warehouses such as Red Shift, Snowflake.
• Overall data engineering experience across traditional ETL & Big Data, either on‑prem or Cloud.
• Data engineering experience in AWS (any CFS2/EDS) highlighting the services/tools used.
• Experience building end‑to‑end data pipelines to ingest and process unstructured and semi‑structured data using Spark architecture.

Pay Range: $107.00 - $113.33 per hour.

W2 Candidates Only!

For quick apply, please contact Ayush Dwivedi;
Ph:  or