Senior Data Engineer DataSF - Office of City Administrator

DataSF is seeking a professional with 3 years of experience to join our growing team. Reporting to the Principal Data Engineer, you will be instrumental in designing, building, and maintaining the City’s data infrastructure, enabling robust data pipelines and reliable data access for analytical and operational needs.

• Platform Administration: Manage our central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization.
• Pipeline Development: Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources.
• Infrastructure as Code (IaC): Use Terraform to define, deploy, and manage data infrastructure.
• Best Practices & Innovation: Champion and implement best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing.
• Collaboration: Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders.
• Monitoring & Support: Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability.

• Education: An associate degree in computer science, computer engineering, information systems, or a closely related field (or equivalent credits).
• Experience: Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.
• Substitution: Additional experience may be substituted for the required degree on a year-for-year basis (up to a maximum of two years).

• Technical Knowledge: Experience with Snowflake/Big Query/Databricks, advanced SQL, dbt, Python (pandas, PySpark), Terraform, orchestration tools (Airflow, Dagster), and data governance.
• Collaboration and Communication
  
  Skills:
  
  Strong problem-solving, excellent communication skills, and a collaborative mindset.
• Mission Alignment: Commitment to equity, transparency, ethical data use, and public service.