Sr. Scala Developer TECHM

Job Title

Python, Spark, AWS, Scala, Terraform Responsibilities

Red Processing:
Our data ingest pipelines, it handles parsing, renaming, encryption of PII and writing of our 20 ingests. These are used to support and help our organization as a whole.

Kiwi:
Our automated documentation application which parses other team's table metadata (as well as our own), formats it and sent it to an external team to display.

Soteria:
Our configuration-based encryption job which other teams use to encrypt and format their PII on ingest.

Our monitoring:
Currently this is a series of jobs that notify us and our stakeholders when our ingests are late, checks for data quality, monitors our daily ingest durations per job and also serves as part of our release test. We're hoping to generalize this and make it available to other teams.

On top of those, we also own a common library for AWS utilities/APIs, Spark utilities or general Scala utilities for the Scala teams inside of our organization.

Tech stack Scala Spark AWS (EMR, Lambda, S3, Deequ, SES, Glue) Python Terraform

Things we are looking for:
Experienced with object-oriented development with Scala and knowledge of functional programming paradigms. Nice to have experience with some or all of the following:
Spark or SQL, AWS (EMR /S3/Lambda), Terraform and Python. Experienced with implementing and maintaining continuous integration and delivery pipelines (Gitlab CI/CD) to build, test, and deploy artifacts. Believe in optimization, efficiency and thorough testing. Believe in incremental delivery and able to take complex business requirements and write software to achieve those requirements.