Senior Big Data Engineer

Senior Big Data Engineer

Data Engineer

Owns the data pipelines that move data from operational source systems onto the data platform extraction, ingestion, transformation, orchestration, and the day-to-day operational health of those pipelines.

Also plays a role as a hands-on builder of Foundational Data Products (raw record-of-truth) and potentially Derivative Data Products (composed from upstream products).

Core skills

SQL & Python (table stakes); Scala/Java for high-throughput streaming

Pipeline build & operations: design, develop, deploy, monitor, and remediate batch and streaming pipelines that land source data on the platform; manage backfills, replays, late-arriving data, schema drift, and SLA breaches

Ingestion patterns: full-load, incremental, change-data-capture (Debezium, Fivetran, Qlik Replicate, Golden Gate), event-driven ingest, API and file-based

intake

Pipeline frameworks: dbt, Apache Spark, Apache Beam, Airflow, Dagster, Prefect

Streaming: Kafka, Kinesis, Flink, Spark Structured Streaming

Data product packaging: schema contracts (Avro/Protobuf/JSON Schema), versioning, SLAS/SLOs, data contracts, output-port design (SQL, file, API, event)

Storage formats: loeberg, Delta Lake, Hudi (open table formats are now the mesh default)

Quality & observability: Great Expectations, Soda, Monte Carlo, dbt tests, data lineage (Open Lineage)

CI/CD for data: Git Ops pipelines, unit + integration tests, environment promotion

Al-adjacent vector store ingestion (pgvector, Pinecone), feature stores (Feast, Tecton), RAG-ready chunking and embedding pipelines

Mesh-specific: knowing when to build a derivative product vs. extending an existing one; consuming upstream products through governed input ports rather than reaching into source systems