Senior Manager, Data Engineering

At PDI Technologies, we empower some of the world's leading convenience retail and petroleum brands with cutting‑edge technology solutions that drive growth and operational efficiency.

By “Connecting Convenience” across the globe, we empower businesses to increase productivity, make more informed decisions, and engage faster with customers through loyalty programs, shopper insights, and unmatched real‑time market intelligence via mobile applications, such as Gas Buddy. We’re a global team committed to excellence, collaboration, and driving real impact. Explore our opportunities and become part of a company that values diversity, integrity, and growth.

You will own how data gets into our platform and how it gets served back out — ingestion, the lakehouse, and the query layer underneath everything analytics and product depend on.

The problem is specific. Data arrives from CDC streams, transactional databases, event topics, partner APIs, and files, and today each source carries its own pipeline, its own failure modes, and its own on‑call story. Your mandate is to collapse that into one ingestion framework and one open lakehouse — reliable enough to publish SLOs against, fast enough to serve interactive query, and cheap enough to defend line by line.

This is a hands‑on role. You will set technical direction, hire, and grow the team — and you will also be in design reviews, in code review, and in the pipeline when a stateful stream will not recover from its checkpoint. Expect roughly half your time in technical work.

You own the full path: how data lands, the table format and its lifecycle, the transformation layer, the engines that serve it, and the SLOs on top of all of it. When a dataset is late or wrong, it is your team's phone that rings — and you are expected to have already built the thing that catches it first.

• Lead, hire, and grow a team of 10+ data engineers — set the technical bar through design and code review, not through status meetings.
• Own the architecture and delivery of a unified ingestion framework: one configuration‑driven path for batch, CDC, and streaming sources, with schema evolution, replay and backfill, idempotency, dead‑letter handling, and data contracts built into the framework rather than reimplemented per pipeline.
• Own production Spark Structured Streaming pipelines — watermarking, stateful joins and aggregations, checkpoint and restart discipline, exactly‑once sinks, lag and back pressure management.
• Own the Apache Iceberg lakehouse: partition and sort strategy, file sizing and compaction, snapshot and orphan‑file lifecycle, schema and partition evolution, and multi‑engine interoperability.
• Set the dbt modeling standard — layering conventions, tests, contracts, CI enforcement, and lineage that stakeholders trust.
• Own Trino catalog design, workload isolation, and query performance for interactive and federated access.
• Define and meet freshness, completeness, and latency SLOs. Run a 24x7 on‑call rotation with a short mean time to restore.
• Own cost: a defensible cost‑per‑pipeline and cost‑per‑dataset number, and the levers to move it.
• Partner with product, analytics, and architecture to sequence the roadmap, and bring rigor to decisions — collect the data, seek dissent, and run pilots rather than arguing from opinion.

• Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.
• 8+ years in data engineering, including 3+ years leading engineers as a manager or tech lead — and you are still hands‑on in code and design.
• Production experience with Spark Structured Streaming at scale: state store growth, checkpoint recovery, watermark tuning, and late or out‑of‑order data.
• Deep Apache Spark and PySpark performance work — diagnosing and fixing skew, shuffle pressure, small‑file problems, and executor memory failures on real workloads.
• Experience building or substantially owning a reusable ingestion framework serving multiple source types — not a collection of individual pipelines.
• Production experience with an open table format (Apache Iceberg preferred) including schema and partition evolution, compaction…