Senior Big Data Engineer

You will

• Design, build, and optimize scalable data processing pipelines using Spark, Airflow, and related big data technologies to support both batch and real‑time use cases.
• Lead and collaborate with technical, application, and security stakeholders to deliver reliable, secure Big Data infrastructure leveraging tools and platforms such as Spark, Dataproc, Redpanda, and Temporal.
• Own and participate in on‑call responsibilities for Big Data platforms, triaging and resolving incidents, responding to tickets, and ensuring systems consistently meet defined SLAs for availability, performance, and data quality.
• Design, develop, and automate large‑scale infrastructure on Kubernetes using Terraform and other infrastructure‑as‑code patterns to support high‑performance data processing, analytics, and AI/ML workloads.
• Define and implement monitoring, alerting, and runbooks to provide end‑to‑end observability and drive continuous improvement in reliability and operational excellence using tools like Grafana.
• Onboard, train, and mentor vendor teams and external partners so they can effectively support, operate, and extend the solutions owned by the Big Data Infrastructure (BDI) team.
• Drive the technical roadmap for emerging data and infrastructure technologies by evaluating options, building proofs of concept (POCs), and authoring solution selection and design documents.
• Champion engineering best practices (code reviews, testing strategies, CI/CD, security and compliance standards) across the Big Data ecosystem to ensure maintainable, resilient, and cost‑efficient systems.

• Build and operate the core Big Data Infrastructure (BDI) platform that powers large‑scale data processing, analytics, and AI/ML workloads across Live Ramp using tools like Dataproc, Redpanda, Airflow and Temporal.
• Provide secure, self‑service, and cost‑efficient data environments that enable product and data teams to experiment, ship, and scale data collaboration applications quickly.
• Partner closely with application, security, SRE, and platform teams to ensure data systems meet Live Ramp’s standards for privacy, compliance, reliability, and performance.
• Evolve common patterns, tooling, and reference architectures that reduce friction for teams adopting new big data, streaming, and AI/ML capabilities.
• Continuously improve platform reliability and developer experience by automating operations, simplifying workflows, and incorporating feedback from internal customers.

• 7+ years of experience in software engineering or data engineering, with a focus on large‑scale data processing and distributed systems.
• Hands‑on experience building and operating pipelines with Apache Spark (batch and/or streaming) and at least one orchestration framework such as Airflow or Temporal.
• Practical experience running Big Data workloads on cloud platforms (for example, using Dataproc or similar managed compute services).
• Proficiency with Kubernetes and infrastructure‑as‑code tools such as Terraform to provision, configure, and manage production services.
• Experience with modern streaming and messaging technologies (for example, Redpanda, Kafka, or similar) and real‑time data processing patterns.
• Solid understanding of systems design, scalability, reliability, and observability for data‑intensive workloads.
• Experience participating in or leading on‑call rotations and incident management processes for production systems.
• Strong collaboration and communication skills, including working with cross‑functional partners (security, application owners, vendors) and documenting designs and decisions clearly.
• Ability to evaluate new technologies, build POCs, and make data‑informed recommendations that influence team and platform roadmaps.
• Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related technical field, or equivalent practical experience.

• Experience working on Big Data Infrastructure, data platform, or core infrastructure teams in a cloud‑native environment.
• 3 years of Software Development experience in one of the languages - Python, Go, Java or Scala.
• 3 years of experience with Linux Operating Systems.
• 3 years of experience in…