Big Data Platform Engineer

We are seeking an experienced Big Data Platform Engineer to design, develop, and optimize large-scale data processing platforms and cloud-native data solutions. The ideal candidate will have deep expertise in Apache Spark, Kubernetes, AWS, and distributed data processing technologies. This role focuses on building scalable data pipelines, architecting Kubernetes infrastructure, optimizing big data workloads, and supporting enterprise data platforms through modern software engineering and Dev Ops practices.

• Design, develop, and maintain large-scale data processing pipelines using Apache Spark, Hadoop, Python, and Scala.
• Architect, deploy, and optimize containerized big data workloads using Amazon EMR on EKS.
• Design and build Kubernetes infrastructure to support scalable Spark applications and enterprise data platforms.
• Develop scalable solutions for data ingestion, storage, transformation, and analytics.
• Optimize existing data pipelines for performance, scalability, reliability, and cost efficiency.
• Monitor, troubleshoot, and resolve production issues affecting data pipelines and big data platforms.
• Develop automated testing frameworks and implement continuous testing to ensure data quality and platform reliability.
• Create and maintain unit, integration, and end-to-end tests for data processing applications.
• Manage Kubernetes clusters, pods, deployments, services, name spaces, Config Maps, Secrets, and storage resources.
• Collaborate with architects, data scientists, analysts, and engineering teams to deliver enterprise data solutions.
• Research and evaluate emerging Big Data, cloud, and AI technologies to improve platform capabilities.
• Participate in architecture discussions and contribute to technical design decisions.

• Bachelor's degree in Computer Science, Information Systems, or a related field, or equivalent professional experience.
• Minimum of 5 years of experience designing and developing enterprise Big Data solutions.
• Strong experience building Kubernetes infrastructure for enterprise applications.
• Hands‑on experience with Kubernetes architecture, including pods, services, deployments, name spaces, Config Maps, Secrets, networking, storage, and security.
• Experience running Apache Spark workloads on Kubernetes, including Amazon EMR on EKS.
• Strong experience with Apache Spark, including Spark architecture, performance tuning, partitioning, caching, broadcast joins, DAG execution, executors, and resource optimization.
• Experience designing and supporting large‑scale data pipelines processing massive datasets.
• Experience with Hadoop, Spark, Hive, and Trino.
• Experience troubleshooting data skew, resource constraints, scalability issues, and Spark job failures.
• Strong experience with AWS services, including Amazon S3, EMR, EMR on EKS, Glue, Lambda, Athena, Amazon EKS, Cloud Watch, and Cloud Trail.
• Experience with AWS IAM Roles for Service Accounts (IRSA), VPC networking, subnets, and security groups.
• Experience with Kubernetes resource management, scheduling, auto‑scaling, Helm charts, kubectl, and YAML manifests.
• Experience integrating Spark with Kubernetes operators and dynamic resource allocation.
• Strong programming experience using Python or Scala.
• Experience writing clean, modular, scalable, and high‑performance code.
• Strong understanding of functional programming concepts, concurrency, memory management, and collections.
• Strong SQL skills, including window functions, complex joins, aggregations, and query optimization.
• Experience using AI development tools such as Git Hub Copilot, Amazon Q Developer, ChatGPT, or Claude.
• Experience applying AI tools to improve software development workflows and engineering productivity.
• Strong analytical, troubleshooting, communication, and problem‑solving skills.
• Experience working in Agile software development environments.

• Experience managing enterprise ETL and production data pipeline platforms.
• Experience with CI/CD tools such as Jenkins, Git Lab CI, Git Hub Actions, or ArgoCD.
• Experience with Infrastructure as Code (Terraform or AWS Cloud Formation).
• Experience with Docker and container image optimization.
• Experience with service mesh technologies such as Istio or Linkerd.
• Experience with monitoring and observability tools including Prometheus, Grafana, or the ELK Stack.
• AWS certifications such as AWS Certified AI Practitioner, AWS Certified Solutions Architect, or AWS Certified Data Analytics/Specialty.
• Kubernetes certifications such as Certified Kubernetes Administrator (CKA) or Certified Kubernetes Application Developer (CKAD).
• Experience with Git Ops deployment practices.
• Master's degree in Computer Science, Information Systems, or a related field.
• Experience working within the Financial Services industry.