Big Data Developer

Talent Acquisition Partner – US & Canada 🇺🇸🇨🇦 at Capgemini | Driving Tech & FS Hires | Vendor & Delivery Management Expert

We’re looking for a seasoned Senior Data Engineer with strong Hadoop to design, build, and scale data pipelines and platforms powering analytics, AI/ML, and business operations. You’ll own end-to-end data engineering—from ingestion and transformation to performance optimization—across large-scale distributed systems and modern cloud data platforms.

• Design & Build Data Pipelines: Architect, develop, and maintain robust ETL/ELT pipelines for batch and streaming data using Hadoop ecosystem, Spark, and Airflow.
• Big Data Architecture: Define and implement scalable big data architectures, ensuring reliability, fault tolerance, and cost efficiency.
• Data Modeling: Develop and optimize data models for Data Warehouse and Operational Data Store (ODS); ensure conformed dimensions and star/snowflake schemas where appropriate.
• SQL Expertise: Write, optimize, and review complex SQL/HiveQL queries for large datasets; enforce query standards and patterns.
• Performance Tuning: Optimize Spark jobs, SQL queries, storage formats (e.g., Parquet/ORC), partitioning, and indexing to improve latency and throughput.
• Data Quality & Governance: Implement data validation, lineage, cataloging, and security controls across environments.
• Workflow Orchestration: Build and manage DAGs in Airflow
  , ensuring observability, retries, alerting, and SLAs.
• Cross-functional Collaboration: Partner with Data Science, Analytics, and Product teams to deliver reliable datasets and features.
• Best Practices: Champion coding standards, CI/CD, infrastructure-as-code (IaC), and documentation across the data platform.

• 7+ years of hands‑off data engineering experience building production‑grade pipelines.
• Strong experience with Hadoop (HDFS, YARN),
  Hive SQL/HiveQL
  , Spark (Scala/Java/PySpark), and Airflow
  .
• Expert‑level SQL skills with the ability to write and tune complex queries on large datasets.
• Solid understanding of Big Data architecture patterns (e.g., lakehouse, data lake + warehouse, CDC).
• Deep knowledge of ETL/ELT and DW/ODS concepts (slowly changing dimensions, partitioning, columnar storage, incremental loads).
• Proven track record in performance tuning for large‑scale systems (Spark jobs, shuffle optimizations, broadcast joins, skew handling).
• Strong programming background in Java and/or Scala (Python is a plus).

• Experience with AI‑driven data processing (feature engineering pipelines, ML‑ready datasets, model data dependencies).
• Hands‑on with cloud data platforms (
  AWS
  , GCP
  , or Azure
  )—services like EMR/Dataproc/HDInsight, S3/GCS/ADLS, Glue/Dataflow, Big Query/Snowflake/Redshift/Synapse.
• Advanced data governance & security (row/column‑level security, tokenization, encryption at rest/in transit, IAM/RBAC, data lineage/catalog).
• Familiarity with Kafka (topics, partitions, consumer groups, schema registry, stream processing).
• Experience with CI/CD for data (Git, Jenkins/Git Hub Actions, Terraform), containerization (Docker, Kubernetes).
• Knowledge of metadata management and data observability (Great Expectations, Monte Carlo, Open Lineage).

Capgemini supports all aspects of your well‑being throughout the changing stages of your life and career. For eligible employees, we offer:

Healthcare including dental, vision, mental health, and well‑being programs

Financial well‑being programs such as 401(k) and Employee Share Ownership Plan

Paid time off and paid holidays

Paid parental leave

Family building benefits like adoption assistance, surrogacy, and cryopreservation

Social well‑being benefits like subsidized back‑up child/elder care and tutoring

Mentoring, coaching and learning programs

Employee Resource Groups

Disaster Relief

Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity/expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by…