Bigdata Developer

Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues around the world, and where you’ll be able to reimagine what’s possible. Join us and help the world’s leading organizations unlock the value of technology and build a more sustainable, more inclusive world.

Role Overview We are looking for a Senior Big Data Developer with 5+ years of experience in big data engineering, API integration, and AI-assisted development. The ideal candidate will design, build, and maintain scalable data pipelines and backend systems in a enterprise environment.

Key Responsibilities Big Data & Spark Design and develop Spark-Scala applications for large-scale data processing on Hadoop/CDP clusters

Build and optimize ETL/ELT pipelines using Spark Data Frames, Datasets and Spark SQLTune Spark jobs for performance (partitioning, caching, broadcast joins, shuffle optimization)
Migrate Spark 2 applications to Spark 3 on Cloudera CDP platforms

Work with Parquet, ORC, Avro file formats on HDFSSQL & Data Engineering Write complex HiveQL / Spark SQL queries including window functions, CTEs, subqueries and aggregations

Design and maintain Hive external/managed tables and partitioned datasets

Optimize slow-running queries and resolve correlated subquery issues

Work with HDFS encryption zones and data governance requirements

Unix / Shell Scripting Develop and maintain bash shell scripts for job orchestration and automation

Handle error management, return codes, logging and alerting in shell scripts

Manage HDFS operations (hdfs dfs commands), file transfers, and data validation

Manage Kerberos authentication (kinit, keytab handling)
API Extraction & Integration Build scripts and pipelines to extract data from REST APIs using curl and Python Handle OAuth2 token generation, bearer token refresh and API health checks

Parse and process JSON API responses and load into HDFS/Hive Manage pagination, error handling and retry logic for API calls

Work with enterprise API gateways and URL parameter construction AI & Copilot Capabilities Leverage Git Hub Copilot / AI coding assistants to accelerate development

Use AI tools for code review, SQL generation, script debugging and documentation

Contribute to AI-assisted data quality and anomaly detection pipelines

Explore and implement LLM-based automation for repetitive data engineering tasks

Scheduling & Orchestration Schedule and manage jobs using AAP (Ansible Automation Platform) / Control-M / cron Build and maintain Ansible playbooks for automated deployments

Manage deployment pipelines including artifact versioning, Vault secret injection and environment-specific configuration

Monitor job health, handle failures and implement alerting

Nice to Have

Experience with Cloudera CDP (7.x) and migration from HDPKnowledge of Kerberos, Vault, HDFS encryption zones

Familiarity with CI/CD pipelines (Helios, Git Hub Actions)

Experience with MSSQL / JDBC connectivity from Spark Understanding of AML / Financial regulatory data domains.

The base compensation range for this role in the posted location is: 79,000 to 102,000.Capgemini provides compensation range information in accordance with applicable national, state, provincial, and local pay transparency laws. The base compensation range listed for this position reflects the minimum and maximum target compensation Capgemini, in good faith, believes it may pay for the role at the time of this posting.

This range may be subject to change as permitted by law.

The actual compensation offered to any candidate may fall outside of the posted range and will be determined based on multiple factors legally permitted in the applicable jurisdiction.

These may include, but are not limited to:
Geographic location, Education and qualifications, Certifications and licenses, Relevant experience and skills, Seniority and performance, Market and business consideration, Internal pay equity.

It is not typical for candidates to be hired at or near the top of the posted compensation  addition to base salary, this role may be eligible for additional…