Senior Data Engineer

Our passion at EXL is enabling organizations to achieve a measurable, competitive advantage through Data Management. We partner with our clients to design and implement scalable, accessible, and innovative solutions. Our goal is to make sense of data to drive client’s business forward through data strategy, data governance and building cloud data platforms. EXL is a leading data analytics and digital operations and solutions company that partners with clients to improve business outcomes and unlock growth.

By bringing together deep domain expertise with robust data, powerful analytics, cloud, artificial intelligence (“AI”) and machine learning (“ML”), we create agile, scalable solutions and execute complex operations for the world’s leading corporations in industries including insurance, healthcare, banking and financial services, media, and retail, among others. Focused on driving faster decision-making and transforming operating models, EXL was founded on the core values of innovation, collaboration, excellence, integrity and respect.

Headquarters in New York, our team is over 44,500+ strong, with more than 50 offices spanning six continents.

Are you committed to building data solutions, and being a Data Management expert? Are you passionate about data? Would you like to work on solutions with tangible impact to our clients?

We are looking for an Sr AWS Data & Solutions Engineer with primary skills on Python & PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources.

You will be responsible for building API services using FastAPI or Flask frameworks.

• Design, build and unit test applications on Spark framework on Python.
• Build Python and PySpark based applications based on data in both Relational databases (e.g. Oracle), No
  
  SQL databases (e.g. DynamoDB, MongoDB) and file systems (e.g. S3, HDFS)
• Build AWS Lambda functions on Python runtime leveraging awswrangler, pandas, json, requests
• Build PySpark based data pipeline jobs on AWS Glue ETL or EMR Clusters
• Build Python based event-driven integration with Kafka Topics, leveraging Confluent libs
• Leveraged Apache Iceberg to manage schema evolution and ACID-compliant CDC merges within the data lake Design and Build API services using FastAPI, understand the swagger metadata files and implement OAuth2/JWT authentication for protected endpoints
• Build the process orchestration pipelines using AWS Step Functions and Eventbridge rules.
• Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
• Deploy applications on Docker and Kubernetes containers
• Leverage copilot/GPT for agentic coding of above tech stack
• Optimize performance of Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's
• Setup the Glue crawlers to catalog OracleDB tables, MongoDB collections and S3 objects
• Ability to monitor, troubleshoot and debug failures using AWS Cloud Watch and Datadog
• Ability to solve complex data-driven scenarios and triage towards defects and production issues
• Participate in code release and production deployment.
• Create documentation for user adoption, deployments, runbook, and support client users for enablement or for any issues encountered.
• Perform code reviews with the team and enable them to develop code for complex scenarios
• Participate…