Data Engineer - Senior

At EY, we're all in to shape your future with confidence.

We'll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go.

Join EY and help to build a better working world.

Designation AIOps Engineer

Job Description

Develop and maintain scalable data pipelines using tools like Apache Kafka, Apache Spark, or AWS Glue to ingest, process, and transform large datasets from various sources, ensuring efficient data flow and processing.
Design and implement data models and schemas in data warehouses (e.g., Amazon Redshift, Google Big Query) and data lakes (e.g., AWS S3, Azure Data Lake) to support analytics and reporting needs.
Collaborate with data scientists and analysts to understand data requirements, ensuring data availability and accessibility for analytics, machine learning, and reporting.
Utilize ETL tools and frameworks (e.g., Apache NiFi, Talend, or custom Python scripts) to automate data extraction, transformation, and loading processes, ensuring data quality and integrity.
Monitor and optimize data pipeline performance using tools like Apache Airflow or AWS Step Functions, implementing best practices for data processing and workflow management.
Write, test, and maintain scripts in Python, SQL, or Bash for data processing, automation tasks, and data validation, ensuring high code quality and performance.
Implement CI/CD practices for data engineering workflows using tools like Jenkins, Git Lab CI, or Azure Dev Ops, automating the deployment of data pipelines and infrastructure changes.
Collaborate with Dev Ops teams to integrate data solutions into existing infrastructure, leveraging Infrastructure as Code (IaC) tools like Terraform or AWS Cloud Formation for provisioning and managing resources.
Manage containerized data applications using Docker and orchestrate them with Kubernetes, ensuring scalability and reliability of data processing applications.
Implement monitoring and logging solutions using tools like Prometheus, Grafana, or ELK Stack to track data pipeline performance, troubleshoot issues, and ensure data quality.
Ensure compliance with data governance, security best practices, and data privacy regulations, embedding Dev Sec Ops  principles in data workflows.
Participate in code reviews and contribute to the development of best practices for data engineering, data quality, and Dev Ops methodologies.
Contribute to the documentation of data architecture, processes, and workflows for knowledge sharing, compliance, and onboarding purposes.
Demonstrate strong communication skills to collaborate effectively with cross-functional teams, including data science, analytics, and business stakeholders.

Desired Profile

Expert proficiency in Terraform and extensive experience across at least two major cloud platforms (AWS, Azure, GCP).
Strong hands-on experience with Kubernetes, Helm charts, and designing/optimizing CI/CD pipelines (e.g., Jenkins, Git Lab CI) is essential.
Proficiency in Python and scripting languages (Bash/Power Shell) is also a must.

Experience with data-related tools and technologies, including Apache Kafka, Apache Spark, or AWS Glue for data processing, as well as familiarity with data storage solutions like Amazon Redshift, Google Big Query, or Azure Data Lake, is highly valued.
Valued experience includes leading cloud migration initiatives, contributing to RFP/RFI processes, and mentoring teams.
Excellent problem-solving, communication, and collaboration skills are critical.

Experience with configuration management tools (Ansible, Puppet) and a solid understanding of Dev Sec Ops  principles is required; experience with Open Shift is a plus.

Experience 6 Years And Above

Education

B.Tech. / BS in Computer Science

Technical Skills & Certifications

Certifications in cloud platforms (e.g., AWS Certified Data Analytics, Google Professional Data Engineer, Azure Data Engineer Associate).
Proficiency in data processing tools and frameworks such as Apache Kafka, Apache Spark, and AWS Glue.
Strong experience with data storage solutions like Amazon Redshift, Google Big Query, and Azure Data Lake.
Expertise in ETL tools and frameworks (e.g., Apache NiFi, Talend) and scripting languages (Python, SQL, Bash).
Familiarity with CI/CD practices for data engineering workflows using tools like Jenkins, Git Lab CI, or Azure Dev Ops.

Experience with Infrastructure as Code (IaC) tools like Terraform for provisioning and managing data resources.
Knowledge of containerization technologies (Docker) and orchestration (Kubernetes) for managing data applications.
Understanding of monitoring and logging solutions (e.g., Prometheus, Grafana, ELK Stack) to ensure data pipeline performance and quality.
Awareness of data governance, security best practices, and data privacy regulations, embedding Dev Sec Ops  principles in data workflows.

EY | Building a better working world

EY is building a better working world by creating new value for clients, people, society and the planet, while building…