Senior Big Data Engineer — AWS, Spark & Snowflake Pipelines

The Big Data Engineer will be responsible for designing, developing, optimizing, and supporting scalable big data platforms and cloud-native applications within AWS environments. This role requires expertise in both batch processing and API-driven architectures, leveraging technologies such as Spark, Hadoop, Scala, PySpark, Java, and Snowflake. The ideal candidate will have strong experience building enterprise-grade data pipelines, microservices, and cloud-based analytics solutions while ensuring performance, scalability, and reliability.

Key Responsibilities

• Design, develop, implement, test, and maintain Big Data applications supporting both batch and API-based workloads.

• Build and optimize scalable data ingestion, transformation, and processing pipelines using Spark and related technologies.

• Develop and maintain cloud-native applications and data platforms within AWS environments.

• Design, develop, and support REST APIs and SOAP web services for data and application integration.

• Build and maintain microservices architectures using AWS services and modern development frameworks.

• Develop scalable batch processing solutions using Hadoop, Spark, EMR, and distributed computing technologies.

• Optimize data processing jobs and applications for performance, scalability, and cost efficiency.

• Develop and support data workflows using AWS Step Functions, Apache Airflow, and related orchestration tools.

• Design and implement data ingestion and transformation pipelines within Snowflake environments.

• Work with large-scale structured and unstructured datasets across distributed systems.

• Develop automation solutions using Python, shell scripting, and cloud-native tools.

• Support and maintain Cassandra databases and distributed data storage solutions.

• Troubleshoot production issues and perform root cause analysis across data platforms and applications.

• Collaborate with cross-functional teams to gather requirements and deliver scalable solutions.

• Participate in code reviews, testing, deployment activities, and continuous improvement initiatives.

• Maintain technical documentation and operational procedures.

Required Qualifications

• 7+ years of experience in the analysis, development, implementation, and testing of Big Data applications.

• Strong experience working within AWS cloud environments.

• Strong programming experience with Scala.

• Strong experience with PySpark and Apache Spark.

• Strong Java development experience, including REST APIs and SOAP web services.

• Experience with Hadoop ecosystem technologies including Hadoop, HDFS, Spark, and Map Reduce.

• Hands-on experience with Cassandra.

• Experience building and supporting applications using AWS services including:

• Experience working with both batch processing systems and API-driven microservices architectures.

• Strong experience with performance tuning and optimization of Spark, Hadoop, and EMR workloads.

• Strong Linux administration and troubleshooting experience.

• Experience with shell scripting and Python automation.

• Experience building data ingestion and transformation pipelines using Snowflake and Spark.

• Strong analytical, troubleshooting, and problem-solving skills.

• Ability to work independently in a remote environment.

• Strong communication and collaboration skills.

Preferred Qualifications

• Experience with Python and Kotlin development.

• Experience with Apache Airflow for workflow orchestration.

• Experience with Dev Ops tools including Git, Git Hub, Git Lab, and Bitbucket.

• Experience with development tools such as IntelliJ IDEA and PyCharm.

• Experience implementing CI/CD pipelines using Maven, Gradle, Jenkins, and Artifactory.

• Experience with AWS Code Commit and AWS Cloud Formation.

• Experience with cloud-native architecture patterns and distributed systems design.

• Experience supporting enterprise-scale analytics and data platforms.

• Knowledge of data governance, security, and operational best practices.

Required Skills