Lead Big Data Engineer

About The Role

In this role, you will lead complex data migration and modernization programs, helping organizations transform legacy platforms into scalable cloud-native environments. Leveraging AWS technologies, distributed data processing frameworks, and AI-assisted development tools, you will drive large-scale transformation initiatives that accelerate delivery, improve data quality, and unlock business value from modern data platforms.

• Design and implement data migration workflows and cloud-based data solutions on AWS
• Build and maintain scalable batch and streaming data pipelines using Python and Py Spark
• Analyze legacy applications, databases, and data processing workflows to define migration approaches and implementation strategies
• Leverage AI-assisted development tools and LLM-powered solutions to accelerate migration, validation, and automation activities
• Design and implement data quality, reconciliation, and validation frameworks for migration projects
• Develop automation solutions using AWS services such as Step Functions, Lambda, and Cloud Watch
• Support modernization initiatives, including migration from legacy technologies to cloud-native platforms
• Collaborate with stakeholders to define requirements, identify risks, and support delivery planning and execution
• Provide technical guidance to engineering teams and promote best practices in data engineering and cloud migration
• Contribute to architecture discussions, technical reviews, and continuous improvement initiatives

• 7+ years of experience in Data Engineering, Big Data, or Data Migration projects
• Strong proficiency in Python and SQL
• Advanced experience with PySpark and distributed data processing frameworks
• Hands‑on experience with AWS services such as Step Functions, S3, Lambda, and Cloud Watch
• Strong knowledge of ETL processes and scalable data pipeline development
• Experience leveraging AI-assisted workflows and Large Language Models (LLMs) in engineering activities
• Understanding of migration strategies, legacy system modernization, and cloud transformation initiatives
• Experience with data validation, quality assurance, and migration governance processes
• Proficiency in guiding technical implementation activities and collaborating effectively across engineering teams
• Upper‑intermediate or higher level of English
• Experience with video, image, or audio processing pipelines for AI applications (as a plus)
• Experience with IoT data platforms and data processing solutions (advantageous)
• Familiarity with NVIDIA technologies, GPU‑accelerated computing, or AI infrastructure (as a win)
• Knowledge of Infrastructure as Code:
  Terraform, Cloud Formation, or similar (as a plus)

Soft Serve is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.