Senior PySpark & Spark ETL Data Engineer

Work Location

Irving, Texas/ Tampa, FL

• Exp 6 Years Must have good technical experience and should be able to provide technical solutions for multiple modules in parallel on need basis and bring the task to closure on time
• Unix SQL and Shell Scripting experience is a must have
• Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines
• Strong command line knowledge in Unix Linux with Shell scripting using Bash Kornshell or Perl and File processing using awk scripts
• Expertise in SQL querying and complex joins
• Implementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycle
• Expertise with complex Data workflows with Apache Air Flow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls
• Strong Analytical skills and expertise on Spark
  
  SQL for Data analysis and validation ensuring the delivery of clean queryready datasets for business consumption
• Expertise in Data quality checks and monitoring
• Quality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automated
• Handson with Automation Framework Design for ETL and API
• SME in Data Analysis Database testing Messaging queues
• Experience with coding standards code reviews source management build processes CICD pipeline

Mandatory

Skills:

Apache Spark, Big Data Hadoop Ecosystem, Python, Python for DATA, SparkSQL