Senior Spark & Python Developer

A typical day as a Senior Spark Programmer:

• Develop fault tolerant, streaming as well as batch data integration processes using Spark/PySpark, Java and performance enhanced SQL
• Develop and lead development efforts of application programming interfaces to enable integration of fraud detection systems with a host of new reporting and data mining tools
• Design and develop automation of data flow tasks and end-to-end process testing
• Design and develop scalable frameworks to ingest, transform, store, and present loss prevention information to downstream systems
• Implement and lead implementation of Agile best practices and a continuous integration ecosystem
• Implement and lead implementation efforts of Spark/Python based solution architecture, scalable process frameworks, advanced analytics, and responsive RESTful services

• 5+ years experience with processing of structured, unstructured and semi-structured data using in-memory cluster computing technologies, specifically with Spark
• 5+ years experience as a Java programmer
• 3+ years experience programming in Python (PySpark API)
• 3+ years experience with cloud services offered through AWS, like EMR, Redshift, or S3
• 3+ years experience in developing, testing and deploying RESTful APIs for high volume data streams
• Experience with continuous integration tools like Jenkins
• Experience working in a distributed computing infrastructure like Hadoop and/or Spark
• Critical and analytical approach to solving technical problems
• Excellent interpersonal skills and ability to clearly communicate highly technical concepts to business stakeholders and technical developers alike