Senior Hadoop Developer

Senior Hadoop Developer to develop, create, and modify general computer applications software or specialized utility programs.
Job responsibilities and duties include:

• Implement data analytics processing algorithms on Big Data batch and stream processing frameworks (e.g. Hadoop Map Reduce, Python, Spark, Scala, Kafka etc.).
• Perform data acquisition, preparation, and perform analysis leveraging a variety of data programming techniques in Spark using Scala.
• Work on complex issues where analysis of situations and data requires an in-depth evaluation of variable factors.
• Load data from different datasets and decide on which file format is efficient for a task. Hadoop Developers source large volumes of data from diverse data platforms into Hadoop platform.
• Install, configure, and maintain enterprise Hadoop environment.
• Build distributed, reliable, and scalable data pipelines to ingest and process data in real-time. Hadoop Developers deals with fetching impression streams, transaction behaviors, clickstream data, and other unstructured data.
• Define Hadoop Job Flows and manage Hadoop jobs using Scheduler.
• Review and manage Hadoop log files.
• Design and implement column family schemas of Hive and HBase within HDFS
• Assign schemas and create Hive tables with suitable formats and compression techniques.
• Mentor Big Data Developers on best practices and strategic development.
• Develop efficient Pig and Hive scripts with joins on datasets using various techniques.
• Apply different HDFS formats and structure like Parquet, Avro, etc. to speed up analytics.
• Fine tune Hadoop applications for high performance and throughput.
• Troubleshoot and debug any Hadoop ecosystem run time issues.
• Develop and document technical design specifications.
• Design and develop data integration solutions (batch and real-time) to support enterprise data platforms including Hadoop, RDBMS, and No
  
  SQL.
• Lead technical meetings, as required, and convey ideas clearly and tailor communication based on selected audience (technical and non-technical).
• Implement Spark Streaming architecture and integration with JMS queue with custom receivers.
• Develop and deploy API services in Java Spring.
• Create Hive and HBase data source connection to Spring.
• Implement multi-threading in Java/Scala.