Senior Staff Engineer (701656

Big Data Engineer

We are looking for an experienced big data engineer to design and build our data pipeline and ETL processes, which are scalable, highly performant and able to handle diverse datasets including IoT, leveraging the latest big data technologies such as Hadoop/HDFS, Kafka, and HBase etc.

• Build data pipelines and ETL from heterogeneous sources including IoT using Kafka, Flume, Sqoop, Spark Streaming and custom code etc. Your solutions will support both real-time or near real-time applications
• Partner with product owners, data analyst and data scientists, to better understand their requirements, find bottlenecks and come up with the best resolution, etc.
• Expand and grow data platform capabilities to solve new data problems and challenges
• You will work closely with development and QA teams both local and oversea to design ingestion pipelines
• You will also need to mentor other team members

• 6+ years of software development experience with minimum 2-year hands-on experience in Hadoop ecosystem and other Big Data technologies (Kafka, Cassandra, MongoDB and Elastic search)
• Expert level software development experience using Java and/or Python
• Hands-on experience with building big data ingestion pipelines using Hadoop, HBase, Kafka, Elastic Search etc
• Knowledge and hands-on experience with relational databases (Greenplum and Oracle MySQL/Postgres etc) and database schema design
• Experience with Unix/Linux and shell scripting
• Knowledge of professional software engineering practices & best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes and testing
• Ability to take a project from scoping requirements through actual launch of the project
• BS/MS Degree in Computer Science or related field
• Excellent inter-personal and teamwork skills, having experiences working with oversea teams is a plus