Big Data Engineer Hive, Spark, and Scala - W2

Job Description

Big Data Engineer – Hive, Spark, Scala – Phoenix, AZ – Onsite (Hybrid) – Contract W2 – 24+ Months
Position:
Big Data Engineer with Hive, Spark, and Scala

• Minimum 7 years experience as a Big Data Engineer or Scala Developer, with a strong focus on big data technologies.
• Proficient in Scala programming language, including experience with functional programming concepts, type systems, and concurrency.
• Hands‑on experience with big data technologies, such as Apache Hadoop, Apache Spark, Apache Kafka, and Apache Hive.
• Knowledge of data processing patterns and architectural patterns used in big data systems, such as batch processing, stream processing, and lambda architecture.
• Familiarity with data modeling techniques and experience in designing efficient data storage solutions, including relational databases, No
  
  SQL databases, and data lakes.
• Strong analytical and problem‑solving skills, with ability to identify and address performance bottlenecks and data quality issues.
• Excellent communication and collaboration skills, ability to work cross‑functionally and translate technical concepts to non‑technical stakeholders.
• Experience with cloud‑based big data platforms, such as Amazon EMR, Google Cloud Dataproc, or Microsoft Azure HDInsight.
• Familiarity with data visualization and business intelligence tools, such as Tableau, Power BI, or Looker.
• Knowledge of machine learning and data science techniques, and experience in integrating them into big data pipelines.
• Exposure to Dev Ops practices and tools, such as Docker, Kubernetes, or CI/CD pipelines.

• Design and implement scalable and efficient big data processing pipelines using Scala and big data technologies like Apache Spark and Apache Hive.
• Develop and maintain batch and real‑time data ingestion, transformation, and processing workflows.
• Integrate and optimize big data technologies, such as Apache Hadoop, Apache Spark, Apache Kafka, and Apache Hive, to create a robust and high‑performing data ecosystem.
• Write efficient and optimized Scala code for data processing and transformation tasks using Spark's Data Frame and Dataset APIs.
• Develop and maintain complex SQL queries and data transformations using Apache Hive.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Ensure data quality, consistency, and security throughout the data lifecycle, implementing best practices for data governance and compliance.
• Optimize the performance and scalability of big data systems, leveraging techniques like partitioning, indexing, and caching.
• Automate and streamline data processing tasks using tools like Apache Airflow, Jenkins, or custom scripts.
• Stay up‑to‑date with the latest trends and advancements in the big data and Scala technology landscape, and recommend improvements to the company's data architecture.