Software Engineering Senior Spark Scala Developer Professional San Jose

At IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation.

With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world.

We are looking for a seasoned
Spark Scala Developer with 12+ years of software engineering experience, including 5+ years in building and optimizing large-scale data processing solutions using
Apache Spark and
Scala
. The ideal candidate will have strong expertise in distributed computing, data pipelines, and both real-time and batch processing architectures.

Your role and responsibilities

Key Responsibilities:

• Design and optimize big data applications using
  Apache Spark and
  Scala
  .
• Tune Spark jobs for performance and cost efficiency on distributed clusters.
• Maintain reusable libraries and ensure best coding practices.
• Work with storage systems such as
  HDFS
  ,
  Hive
  ,
  HBase
  ,
  Cassandra
  ,
  Kafka
  , and
  Parquet
  .
• Mentor junior developers and lead code reviews.
• Ensure compliance with security and governance standards.
• Troubleshoot and resolve performance issues in big data solutions.

Required education

Bachelor's Degree

Preferred education

Master's Degree

Required technical and professional expertise

Required Qualifications:

• Bachelor’s or Master’s in Computer Science or related field.
• 12+ years of software development experience.
• 5+ years of hands‑on experience with
  Apache Spark and
  Scala
  .
• Strong knowledge of distributed computing and cluster frameworks.
• Proficiency in
  Scala and functional programming principles.
• Expertise in Spark tuning, partitions, joins, and optimization techniques.
• Experience with cloud platforms (
  AWS
  ,
  Azure
  ,
  GCP
  ) and tools like
  EMR
  ,
  Databricks
  ,
  HDInsight
  .
• Familiarity with
  Kafka
  ,
  Hive
  ,
  HBase
  , No
  
  SQL databases, and data lake architectures.
• Knowledge of
  CI/CD
  ,
  Git
  ,
  Jenkins
  , and automated testing.
• Strong problem‑solving and collaboration skills.

Preferred technical and professional experience

Preferred:

• Experience with
  Databricks
  ,
  Delta Lake
  , or
  Apache Iceberg
  .
• Exposure to machine learning pipelines using
  Spark MLlib or integration with ML frameworks.
• Open-source contributions in big data projects.
• Excellent communication and leadership abilities.

IBM Software infuses core business operations with intelligence—from machine learning to generative AI—to help make organizations more responsive, productive, and resilient. IBM Software helps clients put AI into action now to create real value with trust, speed, and confidence across digital labor, IT automation, application modernization, security, and sustainability. Critical to this is the ability to make use of all data, because AI is only as good as the data that fuels it.

In most organizations data is spread across multiple clouds, on premises, in private datacenters, and at the edge. IBM’s AI and data platform scales and accelerates the impact of AI with trusted data, and provides leading capabilities to train, tune and deploy AI across business. IBM’s hybrid cloud platform is one of the most comprehensive and consistent approach to development, security, and operations across hybrid environments—a flexible foundation for leveraging data, wherever it resides, to extend AI deep into a business.

In a world where technology never stands still, we understand that, dedication to our clients success, innovation that matters, and trust and personal responsibility in all our relationships, lives in what we do as IBMers as we strive to be the catalyst that makes the world work better.

Being an IBMer means you’ll be able to learn and develop yourself and your career, you’ll be encouraged to be courageous…