Spark Framework Engineer – Enterprise Data Platform

Outstanding contract opportunity! A well-known Financial Services Company is looking for an Ansible Automation Engineer (with Load Balancing experience) in Columbus or Charlotte (hybrid 3 days on-site). If you're passionate about advancing automation and working with cutting-edge load balancing technology within a collaborative, non-egotistical team environment, this is for you!

Contract Duration: 12

Months

We are seeking a Lead Software Engineer to drive the evolution of Spark Flow
, an enterprise data processing framework built on Apache Spark
. This role will focus on designing clean, composable abstractions, delivering new functional capabilities within the framework, and significantly improving developer experience. The engineer will also design and deploy AI-enabled capabilities and lead the integration of the Spark Flow framework into the Unity control plane.

• Lead the architectural evolution of the Spark Flow framework
  , emphasizing modularity, composability, and clear abstraction boundaries
• Design and implement new functional features within the framework to expand supported use cases and capabilities
• Balance feature delivery with framework quality, ensuring performance, backward compatibility, and governance standards
• Improve developer experience by simplifying APIs, configuration models, extension points, and onboarding patterns
• Design and implement AI driven solutions that assist developers and enhance framework usability, productivity, and operability
• Integrate Spark Flow as a first class framework within the Unity control plane
• Collaborate with data, platform, and infrastructure teams to ensure consistent adoption across environments
• Provide technical leadership through design reviews, code reviews, and architectural guidance

• Deep experience with Apache Spark (Scala and/or Java; Python a plus)
• Proven success building reusable frameworks or libraries
  , including ownership of functional feature development
• Strong skills in API design, abstraction design, and extensibility patterns
• Experience applying AI/ML or LLM based techniques to developer tooling or data frameworks
• Experience working in large?scale, enterprise data environments

• Experience integrating frameworks into control planes or internal developer platforms
• Familiarity with Spark SQL–centric architectures and data governance patterns
• Experience modernizing or replacing legacy ETL frameworks