Lead Software Engineer - Big Data

Working independently, the Lead Software Engineer owns development of software products and works on improving the overall quality of the product throughout the software development life cycle and mentors other Software Engineers. Reports directly to a Director or Senior Director.

Essential Job Functions:

Seasoned contributor that works on complex projects requiring extensive knowledge of specialty area and industry practices

Manages processes, projects or products to improve efficiency within org or product area Influences outcomes in cross-functional settings

Demonstrates solid understanding of systems and processes within job area and recommends improvements

Designs and develops industry-leading solutions that meet defined quality standards, reliability specifications, and help achieve key strategic objectives

Ensures comparison of tasks and quality of work, supporting peers to ensure the team delivers consistently

Excellent understanding in computer science fundamentals - Algorithm design, Problem solving, Complexity analysis and data structures.

Engineering/Operations Excellence:

Helps identify gaps between system components and designs and delivers solutions

Implements methods for managing and maintaining the organization’s software development environments to ensure software engineers are equipped with the tools and resources necessary to create and validate builds

Takes broad perspective to problems and exercises independent judgement in techniques and evaluation criteria to obtain results

Develops actionable insights from analyzing software development trends, software engineering best practices, and innovative test environment management tools and communicates integration recommendations to management

Identifies ways to overcome challenging situations that jeopardize the achievement of business goals

Collaboration:

Coaches and trains less experienced colleagues in techniques, processes, and responsibilities

Integrates the culture of intelligence, integrity, and initiative into team goals and decisions

Partners within and across teams to remove barriers and identify shared goals and priorities

Openly shares progress and priorities with key stakeholders, communicating complex or technical information at the right level.

Demonstration of FINRA’s values.

Collaboration, both in-person and virtually, in furtherance of FINRA’s mission of investor protection and market integrity.

Essential Technical

Skills:

Big Data Technologies:
Extensive hands-on experience with Hadoop, Spark, Hive, and Trino for processing petabyte-scale datasets. Proven ability to diagnose and resolve data skew, resource limitations, scalability challenges, and job failures in production environments.

Apache Spark Expertise:
Deep understanding of Spark architecture including executors, tasks, stages, and DAG execution.

Demonstrated proficiency in performance tuning techniques such as partitioning strategies, caching, broadcast joins, and optimization of large-scale data processing jobs.

SQL Mastery:
Advanced proficiency in complex SQL including window functions, multi-table joins, aggregations, and query optimization. Experience handling edge cases involving NULLs, duplicates, and ordering in production environments.

Cloud Technologies:
Strong experience with AWS services including S3, EMR, Glue, Lambda, Athena, EKS, and serverless architectures. Practical knowledge of file format optimization, consistency management, and cloud-based data processing workflows.

Programming (Python/Scala):
Proficiency in writing clean, modular, and performant code using Python or Scala. Strong understanding of functional programming concepts (immutability, higher-order functions), collections, concurrency, and memory management for scalable data processing.

AI Tool Proficiency:
Hands-on experience with AI development tools (Git Hub Copilot, Q Developer, ChatGPT, Claude, etc.) and AI workflow design. Proven ability in prompt engineering and leveraging AI coding assistants for enhanced productivity.

Agile Methodology & Change Management:
Extensive experience with Scrum, Kanban, and continuous improvement practices. Demonstrated experience leading teams through AI…