Lead Associate Principal, Software Engineering: Software Development Test; SDET

What You'll Do:

OCC's Enterprise Quality Engineering organization seeks an Associate Principal in Performance Testing to design and execute comprehensive performance and failover testing strategies across mission-critical clearing, risk, and settlement systems. This role combines hands-on performance engineering expertise to ensure systems meet regulatory RTO/RPO requirements, handling multi-region failover scenarios, cloud-native architectures, and distributed event-driven systems.

To perform this job successfully, an individual must be able to perform each primary duty satisfactorily.

• Define test plans, load profiles, and ramp scenarios for high-volume transaction systems

• Execute tests using Load Runner, JMeter, and Gatling; analyze results and recommend optimizations

• Lead DR testing across primary, secondary, and tertiary regions

• Build and maintain performance test automation using Python, Java, and open-source tools

• Develop CI/CD-integrated pipelines (Jenkins, Git Hub Actions) for continuous performance validation

• Implement regression testing to catch performance degradation from code changes

• Validate application behavior across multi-region deployments

• Define and validate RTO/RPO targets for critical applications

• Design failover scenarios (region switching, data replication, automation, post-failover validation)

• Partner with infrastructure/application teams to verify state consistency and data integrity during failover

• Test replication lag, failover triggers, and cross-region data consistency

• Validate Kafka producer/consumer behavior under failover conditions

• Confirm performance SLAs hold during region-switching scenarios

• Integrate test results with observability platforms

• Maintain and evolve Load Runner scripts and environments

• Document methodologies, baselines, and trend analysis

• Mentor QE engineers on performance and failover testing techniques

• Collaborate with development, Dev Ops, and infrastructure teams on requirements and capacity planning

• Ensure testing aligns with PGP compliance and financial market infrastructure regulations

• Document test evidence to support regulatory audits

• Validate performance SLAs meet regulatory RTO/RPO requirements

• None

The requirements listed are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the primary functions.

• 7+ years hands-on experience in performance testing and engineering, including design and execution of complex performance test scenarios

• 5+ years designing and executing failover/DR testing across multi-region architectures (cloud and hybrid environments)

• Strong expertise in performance testing tools:
  Load Runner (enterprise scaling), JMeter, Gatling, with ability to evaluate tool trade-offs

• Proficiency in scripting/programming:
  Python, Java, JavaScript, with ability to develop performance testing frameworks

• Solid hands-on experience with AWS cloud environments, including EC2, RDS, Kafka, and performance monitoring

• Working knowledge of Kubernetes and containerized application performance testing (Docker, Kafka, event-driven systems)

• Experience with UC4 (Automic) job scheduling in test execution and orchestration

• Hands-on experience testing Kafka producer/consumer performance, lag validation, and message throughput

• Strong understanding of relational databases (SQL Server, Oracle, DB2) and ability to write queries for performance analysis

• Experience with CI/CD pipelines (Jenkins, Git Hub, Git Lab CI) and integration of performance tests into automation

• Proficiency with APM/observability platforms (Splunk, Dynatrace, App Dynamics) and correlation of test results with production telemetry

• Working knowledge of Linux operating systems and command-line tools for performance analysis

• Experience with test management tools (Jira, Zephyr, Spira Test) and defect tracking integration

• Background in finance domain OR highly regulated, low-latency, high-availability systems (clearing, settlement, risk)

• Experience with multi-region testing (primary/secondary/tertiary architectures) with performance validation…