Senior QA Analyst, AI & Threat Analytics

About Keeper

Keeper Security is transforming cybersecurity for organizations around the world with next-generation privileged access management. Keeper’s zero-trust and zero-knowledge cybersecurity solutions are FedRAMP and State

RAMP Authorized, FIPS 140-2 validated, as well as SOC 2 and ISO 27001 certified. Keeper deploys in minutes, not months, and seamlessly integrates with any tech stack to prevent breaches, reduce help desk costs and ensure compliance. Trusted by thousands of organizations to protect every user on every device, Keeper is the industry leader for best-in-class password management, secrets management, privileged access, secure remote access and encrypted messaging.

Learn more at

The Senior QA Analyst AI & Threat Analytics will lead testing efforts for AI-driven features and machine learning models across Keeper’s platform, including LLM-based summarization, autofill and classification systems, and behavior analysis pipelines. You will design and execute test strategies that validate model accuracy, reliability, guardrails, data quality, and integration with production systems. This role requires hands-on Python automation expertise, strong analytical and evaluation skills, and the ability to test nondeterministic systems with rigor and consistency.

You will collaborate closely with ML engineers, security teams and product engineering to ensure AI features meet Keeper’s quality, safety and performance standards.

• Design and execute test plans for AI and ML-driven features, including classification, summarization, anomaly detection and autofill systems
• Develop Python-based automation and evaluation scripts to validate model performance, reliability and output quality
• Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk and behavior under edge cases
• Validate data pipelines, input/output transformations and model integration points across Keeper’s platform
• Conduct regression testing on ML models and AI workflows to ensure reliability across releases
• Test enterprise-scale behavior analysis features driven by data from ARAM, PAM and session recordings
• Validate security, privacy and zero-trust requirements for AI model interactions and data handling
• Collaborate with ML engineers and cross-functional teams to refine model behavior, identify defects and improve output quality
• Analyze logs, metrics and model outputs to detect anomalies, performance issues or silent failures

• 3+ years of experience in QA, test automation or evaluation of complex systems
• Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines
• Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection or behavior analysis
• Experience testing LLM-integrated features or AI-driven workflows
• Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar
• Experience with API testing, JSON validation and test data generation
• Ability to test nondeterministic systems and evaluate model quality with structured methodologies
• Strong analytical, troubleshooting and communication skills
• Experience testing cloud-integrated applications or systems deployed on AWS
• Bachelor’s degree in Computer Science, Information Systems or related field, or equivalent practical experience

• Experience testing security-focused AI or threat analytics systems
• Familiarity with ML observability tools, vector databases or data processing frameworks
• Experience testing video, image or behavior analysis pipelines
• Knowledge of zero-trust principles and secure data handling
• Experience working with LLM providers or AI guardrail frameworks

• Medical, Dental & Vision (inclusive of domestic partnerships)
• Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
• Voluntary Short/Long Term Disability Insurance
• 401K (Roth/Traditional)
• A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
• Above market annual bonuses

Keeper Security, Inc. is an equal opportunity employer and participant in the U.S. Federal E-Verify program. We celebrate diversity and are committed to creating an inclusive environment for all employees.