Senior Software & AI Test Engineer

We are hiring a hands‑on Senior Software & AI Test Engineer to design and operationalize a scalable, automation‑first quality framework across our software and AI‑driven systems. This role owns test strategy, infrastructure, and execution, ensuring high‑confidence releases across APIs, cloud services, data pipelines, and AI/ML components.

The mandate is twofold: (1) build robust, modern testing systems and (2) embed a pragmatic culture of quality that keeps pace with rapid product development.

• Design and implement a unified test framework across backend services and APIs, cloud platforms and distributed systems, data pipelines and data quality layers, and AI application and evaluation systems.
• Define test environments, mocking/simulation strategies, and synthetic data generation.
• Build and maintain CI/CD pipelines.
• Integrate testing deeply into CI/CD pipelines with clear gating signals.

• Build and maintain automated test suites: unit, integration, system, regression, and performance testing.
• Develop test orchestration, reporting dashboards, and failure triage workflows.
• Ensure tests are deterministic, reproducible, and fast enough for developer iteration.

• Establish validation strategies for data pipelines: schema validation, anomaly detection, and data integrity checks.
• Build automated tests for ETL workflows and downstream system dependencies.
• Ensure reproducibility between offline experimentation and production behavior.

• Lead investigation of complex failures across services, data, and AI layers.
• Establish structured approaches to failure classification and regression prevention.

• Build continuous evaluation pipelines tied to model releases.
• Define acceptance criteria and release gates for AI features.
• Develop benchmarking tools for comparing models across datasets and scenarios.

• Introduce scalable quality practices: shift‑left testing and testability in design.
• Definition of done includes validation and observability.
• Partner with engineering and product to define measurable quality metrics (defect escape rate, test signal quality) and release criteria aligned with risk.
• Balance thoroughness with speed—avoid over‑engineering test systems.

• 6–10+ years in software testing, SDET, or quality engineering.
• Strong programming skills (Python required; experience with backend systems preferred).
• Proven track record building test frameworks and automation from scratch.
• Deep understanding of API testing, distributed systems, cloud architectures, CI/CD systems (e.g., Git Hub Actions, Jenkins), test methodologies (boundary testing, fuzzing, fault injection).
• Hands‑on experience validating AI/ML systems, including model evaluation metrics, dataset validation, experiment tracking tools (e.g., MLflow, Weights & Biases).
• Experience with LLM or computer vision evaluation is a plus.

• Experience testing data‑intensive systems or analytics platforms.
• Familiarity with data engineering tools and workflows.
• Experience with performance, scalability, and reliability testing.
• Exposure to observability tooling (logs, metrics, tracing) for test validation.
• Experience working in fast‑paced product environments with evolving requirements.
• Design and implement evaluation frameworks for AI/ML systems: model performance, robustness, edge cases, failure modes, data quality, drift detection, dataset versioning.

• Builder mindset:
  Creates frameworks and tools, not just test cases.
• Systems thinker:
  Understands interactions across APIs, data, and AI layers.
• Pragmatic operator:
  Applies the right level of rigor for the stage of product maturity.
• Quality driver:
  Elevates engineering standards without becoming a bottleneck.
• Hands‑on depth:
  Writes code, debugs systems, and owns outcomes.

• Production‑grade automated test framework integrated into CI/CD.
• Clear, adopted AI evaluation framework used across AI application releases.
• Reduction in escaped defects and regression incidents.
• Measurable improvement in data pipeline reliability and validation coverage.
• Increased developer confidence in releases without slowing iteration speed.

At Covalent, you’ll work alongside world‑class scientists and engineers in a dynamic, collaborative environment. We empower our team members to take ownership of their work, innovate constantly, and engage directly with customers, shaping the future of technology.

125,000 – 175,000 USD per year (Covalent Sunnyvale)