More jobs:
Job Description & How to Apply Below
Skill Area
What the QA Engineer Must Be Able to Do
- Distributed Systems Testing
Validate data consistency across services, handle async events, retries, eventual consistency
- Observability Validation
Test correctness of traces, spans, metrics, and logs correlation
- Telemetry Pipeline Testing
Validate ingestion → transformation → storage → dashboard integrity
- Schema & Contract Testing
Validate agent execution payloads, tool calls, telemetry formats
- Non-Deterministic AI Validation
Create scoring frameworks instead of binary pass/fail
- Cross-Agent Framework Testing
Validate ingestion from Lang Chain, Llama Index, Microsoft Auto Gen, etc.
- Agent Workflow Validation
Validate multi-step planning, tool selection, state propagation
- Responsible AI Testing
Bias detection, toxicity detection, hallucination scoring
- Data Quality Engineering
Detect telemetry gaps, duplication, timestamp drift
- Performance & Scale Testing
Validate ingestion at high token volume / agent concurrency
- Security Testing
Prompt injection resilience, cross-tenant data isolation
- Multi-Tenant SaaS Testing
RBAC validation, data segregation, access policies
Testing areas:
Testing & Validation Tooling
Category
Tools
- API Testing
Postman, REST Assured
- Contract Testing
Pact
- Load Testing
k6, Locust
- LLM Evaluation
Promptfoo, Deep Eval
- Security Testing
OWASP ZAP
- Chaos Testing
Gremlin
- Data Validation
Great Expectations
- Synthetic Monitoring
Datadog Synthetics
Exposure:
Observability & Telemetry Tool Exposure
Since your platform aggregates observability data, QA must deeply understand these ecosystems:
Category
Tools / Platforms
- Tracing
Open Telemetry
- Metrics
Prometheus
- Visualization
Grafana Labs
- Log Aggregation
Elastic (ELK), Datadog
- Agent Observability
Lang Chain Lang Smith
- Experiment Tracking
Weights & Biases
- Data Warehousing
Snowflake, Big Query
Responsible AI & Governance Testing
Critical for an Agent Ops platform:
Area
What to Test
- Bias Monitoring
Demographic skew in responses
- Toxicity Detection
Inappropriate output scoring
- Explainability Validation
Traceability of decisions
- Audit Logging
Regulatory compliance trails
- Data Privacy
PII detection & masking
- Model Versioning Impact
Regression after model upgrades
Note that applications are not being accepted from your jurisdiction for this job currently via this jobsite. Candidate preferences are the decision of the Employer or Recruiting Agent, and are controlled by them alone.
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
Search for further Jobs Here:
×