BI Architect

Job Summary

Deterministic Testing & Data Validation:
Validate generative AI tool outputs for structured, rules‑based use cases by reconciling results against trusted data sources and established SQL-based metrics. Ensure consistency, explainability, and auditability of outputs by confirming alignment with existing data pipelines and query logic. Expand and maintain test coverage across prioritized use cases to establish a robust, high‑confidence baseline for the platform. Partner with data engineering and analytics teams to identify and resolve discrepancies in underlying data or logic.

Non‑Deterministic Testing & Scenario Evaluation:
Design and execute scenario‑based testing for more complex, AI‑driven outputs where direct validation is not always possible. Evaluate results based on intent accuracy, reasonableness, and confidence thresholds rather than exact match validation. Prioritize testing across higher‑risk and high‑impact use cases using curated question sets and real‑world scenarios. Identify patterns in output variability and drive iterative refinement to improve reliability and user trust.

Human‑in‑the‑Loop Review & Continuous Monitoring:
Conduct ongoing review of generative AI tool interactions post‑launch, validating outputs and ensuring quality across all user scenarios. Identify edge cases, inconsistencies, and emerging risks, and elevate findings to product and engineering teams. Synthesize insights from testing and live usage to inform enhancements, training data improvements, and governance practices. Serve as an accountable reviewer, providing a critical control point for responsible AI deployment and continuous improvement.

Location: Malvern, Pennsylvania.

Job Type: Contract.

Salary: $55 - 60 per hour.

Work Hours: 8am to 5pm.

Education: Bachelors.

• Deterministic Testing & Data Validation:
  Validate generative AI tool outputs for structured, rules‑based use cases by reconciling results against trusted data sources and established SQL-based metrics. Ensure consistency, explainability, and auditability of outputs by confirming alignment with existing data pipelines and query logic. Expand and maintain test coverage across prioritized use cases to establish a robust, high‑confidence baseline for the platform. Partner with data engineering and analytics teams to identify and resolve discrepancies in underlying data or logic.
• Non‑Deterministic Testing & Scenario Evaluation:
  Design and execute scenario‑based testing for more complex, AI‑driven outputs where direct validation is not always possible. Evaluate results based on intent accuracy, reasonableness, and confidence thresholds rather than exact match validation. Prioritize testing across higher‑risk and high‑impact use cases using curated question sets and real‑world scenarios. Identify patterns in output variability and drive iterative refinement to improve reliability and user trust.
• Human‑in‑the‑Loop Review & Continuous Monitoring:
  Conduct ongoing review of generative AI tool interactions post‑launch, validating outputs and ensuring quality across all user scenarios. Identify edge cases, inconsistencies, and emerging risks, and elevate findings to product and engineering teams. Synthesize insights from testing and live usage to inform enhancements, training data improvements, and governance practices. Serve as an accountable reviewer, providing a critical control point for responsible AI deployment and continuous improvement.

• Required Skills & Experience
  • Robust SQL skills required.
  • Robust analytical background with experience in data validation, SQL, and analytics workflows.
  • Ability to assess outputs both quantitatively (data accuracy) and qualitatively (reasonableness, business context).
  • Demonstrated critical thinking and sound judgment, especially in ambiguous or non‑deterministic environments.
  • Experience working with large datasets, reporting tools, or analytics platforms.
• Preferred Qualifications
  • Exposure to AI/ML or generative AI tools and associated testing or validation frameworks.
  • Experience in scenario‑based testing, UAT, or model validation.
  • Familiarity with financial services, retirement, or plan sponsor analytics.

Medical, prescription, dental, vision, AD&D, and life insurance offerings, short‑term disability, and a 401K plan (all benefits are based on eligibility).

Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status. At Open Kyber, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact.

For applications and inquiries, contact: