Specialist, QA Test Automation

Our culture lifts you up—there is no ego in the way. Our common purpose? We all want to win for our customers. We aim to always be evolving, dynamic, and ambitious. We believe in the power of genuine connections. Each employee is a part of what makes us unique on the market: agile and dedicated.

Regular

Specialist, QA Test Automation

We aren't looking for an all-knowing AI guru to define our entire organizational strategy from scratch. We are looking for a sharp, curious Specialist, QA Test Automation who can bridge the gap between strategic vision and day-to-day technical execution. Working closely with the Chapter Lead, QA — who provides the overarching AI QA strategy—you will be embedded within Cogeco’s AI squads to bring that strategy to life.

In this role, you will move past standard "pass/fail" scripts to tackle the complex world of non-deterministic outputs. You will focus heavily on probabilistic testing, curating golden datasets, and leveraging cutting-edge LLM evaluation tools like Promptfoo. If you have a solid foundation in QA or software engineering, a healthy obsession with automation, and a restless drive to learn, you’ll fit right in.

• Implement and refine testing workflows tailored for AI powered solutions where outputs are probabilistic rather than deterministic.
• Build guardrails to identify model drift, prompt regression, hallucination risks, and safety violations before they hit production.

• Own the creation, maintenance, and scaling of high-quality golden datasets used as the source of truth for benchmarking model changes.
• Champion programmatic evaluation tools such as Promptfoo, Deep Eval, or Ragas to automate the assessment of prompt variants and model performance.

• Partner closely with the Chapter Lead, QA to translate high-level AI QA frameworks into actionable, daily tasks for technical teams.
• Provide real-time feedback to the Chapter Lead on how the strategy is performing in the trenches, recommending practical pivots based on real-world results.

• Collaborate with Dev Ops specialists to inject AI evaluation metrics directly into modern CI/CD pipelines, ensuring automated gates handle fluid AI responses.
• Validate data pipelines feeding our models, ensuring that data integrity is maintained from ingestion to inference.

• College diploma or Degree in Computer Science, Engineering, or a related discipline (or equivalent practical experience).
• A strong foundation in core QA principles combined with a passion for AI/ML. You don't need a PhD; you just need to be eager to learn how LLMs behave.

• 3+ years of experience in Quality Assurance, Software Engineering, or Data Engineering.
• Hands‑on scripting:
  Experience in Python or JavaScript/Type Script (essential for configuring evaluation frameworks like Promptfoo and manipulating test data).
• Modern QA ecosystem:
  Experience with test automation frameworks, API testing, and code repositories (Git).

• Exposure to golden datasets:
  Creating, maintaining, and scaling high-quality golden datasets used as the source of truth for benchmarking model changes.
• Exposure to AI evaluation tooling:
  Champion programmatic evaluation tools such as Promptfoo, Deep Eval, or Ragas to automate the assessment of prompt variants and model performance.
• CI/CD familiarity:
  Experience working with pipeline tools (e.g., Bitbucket pipelines, Google Cloud run, Git Hub Actions, Git Lab CI, Jenkins).

• Analytical deep‑dives:
  Comfortable analyzing data inputs/outputs and debugging prompts to identify the root causes of unexpected AI behavior.
• Tool adaptability:
  Ready to quickly learn, test, and implement new open‑source AI evaluation and observability tooling as the ecosystem evolves.

• Comfortable with ambiguity:
  Thrives in a high‑uncertainty environment where standard QA blueprints don’t always apply.
• Execution focus:
  Excel at taking an established roadmap or framework and running with…