Testing Engineer

Location:

Colombia

Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurable results for our clients. We’re all in on AWS, combining deep UX capability with senior engineering talent to get AI into production fast and keep it there.
We’ve earned the trust of leaders across Consumer Products and Retail, Education, Energy, Financial Services, Healthcare, and Manufacturing and more, and earned a reputation as the nimble alternative to traditional global systems integrators. Founded in 2009, with delivery centers in Canada, the United States, Eastern Europe, and Latin America, we are smaller, faster, and more senior by design. Our teams average 15+ years of experience.

We move fast, sweat the details, and build things that actually ship.

We’re looking for a Testing Engineer to build and maintain quality across software features and AI/ML systems. This role is ideal for an engineer who can confidently own testing responsibilities across automation and AI, contributes to quality strategy discussions, and is ready to take on real responsibility from day one.

In this role, you will work as a contributing member of a cross-functional team, writing automated tests, validating AI/ML pipelines and outputs, and integrating quality practices into the delivery process. You’ll be joining real, in-flight work where quality and delivery speed matter, collaborating with experienced engineers and growing the technical instincts that come from shipping things that matter.

At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate.
Every role here contributes directly to that mission. Here, you shape how AI systems integrate into enterprise operations, how teams move at real velocity, and how products create measurable impact for clients and the people they serve. We ship production-ready AI in 30 to 45 days. That pace demands people who take ownership, lead with craft, and care deeply about what they put their name on.

• Write and maintain automated test scripts for UI, API, and integration testing (e.g., Selenium, Cypress, Playwright, PyTest)

• Validate ML model outputs including accuracy, consistency, and expected behavior against defined criteria

• Test data pipelines and AI workflows, performing dataset validation and data quality checks

• Execute automated test suites, analyze results, and identify, document, and track defects (e.g., Jira, Test Rail)

• Integrate automated tests into CI/CD pipelines to support continuous quality (e.g., Git Hub Actions, Git Lab CI, Jenkins)

• Support model evaluation processes and assist in testing AI workflows end-to-end

• Test AI/ML systems, including LLM evaluation (accuracy, hallucination, toxicity, bias), RAG retrieval quality and grounding, model drift monitoring, prompt regression testing, and AI safety and red-teaming

• Bring an AI-forward mindset to your daily work, using tools like Claude, Cursor, and other modern AI assistants to ship higher-quality work at pace

• Collaborate closely with developers, ML engineers, and data scientists to align quality efforts with delivery goals

• Participate in sprint ceremonies and contribute to quality strategy discussions

• Provide clear testing reports and communicate quality status to the team

• Take ownership of testing responsibilities end-to-end and follow through…