Skip to content
×

Conversational Designer

Job in San Antonio, Bexar County, Texas, 78201, USA
Listing for: Catapult Solutions Group
Full Time position Listed on 2026-11-16
Job specializations:
Salary/Wage Range or Industry Benchmark: 57000 - 82000 USD Yearly USD 57000.00 82000.00 YEAR

Job Description & How to Apply Below

Conversation Designer - AI Chatbot Evaluations

Location: 100% Remote (US) - must work Pacific Time business hours

Role Type: Contract, 12 months (W-2)

About Our Client

Our client is a global financial technology company, headquartered in the San Francisco Bay Area. Its tax, accounting, personal finance and marketing products serve roughly 100 million consumers and small businesses worldwide, and its small-business accounting platform is one of the most widely used in the US. Over the past few years, the company has been transforming into an AI-driven platform, building generative-AI assistants and agents into its products.

Its mission is to power prosperity around the world, and it has a reputation for customer obsession, innovation and a strong engineering and design culture.

Job Description

Our client's small-business platform includes an AI chatbot, built on a leading large language model and extended with company-specific skills and agents, that helps business owners get answers and complete tasks in the product. As more customers rely on it, making sure every answer is accurate, relevant and on-brand is critical. That's why the team is adding three Conversation Designers focused on AI chatbot evaluation and quality.

In this role, you'll build the question-and-answer datasets used to test the chatbot, review and score its responses against approved source content and quality rubrics, and track accuracy over time. When quality drops in an area, you'll dig in, find the pattern behind the errors and give Engineering clear, actionable feedback. You'll work closely with conversation designers, product managers, engineers and data scientists on the conversational AI team.

This is not a visual UX/UI design role and not a software engineering role, and no Python or coding is required. It's ideal for an analytical, detail-oriented person who loves AI, enjoys figuring out why a chatbot got something wrong, and uses AI tools (such as Claude Code or ChatGPT) to work faster and smarter.

Duties and Responsibilities
  • Create large datasets of realistic customer questions with verified correct answers.
  • Develop question-and-answer pairs used to evaluate chatbot performance.
  • Review AI-generated responses for accuracy, relevance and overall quality.
  • Compare chatbot answers against approved source content and documentation.
  • Score responses using established evaluation rubrics and conversation design principles.
  • Identify incorrect answers, inconsistencies and recurring error patterns.
  • Calculate accuracy percentages and monitor performance trends over time.
  • Investigate areas where chatbot quality is declining and pinpoint likely causes.
  • Provide clear, actionable feedback to Engineering and product partners.
  • Use AI tools to streamline evaluation and quality-assurance workflows.
  • Build simple productivity tools or automations with AI-assisted tools (e.g., Claude Code) when helpful.
Required Experience/Skills
  • 3-5 years of experience in conversation design, chatbot design, conversational AI, AI/LLM evaluation or closely related work (strong candidates with less experience will be considered).
  • Hands-on experience evaluating chatbot or AI responses for accuracy and relevance.
  • Experience working with structured content, evaluation criteria or quality rubrics.
  • Comfortable reviewing large datasets and identifying patterns and trends.
  • Ability to calculate basic metrics (percentages, accuracy rates) and analyze trends.
  • Strong analytical and logical reasoning skills.
  • Ability to understand and consistently apply existing design principles and guidelines.
  • Exceptional attention to detail.
  • Comfortable learning and using AI-powered tools in daily work.
  • Clear written communication for documenting findings and giving feedback.
  • Able to work Pacific…
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary