Response Comparison Rater

The Response Comparison Rater role is a part-time, fully remote position focused on the quality and logic of written text. You will look at various outputs from language models and determine which ones communicate most effectively based on a specific set of guidelines. This work helps ensure that the tools being built are helpful, accurate, and free of the vague language that often clutters professional communication.

In this role, you will help us evaluate the performance of AI models by comparing their generated responses. Olive Tree Consulting values clarity and brevity, and this position supports that goal by identifying which machine-generated answers are truly useful and which are simply filler. You will be looking at prompts and assessing pairs of responses for factual accuracy, logical consistency, and tone.

This is an entry-level contract position that requires no previous experience in consulting or data science, but it does require a high level of comfort with the English language and a critical eye for detail. Since this work is conducted entirely online, you can complete your tasks from any location with a reliable internet connection.

• Review pairs of AI-generated responses to see which one better follows the user instructions.
• Rank responses based on a provided rubric that covers dimensions like truthfulness and clarity.
• Write short, clear justifications explaining why one specific response is superior to another.
• Identify and flag instances where an AI model provides incorrect information or makes up facts.
• Check for "hallucinations" or logical leaps that would make a response untrustworthy.
• Evaluate the formatting and structure of written outputs to ensure they follow professional standards.
• Provide feedback on whether the responses use plain English or rely on unnecessary jargon.
• Consistently meet quality targets by maintaining a high degree of focus across all rating tasks.

We need people who can read a text and quickly identify where it goes wrong. You should be able to spot slight contradictions or subtle shifts in tone that might escape an average reader. This role is a good fit for someone who enjoys writing and editing and who can describe their reasoning clearly. You do not need a specific degree, but you should have a strong grasp of grammar and a logical approach to problem-solving.

Because this is a remote, part-time position, reliability and the ability to manage your own schedule are essential traits. We value honest assessments over quick ones, so patience is a requirement.

• Previous experience in writing, editing, or proofreading (this can be academic or personal projects).
• Comfort using basic online productivity tools and navigating simple data entry interfaces.
• An interest in the way language works and how machines attempt to replicate human speech.
• The ability to follow complex, multi-step instructions without getting overwhelmed.
• A background in a field that requires close reading, such as law, history, philosophy, or English.
• Experience working in a remote layout where clear written communication is the primary way to stay in touch.

This is a flexible, remote position that allows for a steady flow of work. You will be working at your own pace within agreed timelines, making it a suitable option for those balancing other commitments. The position is paid, and the pay is competitive for the role and market. You will gain direct experience with the processes used to train modern language models and contribute to work that prioritizes clarity over corporate buzzwords.

While the work is independent, we provide clear documentation and support to help you understand the rating…