LLM Analyst

Remote (Work From Anywhere)

Experience:

3+ Years

About the Role
We are seeking a highly analytical and detail-oriented

LLM Analys t to evaluate, optimize, and improve the performance of Large Language Models (LLMs) and Generative AI systems. The ideal candidate will have experience in AI evaluation, prompt engineering, content quality analysis, research, data analysis, or AI operations.
As an LLM Analyst, you will play a key role in assessing model outputs, identifying performance gaps, analyzing user interactions, and providing actionable insights that enhance AI quality, accuracy, safety, and overall user experience.

Key Responsibilities
LLM Evaluation & Analysis
Evaluate responses generated by Large Language Models (LLMs) across diverse domains and use cases.
Assess outputs for accuracy, relevance, completeness, consistency, reasoning quality, and instruction adherence.
Identify hallucinations, factual inaccuracies, biases, and safety concerns.
Analyze model strengths, weaknesses, and performance trends.
Prompt Analysis & Optimization
Review and optimize prompts to improve AI-generated outcomes.
Conduct prompt testing across various scenarios and user journeys.
Develop prompt evaluation methodologies and best practices.
Recommend improvements to increase response quality and reliability.
AI Performance Monitoring
Track and analyze AI performance metrics and quality indicators.
Identify recurring issues and opportunities for model improvement.
Conduct root cause analysis on AI failures and inconsistencies.
Support benchmarking initiatives across different AI models and versions.
User Interaction & Behavioral Analysis
Analyze user interactions and feedback to understand model effectiveness.
Identify patterns in user behavior and common failure scenarios.
Recommend enhancements based on user needs and business objectives.
Support initiatives focused on improving user satisfaction and engagement.
Quality Assurance & Validation
Design and execute AI testing and validation activities.
Review AI-generated content against quality and compliance standards.
Support regression testing to validate model updates and improvements.
Participate in calibration exercises to maintain evaluation consistency.
Research & Insights
Conduct research on LLM advancements, industry trends, and emerging AI technologies.
Compare model performance against industry benchmarks and competitors.
Generate insights and recommendations to support AI strategy and development.
Assist in defining evaluation frameworks and quality standards.
Data Review & Dataset Quality
Review training and evaluation datasets for quality and relevance.
Validate human feedback, annotations, and evaluation results.
Identify data-related issues that may impact model performance.
Support continuous improvement of AI training and evaluation datasets.
Documentation & Reporting
Create reports, dashboards, and performance summaries.
Document findings, recommendations, and evaluation methodologies.
Present insights to product, engineering, research, and business teams.
Maintain knowledge repositories and AI performance documentation.
Cross-Functional Collaboration
Collaborate with AI Researchers, Product Managers, Engineers, Quality Analysts, and Operations Teams.
Translate analytical findings into actionable business and technical recommendations.
Support AI product launches, testing programs, and optimization initiatives.
Participate in AI strategy discussions and continuous improvement projects.

Required

Skills & Qualifications
Must Have
3+ years of experience in AI Evaluation, Quality Analysis, Business Analysis, Research, Data Analysis, Operations Analysis, Content Quality, or related fields.
Strong understanding of Large Language Models (LLMs), Generative AI, and conversational AI systems.
Experience evaluating AI-generated content for quality, accuracy, and compliance.
Strong analytical, problem-solving, and critical-thinking abilities.
Excellent written and verbal communication skills.
Experience interpreting data and generating actionable insights.
Ability to work independently in a remote environment.

Good to Have

Experience with Prompt Engineering and prompt optimization techniques.
Hands-on experience with AI platforms such as ChatGPT, Claude, and Gemini.
Familiarity with RLHF (Reinforcement Learning from Human Feedback) and AI evaluation methodologies.
Experience using Excel, Google Sheets, SQL, Power BI, Tableau, or reporting tools.
Knowledge of AI quality assurance, trust & safety, and content moderation principles.
Basic understanding of machine learning concepts and AI model development.