Limited License Master Social Worker

Job Title:

LLM – AI Quality Analyst (Personalization) – English

Location:

Remote

Employment Type:

Full-Time (Hourly Based)
Payment Model:  Hourly payout
Experience

Required:

1+ Year
Shift Timing:  7:30 PM – 12:30 AM IST (Remaining 4 hours flexible)
Overlap Requirement:  Minimum 4 hours overlap with PST timezone
Total Shift Duration:  8 Hours (Full-Time Availability Mandatory)

Role Overview
We are hiring an  LLM – AI Quality Analyst (Personalization)  to evaluate and enhance personalized AI interactions within Google Gemini.
In this role, you will assess how effectively the AI model uses personal data (Gmail, Google Search history, You Tube activity, and prior Gemini conversations) to deliver relevant, grounded, and helpful responses.
This role requires a combination of:
Analytical thinking
Creative prompt engineering
Structured evaluation skills
Strong written communication

Key Responsibilities
1️⃣ Prompt Design & Personalization Testing
Design and execute 1–5 turn conversational prompts based on your own personal context.
Test how effectively the AI leverages personal information.
Simulate real-world personalization scenarios.
2️⃣ Model Response Evaluation
Evaluate responses on the following dimensions:
Grounding:  Are claims about you supported by real data?
Integration:  Is personal data naturally woven into the response?
Helpfulness:  Is the output practical and relevant?
Over-narration:  Is the response robotic or unnecessarily verbose?
3️⃣ Side-by-Side (SxS) Comparison
Compare two AI responses and rank the better one.
Identify subtle quality differences.
Write structured rationales referencing specific conversation turns.
4️⃣ Debug & Data Validation
Extract and verify debug information.
Confirm proper use of chat summaries and data sources.
Maintain strict data hygiene by deleting evaluation chats post-review.
5️⃣ Feedback & Annotation
Provide clear, structured annotations.
Identify incorrect personalization or hallucinations.
Suggest improvements where applicable.
Mandatory Requirements
High English proficiency (reading & writing).
Willingness to use your  primary personal Google account  (not a testing account).
Full-time availability (8 hours daily).
Minimum 4 hours overlap with PST timezone.
Desktop/Laptop with stable internet connection.
Ability to work independently in a remote environment.

Preferred Experience
Data annotation
AI model evaluation
Content moderation
Quality analysis
Prompt engineering
Required Skills
Exceptional analytical thinking.
Strong evaluation and comparison skills.
Meticulous attention to detail.
Clear and structured written communication.
Ability to identify flawed inferences and incorrect personalization.
Strong collaboration and communication skills.
Education
Bachelor’s Degree (BS/BA) or equivalent experience in:
Computer Science
Linguistics
Journalism
Policy / Law / Ethics
Or related analytical fields
Selection Process
Interest Check Form
Assessment Evaluation