AI Evaluation Rubric Auditor
We are testing AI agents on realistic shopping and delivery tasks. Pay as listed: $60–$85/hr. Remote (Latin America, Canada, United States).
- Pay (as listed on Mercor)
- $60–$85/hr · Hourly
- Commitment
- 15 hrs/week
- Location
- Remote · Latin America, Canada, United States, United Kingdom
- Languages
- English
- Category
- Generalist
- Posted
- Nov 17, 2026
Job description
The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
- Was the information available?
- Does the rubric accurately grade what the prompt asked for?
- Would a valid answer have been marked wrong?
- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
- Experience providing feedback to writers/reviewers
- Consistent availability
- Can read a long agent transcript and find where it went wrong.
- Precise writing, and you check the data instead of trusting the reference answer.
How this role compares
This role’s top listed rate is $85/hr, above the $70/hr median for the 139 open Generalist roles that list hourly pay (8 in 10 top out between $15 and $115/hr).
Across Mercor’s 344 roles with hourly pay, the median is $90/hr, so this one is below it.
152 Generalist roles are open on this board, from 9 platforms:
- AfterQuery (58)
- micro1 (39)
- Turing (27)
- Mercor (15)
- Terac (6)
- Meridial (3)
- Outlier (2)
- Handshake AI (1)
- SME Careers (1)
Mercor’s listings were last checked on Nov 22, 2026.
Apply on Mercor — this is a referral link; we may receive a reward if you’re hired, at no cost to you.
Listed on Mercor. Details can change, so confirm on Mercor before applying.
Similar roles
- Architectural & Interior Visualization Specialist (AI Evaluation) — Mercor · $70–$120/hr
- Brand & Advertising Art Director (AI Evaluation) — Mercor · $70–$120/hr
- Technical Task Auditor — Meridial · $60/hr
- Senior Internal Auditor (SOX and IT Audit) — AfterQuery · $80/hr