SWE-Bench Task Auditor
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to… Pay as listed: $70–$90/hr. Remote (United States).
- Pay (as listed on Mercor)
- $70–$90/hr · Hourly
- Commitment
- 40 hrs/week
- Location
- Remote · United States
- Languages
- English
- Category
- Coding
- Posted
- Aug 29, 2026
Job description
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
Basic Qualifications • 3+ years professional software engineering • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles) • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar repository benchmarks • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) • Prior code-review or task-grading experience
How this role compares
This role’s top listed rate is $90/hr, above the $75/hr median for the 172 open Coding roles that list hourly pay (8 in 10 top out between $35 and $140/hr).
Across Mercor’s 344 roles with hourly pay, the median is $90/hr, so this one is the same as it.
302 Coding roles are open on this board, from 10 platforms:
- Turing (104)
- SME Careers (57)
- Mercor (55)
- micro1 (39)
- AfterQuery (33)
- Terac (9)
- Ethos (2)
- Handshake AI (1)
- Meridial (1)
- Outlier (1)
Mercor’s listings were last checked on Nov 22, 2026.
Apply on Mercor — this is a referral link; we may receive a reward if you’re hired, at no cost to you.
Listed on Mercor. Details can change, so confirm on Mercor before applying.
Similar roles
- Kubernetes Task Auditor — Mercor · $70–$90/hr
- AI Developer Trace Task Auditor — Mercor · $70–$90/hr
- SWE Bench Engineer - Rust — Turing · $25/task
- SWE Bench Engineer - C# — Turing · $25/task