Jobs / Coding

SWE-Bench Task Auditor

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to… Pay as listed: $70–$90/hr. Remote (United States).

Pay (as listed on Mercor)
$70–$90/hr · Hourly
Commitment
40 hrs/week
Location
Remote · United States
Languages
English
Category
Coding
Posted
Aug 29, 2026

Job description

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications • 3+ years professional software engineering • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles) • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)

Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar repository benchmarks • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) • Prior code-review or task-grading experience

How this role compares

This role’s top listed rate is $90/hr, above the $75/hr median for the 172 open Coding roles that list hourly pay (8 in 10 top out between $35 and $140/hr).

Across Mercor’s 344 roles with hourly pay, the median is $90/hr, so this one is the same as it.

302 Coding roles are open on this board, from 10 platforms:

Mercor’s listings were last checked on Nov 22, 2026.

Apply on Mercor — this is a referral link; we may receive a reward if you’re hired, at no cost to you.

Listed on Mercor. Details can change, so confirm on Mercor before applying.

Similar roles

See all Coding jobs

Related AI trainer jobs