{"html":"<h4>The work</h4>\n<p>We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:</p>\n<p>- Was the information available?</p>\n<p>- Does the rubric accurately grade what the prompt asked for?</p>\n<p>- Would a valid answer have been marked wrong?</p>\n<p>- Is the rubric structure correct per the guidelines?</p>\n<p>This role includes providing feedback to reviewers regarding prompt and rubric issues.</p>\n<h4>Must-haves</h4>\n<p>- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.</p>\n<p>- Experience providing feedback to writers/reviewers</p>\n<p>- Consistent availability</p>\n<p>- Can read a long agent transcript and find where it went wrong.</p>\n<p>- Precise writing, and you check the data instead of trusting the reference answer.</p>"}