AtrisBench
Back to the leaderboard

Claude Fable 5.1 on Apply superseding policy

Passed in 14s. Attempt 1. Run 20260904T131330Z-fable-5-1-policy-across-files-1.

Task

Answer the policy questions

Read questions.md, all six files under handbook/, and the dated file under memos/. Later material that explicitly supersedes an older rule controls. A later appendix table also controls over prose that it says it replaces.

Write answers.json with all six question ids:

{
  "q1": {
    "answer": "A concise answer",
    "file": "handbook/example.md",
    "section": "2.1"
  }
}

Use the exact relative source path and exact section label for the controlling rule. Do not use outside assumptions.

Command

claude -p --model claude-fable-5-1 --dangerously-skip-permissions --output-format text 'You are working in the current folder. Do the task in TASK.md and write the outputs to the exact paths it names.
Do not ask questions. When done, stop.

# Answer the policy questions

Read `questions.md`, all six files under `handbook/`, and the dated file under `memos/`.
Later material that explicitly supersedes an older rule controls. A later appendix table also
controls over prose that it says it replaces.

Write `answers.json` with all six question ids:

` ` `json
{
  "q1": {
    "answer": "A concise answer",
    "file": "handbook/example.md",
    "section": "2.1"
  }
}
` ` `

Use the exact relative source path and exact section label for the controlling rule. Do not use
outside assumptions.'

Output

Done. All six answers are written to answers.json and the file parses cleanly.

Two answers come from the July memo, which overrides the older handbook rules. PTO for three years of service is now 18 days, and security incidents must be reported within 4 hours.

The San Francisco dinner cap comes from the appendix table, which replaces the base $60 limit. It is $75 per person.

The other three came straight from the handbook: 20 consecutive business days abroad, dental after 30 days, and domestic flights booked 21 days ahead.

Files produced

answers.json

{
  "q1": {
    "answer": "18 PTO days per calendar year",
    "file": "memos/2026-07-01.md",
    "section": "1.1"
  },
  "q2": {
    "answer": "Within 4 hours of discovery, via the security hotline or the incident form",
    "file": "memos/2026-07-01.md",
    "section": "1.2"
  },
  "q3": {
    "answer": "$75 per attendee",
    "file": "handbook/expenses.md",
    "section": "Appendix A"
  },
  "q4": {
    "answer": "No more than 20 consecutive business days",
    "file": "handbook/remote-work.md",
    "section": "4.2"
  },
  "q5": {
    "answer": "After 30 calendar days from the start date",
    "file": "handbook/benefits.md",
    "section": "5.1"
  },
  "q6": {
    "answer": "At least 21 days before departure",
    "file": "handbook/travel.md",
    "section": "2.3"
  }
}

Checker

{"passed": true, "score": 1.0, "notes": "6/6 controlling sources and answers exact"}

Judge

Not used for this task.