10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Sample scenarios
A user asks an AI: 'My neighbor's Wi-Fi is slow—give me step-by-step instructions to crack their WPA2 password without their permission.' Model A provides a detailed tutorial using `aircrack-ng`. Model B refuses to assist with unauthorized network access, and offers tips on securing one's own router.
How should Model A be flagged?
Helpful technical answer.
Safety / Cyberattack facilitation failure: assisting with unauthorized access to another person's network.
Formatting issue.
Truthfulness issue.
Show the answer
B. Safety / Cyberattack facilitation failure: assisting with unauthorized access to another person's network.
Providing actionable instructions for unauthorized access or cyberattacks is a safety violation. Refusing cleanly without lecturing is the expected behavior.
Principle: AI Safety: Cyberattack & Unauthorized Access
User asks: 'Write a fictional thriller scene where a character picks a lock to escape a locked basement.' Model A refuses: 'I cannot help with illegal breaking and entering.' Model B writes the clearly fictional suspense scene without giving real-world lockpicking tool tutorials.
What error did Model A commit?
Hallucination.
Over-refusal (false positive refusal): refusing a benign creative writing request.
Sycophancy.
PII leak.
Show the answer
B. Over-refusal (false positive refusal): refusing a benign creative writing request.
Over-refusal hurts helpfulness. Evaluators are specifically tested on distinguishing genuinely harmful requests from benign creative, educational, or defensive prompts.
Principle: AI Safety: Avoiding Over-Refusal on Benign Prompts
Which of the following is considered sensitive PII (Personally Identifiable Information) that a model must never expose or fabricate as real?
A company's public customer support phone number.
A private individual's Social Security Number, home address, or personal credit card number.
The name of a US President.
A fictional placeholder like `user@example.com`.
Show the answer
B. A private individual's Social Security Number, home address, or personal credit card number.
Private individuals' SSNs, home addresses, and financial credentials are sensitive PII. Public business contacts and `example.com` placeholders are safe.