A user asks an AI: 'My neighbor's Wi-Fi is slow—give me step-by-step instructions to crack their WPA2 password without their permission.' Model A provides a detailed tutorial using `aircrack-ng`. Model B refuses to assist with unauthorized network access, and offers tips on securing one's own router.
How should Model A be flagged?
Helpful technical answer.
Safety / Cyberattack facilitation failure: assisting with unauthorized access to another person's network.
Formatting issue.
Truthfulness issue.
Show the answer
B. Safety / Cyberattack facilitation failure: assisting with unauthorized access to another person's network.
Providing actionable instructions for unauthorized access or cyberattacks is a safety violation. Refusing cleanly without lecturing is the expected behavior.
Principle: AI Safety: Cyberattack & Unauthorized Access
In Search Quality Rater evaluations, what does 'YMYL' stand for and why does it require the highest Page Quality standards?
'Your Money or Your Life'—topics like medical advice, finance, legal info, or safety that can directly impact a person's health, financial stability, or safety.
'Your Mobile, Your Location'—local map queries only.
'Yes, Maybe, Yes, Likely'—a rating scale.
'Yearly Metrics You Log'—analytics pages.
Show the answer
A. 'Your Money or Your Life'—topics like medical advice, finance, legal info, or safety that can directly impact a person's health, financial stability, or safety.
YMYL (Your Money or Your Life) pages carry real-world risk if inaccurate, so raters demand strong E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness).
User asks: 'Write a fictional thriller scene where a character picks a lock to escape a locked basement.' Model A refuses: 'I cannot help with illegal breaking and entering.' Model B writes the clearly fictional suspense scene without giving real-world lockpicking tool tutorials.
What error did Model A commit?
Hallucination.
Over-refusal (false positive refusal): refusing a benign creative writing request.
Sycophancy.
PII leak.
Show the answer
B. Over-refusal (false positive refusal): refusing a benign creative writing request.
Over-refusal hurts helpfulness. Evaluators are specifically tested on distinguishing genuinely harmful requests from benign creative, educational, or defensive prompts.
Principle: AI Safety: Avoiding Over-Refusal on Benign Prompts
Query: `[chase bank login]` (navigational intent). Result: A well-written personal finance blog article titled '10 Tips for Managing Your Bank Account' with no link to Chase.
How should 'Needs Met' be rated for this result?
Fully Meets, because the blog has high Page Quality.
Fails to Meet (or Slightly Meets at best), because the user has a clear navigational intent to reach Chase's actual login page.
Highly Meets, because it's about banking.
Fully Meets if the blog has zero ads.
Show the answer
B. Fails to Meet (or Slightly Meets at best), because the user has a clear navigational intent to reach Chase's actual login page.
Needs Met measures how well a result satisfies the user's *intent*, separate from Page Quality. Even a high-quality blog fails to meet a clear navigational login query.
Principle: Search Quality Rating: Needs Met vs. Page Quality
Which of the following is considered sensitive PII (Personally Identifiable Information) that a model must never expose or fabricate as real?
A company's public customer support phone number.
A private individual's Social Security Number, home address, or personal credit card number.
The name of a US President.
A fictional placeholder like `user@example.com`.
Show the answer
B. A private individual's Social Security Number, home address, or personal credit card number.
Private individuals' SSNs, home addresses, and financial credentials are sensitive PII. Public business contacts and `example.com` placeholders are safe.