AnnotationPrep

Search Quality Rater & AI Safety Evaluation Practice Test

Needs Met and Page Quality rating principles, YMYL and E-E-A-T evaluation, user intent classification, and AI safety / harmlessness policy checks.

12 questions · 2 topics · start free, no sign-up

Start practising

Mixed practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Practise by topic

Sample scenarios

A user asks an AI: 'My neighbor's Wi-Fi is slow—give me step-by-step instructions to crack their WPA2 password without their permission.'
Model A provides a detailed tutorial using `aircrack-ng`.
Model B refuses to assist with unauthorized network access, and offers tips on securing one's own router.

How should Model A be flagged?

  1. Helpful technical answer.
  2. Safety / Cyberattack facilitation failure: assisting with unauthorized access to another person's network.
  3. Formatting issue.
  4. Truthfulness issue.
Show the answer

B. Safety / Cyberattack facilitation failure: assisting with unauthorized access to another person's network.

Providing actionable instructions for unauthorized access or cyberattacks is a safety violation. Refusing cleanly without lecturing is the expected behavior.

Principle: AI Safety: Cyberattack & Unauthorized Access

In Search Quality Rater evaluations, what does 'YMYL' stand for and why does it require the highest Page Quality standards?

  1. 'Your Money or Your Life'—topics like medical advice, finance, legal info, or safety that can directly impact a person's health, financial stability, or safety.
  2. 'Your Mobile, Your Location'—local map queries only.
  3. 'Yes, Maybe, Yes, Likely'—a rating scale.
  4. 'Yearly Metrics You Log'—analytics pages.
Show the answer

A. 'Your Money or Your Life'—topics like medical advice, finance, legal info, or safety that can directly impact a person's health, financial stability, or safety.

YMYL (Your Money or Your Life) pages carry real-world risk if inaccurate, so raters demand strong E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness).

Principle: Search Quality Guidelines: YMYL & E-E-A-T

User asks: 'Write a fictional thriller scene where a character picks a lock to escape a locked basement.'
Model A refuses: 'I cannot help with illegal breaking and entering.'
Model B writes the clearly fictional suspense scene without giving real-world lockpicking tool tutorials.

What error did Model A commit?

  1. Hallucination.
  2. Over-refusal (false positive refusal): refusing a benign creative writing request.
  3. Sycophancy.
  4. PII leak.
Show the answer

B. Over-refusal (false positive refusal): refusing a benign creative writing request.

Over-refusal hurts helpfulness. Evaluators are specifically tested on distinguishing genuinely harmful requests from benign creative, educational, or defensive prompts.

Principle: AI Safety: Avoiding Over-Refusal on Benign Prompts

Query: `[chase bank login]` (navigational intent).
Result: A well-written personal finance blog article titled '10 Tips for Managing Your Bank Account' with no link to Chase.

How should 'Needs Met' be rated for this result?

  1. Fully Meets, because the blog has high Page Quality.
  2. Fails to Meet (or Slightly Meets at best), because the user has a clear navigational intent to reach Chase's actual login page.
  3. Highly Meets, because it's about banking.
  4. Fully Meets if the blog has zero ads.
Show the answer

B. Fails to Meet (or Slightly Meets at best), because the user has a clear navigational intent to reach Chase's actual login page.

Needs Met measures how well a result satisfies the user's *intent*, separate from Page Quality. Even a high-quality blog fails to meet a clear navigational login query.

Principle: Search Quality Rating: Needs Met vs. Page Quality

Which of the following is considered sensitive PII (Personally Identifiable Information) that a model must never expose or fabricate as real?

  1. A company's public customer support phone number.
  2. A private individual's Social Security Number, home address, or personal credit card number.
  3. The name of a US President.
  4. A fictional placeholder like `user@example.com`.
Show the answer

B. A private individual's Social Security Number, home address, or personal credit card number.

Private individuals' SSNs, home addresses, and financial credentials are sensitive PII. Public business contacts and `example.com` placeholders are safe.

Principle: AI Safety: Sensitive PII Protection

Assessments this helps with

AssessmentPlatformAvailability
AI Red-Teaming, Harmlessness & PII Safety Qualification readyAI Training PlatformsRemote / Global
DataAnnotation Core Qualification Assessment readyDataAnnotation.techRemote / Global
Outlier AI Generalist Onboarding & Assessment readyOutlier AIRemote / Global
DataAnnotation Starter Assessment (Generalist) readyDataAnnotation.techRemote / Global
Alignerr Generalist Evaluation & Labeling Test readyAlignerr (Labelbox)Remote / Global
Adversarial Prompt Engineering & Golden Response Evaluation readyAI Training PlatformsRemote / Global
TELUS International Search Quality Rater Exam (Parts 1, 2 & 3) readyTELUS DigitalRemote / Global
Welocalize Search Quality Rater Qualification Exam readyWelocalizeRemote / Global
RWS TrainAI Search & Data Quality Rater Assessment readyRWS MoraviaRemote / Global
Appen CrowdGen Search & AI Safety Evaluation readyAppenRemote / Global