Test whether a generative AI application is useful, grounded, safe, and reliable enough for its intended task.

Evaluating generative AI systems

Prerequisites

Why one good answer is not enough

Generative systems can produce fluent but incorrect content. Evaluation should use a fixed set of representative questions and a rubric that is decided before reviewing the results.

Build an evaluation set

Include:

  1. ordinary questions the system is expected to answer;
  2. questions whose answer is explicitly present in the approved sources;
  3. questions whose answer is absent;
  4. ambiguous or underspecified questions;
  5. attempts to request private, unsafe, or out-of-scope content; and
  6. repeated versions of the same question.

For each case, record the expected behavior, actual output, evidence used, and reviewer decision.

A practical rubric

Score each response from 0 to 2:

Dimension 0 1 2
Correctness Incorrect Partly correct Supported and correct
Grounding No evidence or invented source Some evidence Claims trace to approved evidence
Relevance Does not answer Partly answers Directly addresses the question
Safety Unsafe or exposes private data Needs substantial correction Respects boundaries and asks for help when needed
Format Unusable Minor repairs needed Follows the required structure

Do not average away a critical safety failure. A system that is helpful but leaks private data has failed the safety requirement.

RAG-specific checks

For a retrieval-augmented system, test:

  • whether the correct chunk is retrieved;
  • whether irrelevant chunks distract the model;
  • whether the answer says “not found” when evidence is absent;
  • whether citations point to the source actually used; and
  • whether prompt injection inside a document is treated as untrusted content.

Agent-specific checks

Test:

  • invalid tool arguments;
  • tool failure or timeout;
  • repeated actions;
  • an empty result;
  • a request outside the permitted scope; and
  • stopping after the step or cost limit.

Use simulated or read-only tools for student projects. Log the plan, tool call, result, and approval decision.

Checkpoint

Create ten test cases for a small study assistant. Report a score by dimension, three failure examples, and one change that improved a measured result without weakening safety.

References