AI evaluation and comparison labs
These labs cover exercises 1–5, 9–11, 14, and 18: model comparison, grading, evidence testing, RAG comparison, multimodal evaluation, embeddings, tokenization, cost dashboards, and local-versus-hosted models.
Prerequisites and setup
Know Python functions, JSON/CSV files, and basic pandas. Create a virtual environment, install only the client and measurement packages needed by your chosen experiment, and verify the interpreter with import sys; print(sys.executable). Use synthetic or public test data. If a hosted model is required, load its key from an environment variable and record the exact model ID without printing the key.
Experiment method
Begin with a rules-based or human-reviewed baseline. Write the test set, rubric, and metric definitions before calling a model. Keep the same inputs and prompt settings across alternatives, save one structured record per case, and inspect failures rather than reporting only an average score.
1. Model comparison laboratory
Goal
Compare at least two models on identical inputs and recommend a model for a defined use case.
Procedure
- Choose one task, such as extracting fields from a science paragraph.
- Create 20 test cases: 12 normal, 4 ambiguous, and 4 cases that should be refused or marked uncertain.
- Define a rubric before running models:
- correctness: 0–2
- required fields present: 0–2
- evidence or explanation: 0–2
- safety/abstention: 0–2
- format validity: 0–2
- Use the same prompt, temperature, input, and output schema.
- Save model ID, timestamp, latency, input, output, score, and error type.
- Inspect five failures from each model.
- Write a recommendation with quality, speed, cost, privacy, and safety trade-offs.
import json
import time
from pathlib import Path
def run_case(client, model, prompt):
started = time.perf_counter()
response = client.messages.create(
model=model,
max_tokens=300,
temperature=0,
messages=[{"role": "user", "content": prompt}],
)
return {
"model": model,
"latency_ms": round((time.perf_counter() - started) * 1000, 1),
"output": response.content[0].text,
}
results = []
for model in MODELS:
for case in TEST_CASES:
result = run_case(client, model, case["prompt"])
results.append({**case, **result})
Path("results").mkdir(exist_ok=True)
Path("results/model-comparison.json").write_text(json.dumps(results, indent=2))
Deliverable
Submit a score table, five inspected failures per model, a model card, and a recommendation that names the conditions under which it is valid.
2. Prompt evaluation and grading
Create a dataset with input, expected_behavior, reference_answer, and rubric. Compare exact-match, rule-based, human, and model-assisted grading. Validate the grader on examples where the correct answer is known.
def grade_contains(answer, required_terms):
answer_lower = answer.lower()
return sum(term.lower() in answer_lower for term in required_terms) / len(required_terms)
def grade_record(record):
return {
"rule_score": grade_contains(record["answer"], record["required_terms"]),
"length_ok": 20 <= len(record["answer"]) <= 500,
}
Never treat a model grader as ground truth. Report agreement with human scores and disagreements.
3. Hallucination and evidence testing
Create answerable, unanswerable, incomplete, and conflicting evidence cases. Require the system to cite the supplied evidence or abstain.
Measure:
- supported claims / total claims;
- citation correctness;
- unsupported claims;
- correct abstentions;
- evidence coverage.
def evidence_check(answer, evidence):
return {
"mentions_evidence": any(
sentence.strip() in answer
for sentence in evidence.split(".") if sentence.strip()
),
"answer": answer,
}
The important result is not confidence wording; it is whether claims can be traced to evidence.
4. RAG pipeline comparison
Use the existing chunking exercise as the starting point. Compare character, sentence, section, BM25, embedding, hybrid, and reranked retrieval on the same questions.
def recall_at_k(retrieved_ids, relevant_ids, k=3):
return int(bool(set(retrieved_ids[:k]) & set(relevant_ids)))
Record chunk ID, source document, retrieval score, rank, answer, and citations. Measure recall@k, precision@k, latency, faithfulness, and citation completeness.
5. Structured-output reliability
Ask for the same result as plain text, JSON, schema-constrained JSON, and a validated Python object. Test missing fields, wrong types, nulls, extra fields, and malformed JSON.
from pydantic import BaseModel, Field, ValidationError
class Book(BaseModel):
title: str
year: int = Field(ge=0)
topics: list[str]
def validate_output(raw):
try:
return {"valid": True, "value": Book.model_validate_json(raw).model_dump()}
except ValidationError as error:
return {"valid": False, "error": error.errors()}
Report valid-output rate, repair rate, and silent data-loss cases.
9. Multimodal AI evaluation
Use non-sensitive diagrams, charts, or public images. Ask the model to separate visible observations from inferences and uncertainty. Score text extraction, chart interpretation, comparison accuracy, and unsupported assumptions. Do not use student faces, private documents, medical images, or identifying data.
from pathlib import Path
import base64
def image_as_data_url(path):
encoded = base64.b64encode(Path(path).read_bytes()).decode("ascii")
return f"data:image/png;base64,{encoded}"
request = {
"image": image_as_data_url("data/public-chart.png"),
"instruction": (
"Return JSON with observations, inferences, and uncertainties. "
"Do not infer facts that are not visible."
),
}
Use a public chart or diagram, keep the image in the test fixture, and score observations separately from inferences.
10. Embeddings visualization
Create 20 student-written sentences, compute embeddings, normalize vectors, and compare cosine similarity. Visualize with PCA only after measuring nearest neighbors.
import numpy as np
def cosine(a, b):
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
Discuss semantic similarity, wording effects, clusters, outliers, and the limits of interpreting a two-dimensional plot.
11. Tokenization and context-window experiment
Measure characters, words, tokens, and estimated context usage for prose, code, punctuation-heavy text, and multiple languages. Compare truncation, summarization, and retrieval strategies. Never assume a character count equals a token count.
def text_stats(text, token_counter):
return {
"characters": len(text),
"words": len(text.split()),
"tokens": len(token_counter(text)),
}
def keep_context(text, max_tokens, token_counter):
tokens = token_counter(text)
return token_counter.decode(tokens[:max_tokens])
Record which text was removed and test whether the answer changed after truncation.
14. Cost, latency, and quality dashboard
Store one row per request:
record = {
"experiment_id": "model-comparison-001",
"model": model,
"prompt_version": "v3",
"input_tokens": usage.input_tokens,
"output_tokens": usage.output_tokens,
"latency_ms": latency_ms,
"quality_score": quality_score,
"safety_score": safety_score,
}
Plot quality versus latency and quality versus estimated cost. Do not select a model from one metric.
18. Local small model versus hosted model
Compare a local model, a hosted general model, a hosted specialized model, and a rules baseline. Include setup time, hardware, privacy, offline behavior, quality, latency, and operating cost. State which requirements make each option unsuitable.
options = [
{"name": "rules", "offline": True, "privacy": "high", "quality": None},
{"name": "local-small", "offline": True, "privacy": "high", "quality": None},
{"name": "hosted-general", "offline": False, "privacy": "provider-dependent", "quality": None},
]
def reject_if_requirement_fails(option, needs_offline, needs_private_data):
if needs_offline and not option["offline"]:
return "reject: requires network access"
if needs_private_data and option["privacy"] != "high":
return "reject: privacy review required"
return "candidate"
Do not make a local-versus-hosted claim from a single prompt. Use the same test set and report setup and maintenance costs.
Completion checklist
- Dataset and rubric were written before evaluation.
- A non-AI baseline was included.
- Results are saved in a structured file.
- At least five failures were inspected.
- Model IDs and prompt versions are recorded.
- Recommendation includes privacy, cost, latency, and safety.