A repeatable workflow for completing any project-based AI engineering exercise.

Advanced lab workflow

This is a workflow guide, not a second index. Choose the project from the single AI exercise index, then use this page to plan, run, and document it. The detailed module pages contain the exercise-specific explanation and code.

Before you begin

Prerequisites are Python functions, files, exceptions, basic data tables, and the ability to run a notebook or script. Read the index setup instructions before requesting a provider key. Use synthetic or public data unless a lesson explicitly says otherwise.

Experiment sequence

  1. Define the user, task, allowed data, unacceptable failure, and a measurable success criterion.
  2. Write a non-AI baseline before adding a model, retrieval system, tool, or agent.
  3. Keep test inputs separate from examples used to design the prompt.
  4. Pin the model identifier, prompt version, package versions, random seed, and date.
  5. Save structured JSON or CSV results; screenshots are supplementary evidence only.
  6. Score successes and failures using criteria written before the run.
  7. Inspect at least five failures manually and classify their causes.
  8. Record what the system must not do, who approves side effects, and how to disable it.
experiment = {
    "hypothesis": "A section-based splitter improves retrieval recall@3.",
    "baseline": "fixed-character chunks",
    "test_set": "data/questions.jsonl",
    "model": "record-the-exact-model-id",
    "prompt_version": "v1",
    "metrics": ["recall_at_3", "citation_completeness", "latency_ms"],
}

Required evidence

  • A short hypothesis and test-set description.
  • The runnable notebook or script and dependency record.
  • Inputs, outputs, scores, and error categories in JSON or CSV.
  • A rubric or metric definition written before evaluation.
  • At least five manually reviewed failures where the module requests them.
  • A conclusion that states whether the evidence supports the hypothesis and what remains uncertain.

Safety and privacy

Never commit tokens, private data, student records, faces, or model outputs containing personal information. Treat retrieved documents and tool arguments as untrusted input. Use least-privilege, read-only or simulated tools, timeouts, step limits, validation, and human approval before side effects.

References