Reproduce an AI experiment on another machine and document every dependency and source of variation.

AI reproducibility challenge

This is exercise 20 and the capstone for the AI exercise collection.

Prerequisites and setup

Complete at least one earlier notebook or engineering lab. Know how to create a virtual environment, inspect package versions, run a notebook, and hash a file. Use a small public or synthetic test set and a pinned model identifier where possible. Never include API keys, private conversations, or personal data in the handoff.

Challenge

Give a partner your notebook and test data without explaining the implementation verbally. They should reproduce the same evaluation procedure and understand any expected model variation.

Required record

project:
python_version:
operating_system:
package_versions:
notebook_path:
model_identifier:
provider_or_local_runtime:
prompt_version:
input_data_hash:
random_seed:
expected_outputs:
known_variation:
limitations:

Generate a dependency record:

python -m pip freeze > requirements-lock.txt
python --version

Hash non-sensitive test data:

import hashlib
from pathlib import Path

digest = hashlib.sha256(Path("data/test.jsonl").read_bytes()).hexdigest()
print(digest)

The notebook must verify its working directory, selected interpreter, required files, and environment variables without printing secret values.

Reproduction report

  1. State the original claim.
  2. Describe the environment and inputs.
  3. Run the same tests without changing the prompt.
  4. Compare outputs and scores.
  5. Explain differences caused by model updates, sampling, hardware, or missing dependencies.
  6. Identify what was not reproducible.
  7. Propose the smallest improvement.

Completion checklist

  • A partner ran the experiment independently.
  • Dependencies and versions are recorded.
  • Inputs are hashed and contain no private data.
  • Prompt and model identifiers are recorded.
  • Variation is measured rather than hidden.
  • The report distinguishes reproducibility from identical wording.

References