Decide between prompting, retrieval, rules, and training while engineering memory, versioned prompts, and safe product behavior.

AI data and product system labs

These labs cover exercises 12, 15, and 19.

Prerequisites and setup

Know Python data structures, JSON files, hashing, and basic experiment tracking. Create a local project with prompts/, tests/, and results/ directories. Use synthetic conversations and non-sensitive documents. Define retention, deletion, human review, and rollback rules before storing any memory or changing a prompt.

12. Fine-tuning versus prompting decision exercise

Given a task and dataset, compare rules, prompting, few-shot examples, RAG, and fine-tuning. Decide using privacy, dataset size, update frequency, cost, latency, stability, explainability, and accuracy.

decision = {
    "task_changes_weekly": True,
    "private_knowledge_changes": True,
    "labeled_examples": 80,
    "strict_format": True,
}

if decision["private_knowledge_changes"]:
    recommendation = "retrieval"
elif decision["labeled_examples"] < 100:
    recommendation = "prompting or few-shot"
else:
    recommendation = "benchmark prompting before considering training"

The output is a decision record, not a claim that one method is universally best.

15. Prompt version control

Store prompts as files with tests:

prompts/
  summarizer_v1.txt
  summarizer_v2.txt
tests/
  summarizer_cases.json
results/
  summarizer_v2.json

from pathlib import Path
import hashlib

prompt = Path("prompts/summarizer_v2.txt").read_text()
prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()[:12]

For every change, record the reason, expected improvement, evaluation score, regressions, and rollback choice.

19. Conversation memory experiment

Compare no memory, full history, summarized history, user-profile memory, and retrieval memory. Measure continuity, context use, stale facts, contradictions, privacy exposure, and forgetting.

def keep_recent(messages, limit=6):
    return messages[-limit:]

def memory_record(kind, messages, facts):
    return {"kind": kind, "message_count": len(messages), "facts": facts}

Never store sensitive student information as an experiment shortcut. Provide deletion and inspection controls.

Product review template

intended user:
task:
non-goals:
data sources:
failure that must be prevented:
human review:
privacy and retention:
success metric:
rollback plan:

Completion checklist

  • At least three implementation strategies were compared.
  • Prompt changes have version IDs and regression results.
  • Memory data can be inspected and deleted.
  • Product limits are visible to users.
  • The system has a rollback or safe-disable path.

References