Test AI systems against ambiguity, untrusted content, prompt injection, data leakage, and unsafe tool behavior.

AI reliability and security labs

These labs cover exercises 6, 13, 16, and 17. They are defensive experiments. Test only local systems and harmless inputs.

Prerequisites and setup

Know Python validation, exceptions, JSON, and basic logging. Use a local prototype with synthetic or public data; do not test against a real person, school system, or production service. Create a virtual environment, set timeouts, and keep secrets in environment variables. Write the intended behavior and safe failure behavior before generating test cases.

6. Prompt robustness and adversarial inputs

Procedure

  1. Define the intended behavior and an invariant that must never change.
  2. Create transformed inputs: whitespace, capitalization, typos, long irrelevant context, contradictions, and ambiguous wording.
  3. Add benign prompt-injection strings inside untrusted documents.
  4. Run every input through the same system.
  5. Label failures by type and add each fixed failure to a regression set.
def preserve_invariant(result):
    required = {"answer", "source"}
    return required.issubset(result) and isinstance(result["source"], list)

Never concatenate untrusted text into instructions without delimiters and clear trust boundaries.

13. AI safety red-team exercise

Create a table:

test_id | input | expected_behavior | actual_behavior | risk | mitigation | regression_test

Test secret requests, unsafe instructions, private-data extraction, overconfident answers, hidden document instructions, and excessive tool permissions. Report the risk, not exploit details. A successful defense is a safe response, refusal, or escalation to a person.

16. Data poisoning and retrieval contamination

Build a small document set with one misleading document and one authoritative document. Compare naive retrieval, source ranking, trusted-source filtering, conflict detection, and citation requirements.

def filter_sources(chunks, allowed_domains):
    return [chunk for chunk in chunks if chunk["domain"] in allowed_domains]

The system must surface conflicts instead of silently choosing a convenient claim.

17. AI application failure investigation

Start from a deliberately unsafe local prototype and identify:

  • missing input validation;
  • unbounded context;
  • no timeout;
  • no retry limit;
  • no output validation;
  • secret leakage;
  • missing citations;
  • missing logs;
  • excessive tool permissions.

For each issue, record impact, minimal fix, regression test, and residual risk. Do not add destructive capabilities to the prototype.

Threat-model template

asset:
trust boundaries:
untrusted inputs:
allowed actions:
high-impact failures:
preventive controls:
detection and logging:
human approval points:

Completion checklist

  • Tests use synthetic or public data.
  • Every failure has a reproducible test case.
  • Permissions are least-privilege.
  • Secrets are loaded from the environment, never printed.
  • The report distinguishes likelihood from impact.
  • A person reviews high-impact decisions.

References