AI Campus • Practical resources
Build an offline AI-output regression evaluator
Score saved answers against required and forbidden phrases, then use the exit code to catch regressions before publishing a prompt change.
Beginner · 60–90 minutes
Before you start
- Python 3.10 or newer.
- Saved candidate answers you can share locally; start with the sample below.
Cost: No model calls, accounts, API fees, or additional Python packages.
Step-by-step walkthrough
1. Define a small evaluation set
Create cases.json containing [{"id":"capital","expected":["Paris"],"forbidden":["London"]}]. Each case needs a unique ID, a non-empty expected list, and an optional forbidden list.
2. Save candidate outputs
Create answers.json containing {"capital":"Paris"}. In a real project, save outputs from your candidate prompt separately. This evaluator never calls a model for you.
3. Run the evaluator
Save the complete program as evaluate.py. Run: python evaluate.py cases.json answers.json. Expect passed: 1, total: 1, passRate: 1.0 and exit code 0.
4. Reproduce a regression
Change the answer to “London”. Run the same command. Expect passed: 0 and exit code 1. Empty suites or malformed data return exit code 2.
5. Compare two prompt versions
Keep answers-baseline.json and answers-candidate.json. Evaluate both against the same cases. Add adversarial and unsupported-input cases before claiming a prompt improved.
Complete code: evaluate.py
Download Python fileimport argparse
import json
from pathlib import Path
def main():
parser = argparse.ArgumentParser(description="Evaluate saved AI outputs offline")
parser.add_argument("cases")
parser.add_argument("answers")
args = parser.parse_args()
try:
cases = json.loads(Path(args.cases).read_text(encoding="utf-8-sig"))
answers = json.loads(Path(args.answers).read_text(encoding="utf-8-sig"))
if not isinstance(cases, list) or not cases or not isinstance(answers, dict):
raise ValueError("Use a non-empty case list and an answer object.")
seen = set()
results = []
for case in cases:
if not isinstance(case, dict):
raise ValueError("Each case must be an object.")
key = case.get("id")
expected = case.get("expected")
forbidden = case.get("forbidden", [])
if not isinstance(key, str) or not key.strip() or key in seen:
raise ValueError("Case IDs must be unique non-empty strings.")
if not isinstance(expected, list) or not expected:
raise ValueError("Each case needs expected phrases.")
if not isinstance(forbidden, list):
raise ValueError("Forbidden phrases must be a list.")
if any(not isinstance(s, str) or not s.strip() for s in expected + forbidden):
raise ValueError("Phrases must be non-empty strings.")
seen.add(key)
answer = answers.get(key)
text = answer.casefold() if isinstance(answer, str) else ""
missing = [s for s in expected if s.casefold() not in text]
violations = [s for s in forbidden if s.casefold() in text]
passed = isinstance(answer, str) and not missing and not violations
results.append({"id": key, "passed": passed, "missing": missing, "forbiddenFound": violations})
except (OSError, ValueError, TypeError, UnicodeError) as error:
parser.exit(2, f"Invalid evaluation input: {error}\n")
passed = sum(row["passed"] for row in results)
print(json.dumps({"total": len(results), "passed": passed,
"passRate": passed / len(results), "results": results}, indent=2))
return 0 if passed == len(results) else 1
if __name__ == "__main__":
raise SystemExit(main())
Run and verify
python evaluate.py cases.json answers.jsonJSON report with total, passed, passRate and per-case checks. Exit 0 means all cases passed, 1 means regressions, and 2 means invalid input.
Understand the limits
Phrase checks are deterministic smoke tests, not a measure of truth, meaning, safety, or complete model quality. Add human review and task-specific checks.