Intermediate · 1 day
Evaluating and Testing LLM Applications
A one-day course on evaluating non-deterministic systems: golden datasets, semantic checks, LLM-as-judge, retrieval and agent evaluation, and the pipeline that keeps quality from drifting.
1 day · Intermediate · AI Engineering · AI developers, QA engineers and technical leads
Overview
Conventional software testing assumes that the same input should produce the same output. LLM applications violate that assumption while still needing regression control, release judgement and production monitoring. Teams that skip this work discover quality issues only through users.
This course teaches a practical evaluation discipline. You separate deterministic tests from semantic evaluation, build golden datasets, and use human review and LLM-as-judge where they are justified. Retrieval is evaluated separately from generation. Groundedness, hallucination, agent behaviour and prompt or model changes are treated as first-class concerns.
The day is a build. You construct an evaluation suite around a realistic application path, then look at how that suite becomes a pipeline: versioning, model comparison, regression gates and production monitoring.
You leave able to say whether a change made the system better, worse, or merely different.
Audience
AI developers, QA engineers, ML engineers and technical leads responsible for quality, release and monitoring of LLM applications.
Prerequisites
Experience building or testing software. Familiarity with LLM applications is expected. You do not need a research background in evaluation metrics.
Duration
1 day
Can be extended with a second day to instrument a customer system or to deepen agent evaluation.
Learning outcomes
- 01
Explain why conventional tests are necessary but insufficient for LLM applications.
- 02
Design golden datasets that cover the behaviours the organisation actually cares about.
- 03
Implement deterministic checks, semantic evaluation and targeted human review.
- 04
Evaluate retrieval, groundedness and hallucination separately from fluency.
- 05
Compare prompts and models with a regression set rather than with anecdotes.
- 06
Evaluate agent traces: tool choice, loops, permissions and stopping conditions.
- 07
Design an evaluation pipeline that can run in development and in production monitoring.
Outline
- 01
What “correct” means for a non-deterministic system
- The failure of snapshot equality, and the remaining role of deterministic tests.
- Fluency versus usefulness, groundedness, safety and task completion.
- Choosing what to measure before choosing a scorer.
- Exercise: define pass/fail for a realistic assistant or extraction task.
- 02
Datasets and evaluators
- Golden datasets: sources, coverage, versioning and the cost of poor labels.
- Deterministic tests: schemas, citations present, refusals, and forbidden content.
- Semantic evaluation, LLM-as-judge, and when human evaluation is still required.
- Lab: build a small suite with mixed deterministic and semantic checks.
- 03
Retrieval, agents and change
- Retrieval evaluation: recall, ranking, and “did we fetch the right evidence?”
- Groundedness and hallucination detection on generated answers.
- Agent evaluation: tool selection, argument validity, loops and human handoff.
- Prompt versioning, model comparison and regression gates.
- Lab: add retrieval and agent traces to the suite and make a release judgement.
- 04
Pipelines and production monitoring
- Online versus offline evaluation, sampling and review queues.
- Dashboards: quality, cost, latency, retrieval misses and user corrections.
- Where evaluation belongs in CI, release and operations.
- Lab: sketch the pipeline for the system the team will actually run.
Practical work
You build an evaluation suite around a representative application: a golden set, mixed checkers, retrieval and generation scores, and a simple regression report. The closing work is a pipeline design that could be implemented on your own system.
Takeaways
- A working evaluation suite
- Templates for golden datasets and score definitions
- A regression report format for prompt and model changes
- A pipeline sketch for CI and production monitoring
Delivery
Cognitecta delivers private corporate training, on-site or as remote live training. Courses can be run as published, or adapted to your organisation’s stack, domain and experience level.
Instructor
Nicholas Johnson, AI architect and software engineer. He has a degree in Artificial Intelligence and around twenty years of professional technology training, including hundreds of courses for engineering teams and large organisations. About.
Related courses
3 days · Intermediate
AI Engineering for Software Developers
A three-day programme for professional software engineers: how to build real applications with modern AI, from model APIs and RAG through agents, evaluation and production patterns.
2–3 days · Intermediate
Building Applications with Generative AI
A two-to-three-day build course for software engineers: take an AI-powered application from architecture through RAG, tools, evaluation and deployment, ending with an end-to-end system.
1 day · Intermediate
Securing Generative AI Applications
A one-day security course for developers, security engineers and architects: the specific risks created by LLM applications, and the controls that actually reduce them.