Intermediate · 1 day
Securing Generative AI Applications
A one-day security course for developers, security engineers and architects: the specific risks created by LLM applications, and the controls that actually reduce them.
1 day · Intermediate · AI Engineering · Developers, security engineers and architects
Overview
LLM applications create attack surface that conventional application security only partly covers. Prompt injection, indirect injection through retrieved documents, tool misuse, data leakage and excessive agency are not theoretical. They appear as soon as a model can read untrusted text or call a tool.
This course treats those risks as engineering problems. You work through threat scenarios: injected instructions, poisoned knowledge bases, over-privileged tools, unsafe output handling and the supply chain of models, prompts and plugins. Identity, permissions, sandboxing, audit and logging are covered as controls, not as a policy appendix.
The teaching is concrete. Each major risk is paired with a mitigation that can be implemented in application and platform design: argument validation, allow-lists, output encoding, retrieval access control, human approval and least privilege.
You leave able to review an LLM application for the failures that matter, and to put proportionate controls in place before the system is exposed to untrusted input.
Audience
Developers, security engineers and architects responsible for designing, reviewing or operating LLM-powered applications.
Prerequisites
Familiarity with web or API security concepts is expected. Direct LLM implementation experience is helpful. The course does not require cryptography or ML research background.
Duration
1 day
Can be adapted to a threat-modelling workshop against a customer system.
Learning outcomes
- 01
Identify prompt injection, indirect injection and retrieval attacks in realistic application designs.
- 02
Diagnose insecure tool use, excessive agency and secrets exposure in agentic systems.
- 03
Design permission, identity and sandboxing controls around model-initiated actions.
- 04
Handle model output as untrusted data in downstream systems and user interfaces.
- 05
Evaluate model-provider, plugin and supply-chain risk as part of architecture review.
- 06
Specify logging, audit and human-approval requirements that are actually operable.
- 07
Apply a threat-modelling method to an LLM application and produce a mitigation list.
Outline
- 01
The threat model is different
- Where conventional OWASP thinking still applies, and where it does not.
- The model as an interpreter of untrusted natural language, including retrieved text.
- Data leakage through prompts, logs, providers and over-broad retrieval.
- Exercise: annotate a simple assistant architecture with trust boundaries.
- 02
Injection, retrieval and poisoned context
- Direct prompt injection against system instructions.
- Indirect injection via documents, tickets, email and web content.
- Retrieval attacks, poisoned documents and citation spoofing.
- Mitigations: isolation of untrusted content, allow-lists, dual-channel control, and retrieval ACL.
- Lab: exploit a weak RAG or chat path, then apply a control and re-test.
- 03
Tools, agency and output
- Insecure tool use: unconstrained arguments, SSRF-like patterns, and surprise side effects.
- Excessive agency: the model choosing actions the organisation did not intend.
- Secrets, identity, confused deputy problems and unsafe code execution.
- Output handling: XSS, markdown, generated links, and treating model text as data.
- Lab: constrain a tool-calling path with schemas, permissions and a human-in-the-loop step.
- 04
Platform, providers and operations
- Model-provider risk: retention, training, residency and contractual control.
- Supply chain: prompts, plugins, datasets, weights and evaluation models.
- Sandboxing, network egress, identity and environment separation.
- Audit, logging, red-team cases and a review checklist for release.
- Exercise: produce a mitigation plan for a customer-shaped system.
Practical work
You work through realistic threat scenarios on intentionally weak applications, then apply controls. The day includes at least one injection and one tool-permission exercise, plus a structured review of a more complete design. This is not an attack cookbook for third-party systems: the work is on designs your team can change.
Takeaways
- A threat model template for LLM applications
- Mitigation patterns for injection, retrieval, tools and output handling
- A security review checklist
- Logging and approval requirements that can be implemented
Delivery
Cognitecta delivers private corporate training, on-site or as remote live training. Courses can be run as published, or adapted to your organisation’s stack, domain and experience level.
Instructor
Nicholas Johnson, AI architect and software engineer. He has a degree in Artificial Intelligence and around twenty years of professional technology training, including hundreds of courses for engineering teams and large organisations. About.
Related courses
3 days · Intermediate
AI Engineering for Software Developers
A three-day programme for professional software engineers: how to build real applications with modern AI, from model APIs and RAG through agents, evaluation and production patterns.
2–3 days · Intermediate
Building Applications with Generative AI
A two-to-three-day build course for software engineers: take an AI-powered application from architecture through RAG, tools, evaluation and deployment, ending with an end-to-end system.
1 day · Intermediate
Evaluating and Testing LLM Applications
A one-day course on evaluating non-deterministic systems: golden datasets, semantic checks, LLM-as-judge, retrieval and agent evaluation, and the pipeline that keeps quality from drifting.