Skip to content
Cognitecta

Training / Models & Machine Learning

Intermediate · 1 day

Running LLMs Locally

A one-day practical course on local and private inference: model formats, hardware, quantisation, engines, benchmarking and the privacy and cost reasons to run models yourself.

1 day · Intermediate · Models & Machine Learning · Developers, infrastructure and AI engineers

Talk to us

Overview

Local inference is no longer only a hobbyist concern. Teams run models on laptops, on-prem GPUs and private cloud for privacy, cost, latency or air-gapped work. The difficulty is operational: choosing a model size that fits VRAM, a format the engine can load, and a configuration that meets a latency target.

This course is practical. You work through model families, formats, sizes, quantisation, inference engines, CPU versus GPU, throughput, latency, context length and batching. You expose a local API, measure it, and discuss hardware selection and deployment.

Privacy and cost are treated as the usual reasons to do this work, not as slogans. The course is honest about quality loss from aggressive quantisation and about the gap between a laptop demo and a multi-user service.

You leave able to select, run, benchmark and expose a local model, and to decide when local inference is the right deployment rather than a preferred ideology.

Audience

Developers, infrastructure engineers and AI engineers who need to run or support local and private inference.

Prerequisites

Comfortable with the command line and with installing local tooling. A machine with enough RAM, and a GPU if available, makes the labs more realistic. No ML research background is required.

Duration

1 day

Can be extended with a day on serving at team scale, or focused on a particular engine and hardware estate.

Learning outcomes

  1. 01

    Select a model family, size and quantisation that fit a given hardware envelope.

  2. 02

    Run local inference with a current engine and expose it behind a local API.

  3. 03

    Measure latency, throughput, context length and quality trade-offs.

  4. 04

    Compare CPU and GPU paths, including batching and concurrent users.

  5. 05

    Design a private deployment that matches privacy, cost and operational constraints.

  6. 06

    Choose hardware with a defensible view of VRAM, context and expected load.

Outline

  1. 01

    Models, formats and hardware

    • Model families and the practical differences that affect local use.
    • Formats, quantisation and the quality/VRAM trade-off.
    • VRAM, RAM, CPU and what “it will run” actually means.
    • Lab: inventory the machine and choose a model that will fit.
  2. 02

    Running and serving

    • Inference engines, configuration, context length and batching.
    • Local APIs and making a model look like a service to application code.
    • CPU versus GPU, throughput versus latency.
    • Lab: run a model, expose an API, and send structured requests.
  3. 03

    Benchmarking and deployment

    • Measuring tokens per second, time to first token, and quality regressions from quantisation.
    • Privacy, cost and when local inference is cheaper than hosted APIs — including the cases where it is not.
    • Deployment: desktop, on-prem, private cloud, and the jump to multi-user serving.
    • Lab: benchmark two configurations and write a hardware recommendation.

Practical work

You install or use a local inference engine, load a quantised model, expose an API, and benchmark latency and throughput. You compare configurations and produce a short hardware and deployment recommendation.

Takeaways

  • A running local model and API
  • Benchmark notes for latency, throughput and quantisation
  • A hardware and model-selection checklist
  • Deployment options for private inference

Delivery

Cognitecta delivers private corporate training, on-site or as remote live training. Courses can be run as published, or adapted to your organisation’s stack, domain and experience level.

Instructor

Nicholas Johnson, AI architect and software engineer. He has a degree in Artificial Intelligence and around twenty years of professional technology training, including hundreds of courses for engineering teams and large organisations. About.

Related courses

  1. 2 days · Advanced

    Transformers from First Principles

    A two-day course for technically strong practitioners: build a usable understanding of transformer models by implementing simplified components, without unnecessary mathematical formalism.

  2. 2 days · Advanced

    Fine-Tuning Language Models

    A two-day course for ML engineers and experienced AI developers: when to adapt a model, how to do it with parameter-efficient methods, and how to evaluate and deploy the result.

  3. 2–3 days · Advanced

    Training Your Own Language Model

    A two-to-three-day course for advanced developers and ML engineers: build and train a small language model in order to understand the process, not to compete with frontier systems.

Talk to us