Local LLM Inference Architecture: Ollama, llama.cpp, GGUF, and Performance Testing

The core lesson from running Qwen3:4B locally was that model performance depends on an entire stack: the Agent and API, Ollama's service layer, an inference runtime, GGUF weights and quantization, and the CPU, GPU, and memory underneath. A few simple tests also showed that wait time depends on both generation speed and total output length, including thinking content.

I have a computer from 2024 with an AMD Ryzen 8040-series processor and 64 GB of RAM. For a few reasons, it is now semi-retired. I kept wondering what I could still do with a machine whose specifications remained fairly capable.

I use ChatGPT regularly and have worked with its API, but I had never run an LLM locally. I had encountered terms such as Ollama, GGUF, quantization, and GPU offloading, yet I could not clearly explain how they fit together or which layer each one belonged to.

I decided to use the computer for local models. ChatGPT suggested starting with Qwen3:4B, so that became my first experiment.

Once I began working with it, the question I wanted to answer became more specific:

After I enter a prompt locally, which layers does the AI system pass through before the answer finally comes back?

Ollama: A Local Model Runtime Platform

My initial understanding of Ollama was straightforward.

I thought of it as a model manager:

Run ollama pull
Download a model
Start it with ollama run and enter interactive mode
Ask the model questions locally

From a user's perspective, that understanding is not entirely wrong.

Ollama does handle many things for us, including:

Downloading and managing models
Starting a local inference service
Selecting model settings
Providing an HTTP API
Managing model loading and memory
Packaging low-level inference details behind a simpler interface

Ollama acts as a higher-level platform for running local models.

It brings model files, inference technology, parameter settings, API services, and the model lifecycle together so that users do not have to deal directly with all the low-level complexity.

llama.cpp: One Common Underlying Inference Technology

llama.cpp is a local large language model inference project implemented in C and C++.

It is one of the common underlying inference technologies used by Ollama, and it is also an important entry point for understanding the local GGUF inference ecosystem.

It handles lower-level work such as:

Loading model weights
Converting text into tokens
Creating and managing the KV cache
Running Transformer forward passes
Performing matrix operations on the CPU or GPU
Generating new tokens one at a time
Controlling sampling parameters such as temperature, top-p, and top-k

Ollama is more like an integrated media player, while llama.cpp is one of the core technologies that may handle decoding and data processing inside that player.

When I use Ollama, I usually do not need to operate llama.cpp directly.

Understanding that llama.cpp exists, however, let me see the layer beneath Ollama's abstractions for the first time.

I also began to understand that when a model is slow, GPU utilization is low, or memory runs out, the problem is not necessarily that the model itself is bad.

The bottleneck could be in several places:

The model is too large
The quantization format is a poor fit
The GPU does not have enough VRAM
Only some layers are offloaded to the GPU
The remaining computation still runs on the CPU
A long context consumes too much memory in the KV cache
Prompt processing and token generation have different bottlenecks
The inference engine does not fully support the current hardware

That shifted my attention from choosing a model to understanding the system.

GGUF: The Model File Format

Another term I saw repeatedly this week was GGUF.

When I first encountered a name like this:

Qwen3-4B-Q4_K_M.gguf

I found it easy to mistake the entire string for the model name.

Later, I learned how to break it down:

Qwen3 is the model family
4B roughly indicates the parameter count
Q4_K_M is a quantization method
.gguf is the model file format

GGUF is a file format for storing model weights and related metadata, and it is commonly used in the llama.cpp ecosystem.

It can contain:

Model weights
Tokenizer information
Model architecture information
Context-related settings
Special tokens
Quantized tensor data

These concepts can therefore be separated clearly:

Qwen3 is the model. GGUF is the format used to store and distribute it. llama.cpp is one inference technology that can load and run it, while Ollama is the higher-level platform that manages and serves it.

Quantization Makes Large Models Practical on Limited Hardware

Large models are commonly quantized so that they can run on ordinary computers.

Original model weights may be stored in FP16 or BF16. Quantization represents those weights with fewer bits, such as 8-bit, 6-bit, 5-bit, or 4-bit values.

The main goal is to reduce:

Model file size
RAM usage
VRAM usage
Memory bandwidth requirements

For example, the memory requirements of an FP16 version and a 4-bit quantized version of the same 4B model can differ substantially.

Quantization is not free, however.

More aggressive quantization usually creates a greater risk of losing some model accuracy. Different quantization methods also strike different balances among speed, memory use, and output quality.

Q4_K_M therefore represents an engineering choice:

How much precision am I willing to trade for lower memory requirements and a better chance of running the model locally?

This is one of the major differences between local and cloud models.

When I use a cloud API, I usually only need to choose a model name.

In a local environment, I also need to consider:

Which model size to use
Which quantization to use
How large to make the context window
How many layers to place on the GPU
How the CPU and GPU should divide the work
Whether the system has enough memory
How to balance speed and quality

The Complete Local Inference Flow

The following diagram divides a local AI system into its main layers:

flowchart TB
    U[User] --> A[Chat Interface or Agent<br/>Hermes / OpenClaw]
    A --> API[OpenAI-compatible API]
    API --> O[Ollama<br/>Model Management and Service Layer]
    O --> R[Inference Runtime<br/>For Example, llama.cpp]
    R --> M[Qwen3 Model<br/>GGUF / Quantized Weights]

    subgraph H[Local Hardware]
        CPU[CPU]
        GPU[GPU]
        RAM[RAM]
        VRAM[VRAM]
    end

    M --> R
    CPU --> R
    GPU --> R
    RAM --> R
    VRAM --> R

Every layer, from the top-level Agent to the CPU and GPU underneath, has a different job. Ollama is not the model itself. It is the service layer that connects the API, inference engine, model files, and hardware resources.

I now think of the local inference flow as a series of layers.

Layer One: The Application or Agent

At the top is the application that uses the model, such as a chat interface, a coding Agent, or OpenClaw.

This layer usually handles:

Assembling the system prompt
Preserving conversation history
Calling tools
Managing the Agent loop
Sending requests to the model API
Parsing the model's response

The Agent does not perform neural network inference itself. It only needs to know which endpoint should receive the request.

Layer Two: The API Compatibility Layer

The next layer might be an OpenAI-compatible API such as:

http://localhost:11434/v1/chat/completions

This layer allows applications originally connected to OpenAI or Codex to connect to a local model using a similar request format.

A request might include fields such as:

model
messages
temperature
tools
stream
max_tokens

This compatibility is why I could try redirecting the Hermes Agent from its original remote model to Ollama running on my local machine.

API format compatibility does not mean that every capability is identical.

A system may support the Chat Completions API without fully supporting the Responses API, tool calling, reasoning fields, or other Codex behavior.

Beyond checking whether the endpoint is reachable, I also need to verify the protocol and features expected by the higher-level Agent.

Layer Three: The Model Management and Service Layer

Ollama can fill this role.

It receives HTTP requests and finds the requested model. If the model is not already resident in memory, Ollama loads it before managing inference and returning the result as an API response. A resident model can process a new request directly.

It hides the low-level complexity so the application does not need to know where the model files are stored or manage the inference process itself.

Layer Four: The Inference Engine

Below that is the inference engine responsible for executing the model's computations, such as a runtime that uses llama.cpp technology.

sequenceDiagram
    participant User
    participant Agent as Agent / Chat Interface
    participant Ollama as Ollama API
    participant Runtime as Inference Runtime (for example, llama.cpp)
    participant Model as Qwen3 / GGUF
    participant Hardware as CPU / GPU

    User->>Agent: Enter Prompt
    Agent->>Agent: Assemble System Prompt and Conversation History
    Agent->>Ollama: Send Chat Completions Request

    alt Model Is Not Resident in Memory
        Ollama->>Runtime: Load Model
        Runtime->>Model: Read Model Weights and Tokenizer
    end

    Ollama->>Runtime: Pass Prompt and Inference Settings
    Runtime->>Runtime: Tokenize Prompt

    rect rgb(235, 235, 235)
        Note over Runtime,Hardware: Prompt Evaluation
        Runtime->>Hardware: Process All Input Tokens
        Hardware-->>Runtime: Return Model Computation Results
        Runtime->>Runtime: Build KV Cache
    end

    loop Generate One New Token at a Time
        Runtime->>Hardware: Run Transformer Forward Pass
        Hardware-->>Runtime: Return Logits
        Runtime->>Runtime: Sample the Next Token
        Runtime->>Runtime: Update Context and KV Cache
        Runtime-->>Ollama: Stream Output Token
        Ollama-->>Agent: Return Partial Content
        Agent-->>User: Display the Answer as It Is Generated
    end

This layer mainly:

Tokenizes the prompt
Processes the input tokens
Builds the KV cache needed for attention
Runs the computation for every model layer
Calculates the probability distribution for the next token
Selects a token according to the sampling rules
Appends the new token to the input
Repeats the computation until the answer is complete

There are two distinct kinds of work here.

The first is prompt evaluation, which processes all the input tokens sent at the beginning.

The second is token generation, in which the model produces the answer one token at a time.

Their performance characteristics are not identical.

Layer Five: The Model and Its Weights

Only at this layer do we reach the model itself.

In this example, that means the architecture, parameters, and tokenizer of Qwen3 4B.

The model might be stored in GGUF format and quantized using a particular method.

The model determines what it can do, while quantization and inference settings affect how it actually performs on the current hardware.

Layer Six: The Hardware

All computation eventually runs on physical hardware:

CPU
GPU
RAM
VRAM
Memory bandwidth
PCIe data transfers

If the GPU cannot hold the entire model, the inference engine may place only some layers on the GPU and leave the rest on the CPU.

The model can still run, but it may be much slower than expected.

Whether a model supports my GPU is therefore not a simple yes-or-no question.

A more precise question is:

At what speed and quality can this model run with the current inference engine, quantization format, offloading settings, and hardware resources?

I happened to be writing this article on July 17, 2026, a day when semiconductor stocks were falling sharply.

Given the hardware available today, though, ordinary personal computers still have a long way to go before they can run local models with capabilities close to the most advanced cloud models. Chip compute, memory bandwidth, power efficiency, and overall hardware architecture all need to keep improving.

So yes, I am still ALL IN on semiconductors.

A Simple Test

Once we understood the architecture, we ran a few simple tests with Qwen3:4B. The model ran through Ollama, primarily on the CPU, with the context set to 4096. We repeated the same questions and observed total time, output token count, and generation speed. These results were intended to show how this computer performed in practice, not to serve as a complete, reproducible benchmark.

The Ollama API reports several important timing and token metrics:

total_duration: the time spent on the entire request
prompt_eval_count: the number of input tokens processed by the model
prompt_eval_duration: the time spent processing the input prompt
eval_count: the number of tokens generated; with the Ollama and Qwen3 combination used here, the observed value included thinking content and the final answer
eval_duration: the time spent generating output tokens

The prompt processing and generation rates in this article are derived by dividing each token count by its corresponding duration to obtain tokens per second.

The clearest observation was that similar generation speeds could still produce wait times ranging from about 7 seconds to 47.6 seconds. Output length explained most of the difference: one run had an eval_count of about 142, while another reached 918. In these tests, those values also covered thinking content. Evaluating whether a local model is fast enough therefore requires looking at both tokens per second and how much content the model ultimately generates.

Conclusion: I Am Learning a System, Not a Single Tool

I started by trying to make use of a semi-retired computer and find a model suited to its current GPU. If Qwen3 4B did not run well, I planned to switch to another model. Once I began experimenting, I realized that learning how the whole system works matters more at this stage. I can decide which model is best later.

When a model now fails to run or performs slowly, I can investigate each architectural layer in turn: whether the Agent assembled the request correctly, whether the API protocol is compatible, whether Ollama loaded the model, whether the inference engine is using the GPU, whether the quantized version fits the hardware, and whether the bottleneck comes from the hardware or an overly long model output.

When I enter another prompt, I can now picture how the text becomes tokens, passes through the API, inference service, and model layers, runs across the CPU and GPU, and finally turns back into an answer one token at a time.

When the model takes a long time to answer, I also know what to examine next: is the hardware actually slow, or did the model simply think too much?

This may be my first real step toward understanding local AI systems.