What happens when you press Enter — traced from the top of the stack to the bottom.
A deconstruction of a live demo — step by step, layer by layer, all the way down. The goal is to become a knowledgeable, discerning buyer of the AI tools you already use and will be asked to use or evaluate.
Not vibe-coding. Not hype. The math stays out. The mechanism stays in.
We start with a real demo — an AI morning briefing running live. Then we follow that prompt all the way through the stack, from the software that received your command down to the GPU operation that produced each word.
One command. Six tools. A complete morning briefing.
You type /start-my-day. The agent runs this SOP — in order, without you touching anything else.
The agent read six separate tools — each one a real API call to a live service — synthesized everything it found, and returned a single briefing grouped by urgency. It followed the playbook, flagged the items that needed action, and stopped.
That's not impressive because of AI. It's impressive because of everything around the AI: the software that received your command, assembled the instructions, called the tools one by one, managed the loop, and delivered the result.
Now let's follow that prompt. All the way down.
First stop. Before the model sees your message, this runs.
The moment you press Enter, your message passes through a filter. It is not the model. It is a separate system that runs before the model ever reads your words — and it is silent unless your message gets blocked.
What the filter checks depends on who built the product. The base API (Anthropic, OpenAI) enforces the vendor's safety policies — content that could cause harm, attempts to override the model's core instructions, content the vendor prohibits. A product built on top can add its own rules: topic restrictions, age-appropriate content filters, compliance guardrails.
AI products process millions of requests. A small fraction of users probe for weaknesses — attempts to get the model to ignore its instructions, generate prohibited content, or leak information it shouldn't. The guardrail catches these before the expensive GPU operation runs. Cheaper to filter here than after the model has already responded.
Most of the time: nothing. Your message goes through. When it doesn't, you get a refusal — often with no explanation of which rule triggered it.
For an organization deploying AI: this layer is where content restrictions, topic limitations, and compliance rules should live. Not all products surface these controls to administrators. That's a buyer question.
The software that wraps the model, assembles the page, and runs the loop.
The AI model — the neural network itself — does one thing: it reads a page of text and writes a reply. Then it forgets everything. It has no memory of past conversations. It cannot open a file, make an API call, or send an email. On its own, it just takes text in and returns text out.
Every time you interact with an AI product, software builds a fresh page for the model and slides it under its nose. That software is the harness. It bundles your question, the conversation so far, your standing instructions, any files, and the skill's playbook — and hands it all to the model as one large block of text.
The model sees only what's on this page. Nothing else. The harness decides what goes on it — and that decision is most of the product.
Everything you think of as "the AI being smart" is mostly the harness doing its job. Two products running the same model can behave completely differently because their harnesses are completely different.
One iteration. For the morning briefing, this cycle runs at least 6 times.
The page the model reads. Everything it knows lives here — and only here.
The model has no external knowledge during a request. It cannot look things up. It cannot remember your name from last week. All it knows on any given call is what appears on the page the harness assembled — the context. If it's not on the page, it doesn't exist.
This is why the harness rebuilds the context on every loop iteration: after the Slack tool returns results, those results get appended to the page. The model reads them on the next decode pass. Without that append, the model would have no idea what Slack returned.
You write instructions, examples, or constraints directly. System prompts, skills, the anatomy of a good prompt — all prompting. This is what the morning briefing skill is: a block of text the harness injects onto the page before your message arrives.
At query time, retrieve relevant documents from a database and stuff them onto the page alongside your question. This is how "your company's documents" get into the context — not by training the model on them, but by fetching and injecting them at runtime. Updatable without retraining.
Fine-tuning changes the model's weights, not the page. It does not add knowledge reliably — research shows fine-tuning on new facts can increase hallucination, because the model learns the pattern of confidently asserting things, then applies it to things it doesn't actually know. Reach for RAG first. Fine-tune only when prompting and RAG have proven a specific gap they can't close.
Every model has a context limit — a maximum number of tokens (text chunks) that can appear on the page at once. When the limit fills, one of two things happens: older content gets dropped silently, or the model starts degrading. That's context rot. The Bonus screen demonstrates it interactively.
In the morning briefing, each tool result appends to the context. Six tools, six results — the page grows with each loop iteration. Long or verbose tool results can eat the context window quickly.
Where the context becomes computation. The serving infrastructure you don't see.
When a prompt arrives at the serving infrastructure, it runs two distinct operations in sequence.
The entire context — every token on the page — is processed in one parallel pass through the model. Parallel means fast: it doesn't matter if the context is 100 tokens or 10,000, the GPU handles them simultaneously.
The result: a KV cache — a stored computation of every token's relationship to every other token. This is the expensive part of a turn, but it only runs once.
The model generates the response token by token — each new token depends on all prior ones. This cannot be parallelized. Output length directly determines how long decode takes and how much it costs.
This is why a three-sentence answer is faster than a three-paragraph one — even if the input prompt was the same.
Without the KV cache, each loop iteration would re-process the entire context from scratch — every prior message, every tool result, every token. For the morning briefing, that would mean re-running prefill on a growing context six times.
Instead: the serving engine stores the computed key-value pairs from the first prefill. On subsequent loop iterations, only the new tokens (the tool result just appended) need to be prefilled. Everything else is read from cache. This is why adding a tool result mid-loop costs much less than the initial context load.
When you call Claude, GPT, or Gemini via API, Layer 1 is entirely abstracted. The API endpoint receives your context and returns tokens. The serving engine — vLLM, TensorRT-LLM, or whatever the vendor runs — handles prefill, decode, batching, and caching behind the scenes.
You only care about this layer if you're evaluating self-hosted models: open-source weights (Llama, Mistral, Qwen) running on your own GPU fleet. Then Layer 1 choices dominate your infrastructure costs and latency budget.
The atom. Every token the model generates runs this sequence once.
When the model decodes a token, it runs a fixed pipeline. This runs once per output token — thousands of times across a full conversation.
The model samples from a distribution — the same prompt can produce different outputs on different runs. It is not a search engine returning the "correct" answer. It is generating the statistically likely next token given everything that came before. This is why it can be confidently wrong.
Each token depends on all prior tokens, including the ones the model just generated. You cannot generate token 50 before tokens 1-49 are done. Longer outputs always take longer — and cost more, since inference is priced per output token.
Attention is computed across the full context on every forward pass. A message from 40 turns ago can influence what the model generates today. This is also why context fills with noise over a long conversation — old, irrelevant tokens still consume attention and can distort the output. That's context rot.
What "agent" actually means when someone says it — and what to ask at each layer.
Same word. Three very different things. Now that you have the stack, you can disambiguate.
When evaluating any AI product or platform, the stack tells you what to investigate. Here are the load-bearing questions by layer.
Cost and effort climb as you go deeper into the stack. The common mistake: teams over-invest in Layers 1 and 0 (self-hosting, fine-tuning) and under-invest in Layers 2 and 4 (context quality, evals and guardrails). The leverage is almost always higher up the stack — prompting and RAG before fine-tuning, evals before infrastructure.
Watch a conversation fill the AI's memory. Every message re-sends everything.
This is a toy example with a 3,000-word limit. Real systems handle 100K–1M+ tokens, but the exact same problem shows up at scale.
Notice the only fix this demo offers is manual: start a new chat. Stronger harnesses fight the rot for you — they trim long outputs, skip files they've already read, and boil older turns down to a running summary so the important parts survive. That same summary is what gets saved between sessions: it's why you can close a Project or a chat and pick it up days later. The tool kept the gist, not the full transcript.
Shopping signal: does the tool manage its own context and remember across sessions — or is it on you to notice the rot and start over?