Skip to content
Progress

AI Systems

The same discipline, a different cost structure.

Everything from the classical curriculum still applies — caching, queues, timeouts, backpressure, failure isolation. What changes is where the money and the milliseconds go. Latency scales with output length. Spend scales with a context window that grows on every loop iteration. And the slowest dependency in your system is one you do not operate.

LLM Request Trace

Follow an agentic request through tool selection, an MCP server, and final synthesis. Change the model tier or add a tool call and watch what it does to latency and cost.

Request trace
  1. Client18ms

    Request leaves the browser.

  2. API Gateway12ms

    Auth, rate limiting, request validation.

  3. Agent loop6ms

    Assembles the system prompt, tool schemas, and conversation history.

  4. Model call 1 — tool selection1.23s

    Decides which tool to call and with what arguments.

    in 4K tokout 90 tok$0.013
  5. MCP server — tool execution180ms

    Runs the tool and returns a result the model has to read back.

  6. Model call — final response3.3s

    Generates 320 output tokens at roughly 9ms each.

    in 4.5K tokout 320 tok$0.018
  7. Streaming response10ms

    Tokens are forwarded as they are produced.

Time to first token1.88s
Total latency4.76s
Tokens8.9K
Cost / request$0.032

With streaming, the user sees output after the first token, not after the last. Time to first token is the number that governs perceived speed.

Model tier

The usual default. Handles tool use and multi-step reasoning reliably.

1

Each call is another full model round trip, with the growing transcript resent every time.

4K tok
320 tok

Latency scales with output length. Input is processed in parallel; output is generated one token at a time.

Model tiers are generic and prices are illustrative, chosen to show the shape of the trade-off rather than any specific provider’s rates.

Curriculum

Topics marked as covered appear in the trace above. The rest are in development.

LLM Application Architecture

  • LLM Request LifecycleCovered
  • Streaming & Time to First TokenCovered
  • Structured Outputs
  • Model Routing
  • Token & Cost ManagementCovered
  • Model Fallback
  • AI Gateways

Retrieval

  • Embeddings
  • Vector Search
  • Chunking Strategy
  • RAG Pipelines
  • Reranking
  • Semantic CachingCovered

Agentic Systems

  • Tool CallingCovered
  • Agent LoopsCovered
  • MCPCovered
  • Sandboxing
  • Human Approval
  • Agent Memory
  • Checkpointing
  • Long-Running Workflows
  • Multi-Agent Systems

Operating AI Systems

  • AI Observability
  • Evals
  • Guardrails
  • Failure Recovery
  • Provider Rate Limits