AI Gateway

What You Will Build

A production-quality HTTP API that takes a user message and streams the AI response back one token at a time, using Server-Sent Events. By the end of this phase, you will have a Dockerized service you can show to any engineer.

Data Flow Visualization

Watch exactly how your message travels from browser to LLM and back. Use the interactive diagram above to explore each step.

Click any node to see the real data structure at that point in the pipeline. Press Animate Flow to step through the request lifecycle.

Why This Phase Exists

Many senior engineers make the same mistake when they move into AI. They try to learn LangChain, RAG, agents, and Kubernetes all at once. Then they give up before shipping anything.

Phase 1 does one thing: send a message to an AI and get a streamed response back. But it does that one thing well. The API is typed, validated, rate-limited, and containerized. You cannot skip any part of this phase. Every later phase builds on the patterns you set here: the env validation, the service abstraction, the Docker setup. The one to two weeks you spend getting this right is the most valuable time in the whole program.

How LLM APIs Actually Work

Most web developers think of an API call like this: send a request, get JSON back, done. LLM APIs work differently. If you don't update your mental model now, the topics that come later will not make sense: cost, latency, and context limits.

Here is what happens on every request:

• ▸Your text is tokenized first. "Hello world" becomes about 2 tokens. One token is roughly 4 characters, or 0.75 words.
• ▸Tokens go into the context window. This is the model's entire working memory. It remembers nothing outside this window.
• ▸The model does not write a whole sentence at once. It predicts one token at a time. Each prediction depends on every token before it.
• ▸Streaming means each token is sent to you the moment it is generated, instead of waiting for the full response.
• ▸You pay for input tokens (your prompt) and output tokens (the response) as two separate items. Generating tokens costs much more compute than reading them.

Gotcha: if you go over the context window (input plus output combined), you get either a hard error or silent truncation, depending on the provider. Counting tokens is not optional. It decides whether your request works at all.

Why Groq for This Phase

Groq runs open models on custom LPU (Language Processing Unit) hardware instead of GPUs. In practice, that means 500–800 tokens per second, versus about 60 tokens per second on GPT-4o.

We start with Groq for one reason: speed. It is not the only production choice.

• ▸No waiting during development. You see the full streaming behavior in real time instead of watching a spinner.
• ▸A real free tier: 14,400 requests per day and 30,000 tokens per minute on Llama 3.3 70B, with no credit card.
• ▸Groq's SDK is OpenAI-compatible. Switching to Claude or GPT-4 later (Phase 5) is a two-line change, not a rewrite.
• ▸Llama 3.3 70B performs close to GPT-4-class models on most everyday tasks. What you build is not a toy. It has a production shape from day one.

Key API Parameters

Security Middleware You Cannot Skip

Problem: an HTTP endpoint with no protection will be found by someone else's script within hours of going live. That is even more true when the endpoint calls a metered, billable API.

Four middlewares close the obvious holes. Each one earns its place:

• ▸helmet() sets secure HTTP headers. It removes X-Powered-By fingerprinting and adds sane Content-Security-Policy defaults. It is one line, so there is no reason to skip it.
• ▸cors() controls who can call your API. Without an allow-list, any website can call your API from a user's browser using their session. Restrict origin to the domains you actually serve.
• ▸Body size limit (express.json({ limit: '50kb' })) stops oversized payloads. Without it, one malicious request with a 500MB body can crash your process before your route handler runs.
• ▸Rate limiting (express-rate-limit) protects your wallet, not only your server. Every request that reaches Groq costs money. An endpoint with no limit lets a bug or an attacker run up a large bill.

Env Validation — Fail-Fast Pattern

Problem: a missing or malformed environment variable should never show up three requests into production as a confusing undefined is not a function. It should crash the process at startup, loudly, before the server accepts any traffic.

Zod's safeParse does this for you:

 Copy
const envSchema = z.object({
  GROQ_API_KEY: z.string().min(1, 'GROQ_API_KEY is required'),
  GROQ_MODEL: z.string().default('llama-3.3-70b-versatile'),
  PORT: z.string().transform(Number).default('3000'),
});

const _env = envSchema.safeParse(process.env);
if (!_env.success) {
  console.error('❌ Invalid environment variables:', _env.error.format());
  process.exit(1);
}

export const env = _env.data;

You will reuse this pattern in every phase from here on. Define the schema once. Parse it once when the module loads. Call process.exit(1) on failure. You will not scatter env.SOME_KEY ?? 'fallback' through your business logic later. By the time any other file imports env, it is already valid and correctly typed.

Core Streaming Loop — Code Deep Dive

This is the main abstraction the whole phase is built around. It is an AsyncGenerator that wraps Groq's stream and yields clean chunks to whatever calls it:

 Copy
export async function* streamChat(
  messages: ChatMessage[],
  options: { temperature?: number; maxTokens?: number } = {}
): AsyncGenerator<StreamChunk> {
  const stream = await groq.chat.completions.create({
    model: env.GROQ_MODEL,
    messages,
    temperature: options.temperature ?? 0.7,
    max_tokens: options.maxTokens ?? 2048,
    stream: true,
  });

  for await (const chunk of stream) {
    const text = chunk.choices[0]?.delta?.content ?? '';
    const finishReason = chunk.choices[0]?.finish_reason;

    if (finishReason === 'stop') {
      yield { text: '', done: true, usage: /* computed from chunk.x_groq.usage */ undefined };
    } else if (text) {
      yield { text, done: false };
    }
  }
}

Let's walk through it. async function* marks this as a generator. A generator can pause and resume, handing back a value each time with yield instead of returning once with return. The for await...of loop reads the Groq SDK's own stream one network chunk at a time. Each chunk either carries a piece of text (delta.content) or signals completion (finish_reason === 'stop'). On the calling side, for await (const chunk of streamChat(messages)) receives each yielded value the moment it is produced. This is what makes token-by-token streaming work without you managing sockets or buffers by hand.

SSE vs WebSockets vs Polling

This section answers a common interview question, almost word for word: "Why did you use Server-Sent Events instead of WebSockets for streaming?"

• ▸Polling — the client keeps asking "is it done yet?" It is simple, but wasteful and slow. It is the wrong tool for token-by-token output.
• ▸WebSockets — a full two-way, persistent connection. It is powerful, but it is too complex for a one-way text stream. It also needs special handling behind some corporate proxies and load balancers.
• ▸Server-Sent Events (SSE) — plain HTTP, one-way, server to client. It uses the same connection behavior as a normal request. It works through firewalls and proxies that block WebSocket upgrades. The browser's EventSource API also reconnects automatically if the connection drops, so you get that resilience for free.

SSE is the right choice here because the data only flows one way (server to client). Phase 6 introduces WebSockets, and the reason will be clear by then: two-way, low-latency features such as typing indicators and live collaboration, where the client also needs to send data mid-stream. SSE cannot do that.

Docker — Why Now, Not Later

Problem: "works on my machine" is not something you can deploy. Adding Docker later, to a codebase that was not built for it, is much more painful than building it in from the first commit.

The Dockerfile uses a multi-stage build. One stage compiles TypeScript with the full devDependencies tree. A second, clean stage copies over only the compiled dist/ output and the production node_modules:

 Copy
FROM node:20-alpine AS builder
WORKDIR /app
COPY package.json pnpm-lock.yaml* ./
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm build && pnpm prune --prod

FROM node:20-alpine AS production
WORKDIR /app
RUN addgroup -g 1001 -S nodejs && adduser -S agenttelar -u 1001
COPY --from=builder --chown=agenttelar:nodejs /app/dist ./dist
COPY --from=builder --chown=agenttelar:nodejs /app/node_modules ./node_modules
USER agenttelar
HEALTHCHECK --interval=30s --timeout=3s CMD node -e "fetch('http://localhost:3000/health').then(r=>process.exit(r.ok?0:1))"
EXPOSE 3000
CMD ["node", "dist/index.js"]

Gotcha: most people miss this on their first multi-stage build. The final image on alpine comes out around 200MB instead of 1GB or more. This is because the TypeScript compiler, type definitions, and dev tooling never reach the production stage. Add USER agenttelar (never run a container as root) and a HEALTHCHECK that Docker uses to auto-restart a stuck container. This is the same Dockerfile shape you will reuse through Phase 9's Kubernetes work.

Test It — curl Commands + Expected Output

This is the verification gate. Set up your env and start the service first:

 Copy
cp apps/api/.env.example apps/api/.env
# Open apps/api/.env and paste your free Groq key from https://console.groq.com
cd docker && docker compose up --build -d

Compose project name is agenttelar-phase1 (volumes stay isolated from later phases). It loads apps/api/.env into the container. Do not move to the quiz until every command below behaves as shown.

1. Health check:

 Copy
curl http://localhost:3000/health
# {"status":"ok","version":"1.0.0","phase":1,"timestamp":"..."}

2. Streaming chat (watch tokens arrive live):

 Copy
curl -X POST http://localhost:3000/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"message": "Explain RAG in 3 bullet points.", "temperature": 0.3}' \
  --no-buffer

Expected output: a live sequence of SSE events, not one blob:

event: token
data: {"text":"RAG"}
event: token
data: {"text":" (Retrieval"}
...
event: done
data: {"fullResponse":"...","usage":{"promptTokens":45,"completionTokens":120,"estimatedCostUSD":0.000122}}

3. Rate limiting (quick check — no Groq cost):

The live Compose limit is 100, which is fine for daily use but painful to trip in a tutorial. For this check only, temporarily set RATE_LIMIT_MAX_REQUESTS: 5 in docker/docker-compose.yml, recreate the API, then restore 100.

Use GET /api/chat/health (same /api rate limiter as chat, zero LLM calls). Do not use GET /health — that route is outside /api and is not limited.

 Copy
# After editing RATE_LIMIT_MAX_REQUESTS: 5 in docker-compose.yml:
docker compose up -d --force-recreate api

for i in $(seq 1 6); do
  curl -s -o /dev/null -w "$i %{http_code}\n" \
    http://localhost:3000/api/chat/health
done
# 1–5 → 200; 6 → 429 {"error":"Too many requests, please slow down"}

# Restore RATE_LIMIT_MAX_REQUESTS: 100, then:
docker compose up -d --force-recreate api

If every line is 200, the limiter is not wired to /api.

Try It Live

Run the actual phase_1 code in your browser. No local install needed. Once the editor loads, copy .env.example to .env in the file tree. Paste your own free Groq API key to make it respond for real.

Cost Awareness — Numbers Table

Every CTO asks about cost eventually. Start tracking it on your first API call, not after a bill surprises you.

Completion Checklist — Technical

Before you consider Phase 1 done, all of these should be true:

• ▸POST /api/chat returns streaming SSE events — verified with curl
• ▸GET /health returns JSON with status: "ok"
• ▸Temporarily setting RATE_LIMIT_MAX_REQUESTS to 5 and sending 6 requests to GET /api/chat/health triggers a 429 on the 6th
• ▸An invalid request body returns 400 with Zod validation details
• ▸A missing GROQ_API_KEY causes process.exit(1) at startup, not a runtime crash later
• ▸The Docker image builds and runs successfully
• ▸docker compose up --build -d starts the service and passes its health check
• ▸Token usage and estimated cost appear in every done SSE event

Conceptual Checklist — Can You Explain?

If you can't answer these out loud without checking your code, re-read the relevant section above before moving on:

• ▸What is a token, and why does tokenization directly determine cost?
• ▸What is a context window, and what happens when a request exceeds it?
• ▸Why SSE instead of WebSockets for this specific use case?
• ▸When do you set temperature low versus high, and why?
• ▸Why is the system prompt the most powerful lever in the whole request?
• ▸Why a multi-stage Docker build instead of a single FROM node stage?
• ▸What happens, exactly, when GROQ_API_KEY is wrong — and where does that error surface?

What You Will Learn

• ▸Understand tokens, context window, and temperature in depth
• ▸Build a streaming SSE API from scratch, with real security middleware
• ▸Containerize with a multi-stage Docker build from day one
• ▸Know the cost of every API call in dollars before you ship it

Next Steps

Clone the phase1 folder on GitHub, copy apps/api/.env.example to apps/api/.env, add your Groq key, then run cd docker && docker compose up --build -d. Work through the curl tests above until each one matches. When you move to Phase 2, stop this stack first (docker compose down in phase1/docker) so :3000 is free. Complete the quiz below before Phase 2.