How we built the AI chat assistant on our own site

Why we bothered

Most visitors to an agency site have the same handful of questions: what do you build, how does the process work, what does it roughly cost, who's behind this. A contact form answers none of them, it just asks the visitor to do work and wait. We wanted those questions answered in seconds, at any hour, before someone loses interest. So we treated our own site like a client project and shipped a chat assistant for it.

The stack: Groq + Llama 3.3 70B

The widget is a Next.js API route that talks to llama-3.3-70b-versatile running on Groq. We picked Groq for one reason above all: speed. Their inference is fast enough that replies start streaming almost immediately, which matters more in a chat widget than raw model intelligence, a visitor who waits three seconds for the first word assumes it's broken. The 70B Llama model is comfortably smart enough for "answer questions about one company" and costs a fraction of frontier models. We also kept the model swappable behind an environment variable, so we can change it without touching code:

// set CHATBOT_MODEL to override (e.g. a smaller
// model for lower latency/cost)
const MODEL =
  process.env.CHATBOT_MODEL || "llama-3.3-70b-versatile";

No RAG, no vector database, on purpose

Every AI tutorial wants you to chunk your content, embed it, and stand up a vector database. For a site this size that's machinery for machinery's sake. Our entire site, services, process, FAQs, case studies, team, testimonials, already lives in one typed data file that the pages render from. So the bot's system prompt is generated from that same file at request time:

/** Knowledge is composed from lib/data.ts
    so the bot always matches the site. */
function systemPrompt(): string {
  const serviceLines = services.map(/* ... */);
  const faqLines = faqs.map(/* ... */);
  const workLines = caseStudies.map(/* ... */);
  // ...one prompt, built from the site's
  // single source of truth
}

This is the single best decision in the build. The bot literally cannot drift out of sync with the site: update a service page or add a testimonial, and the assistant knows about it on the next message. No re-indexing, no embedding pipeline, no second copy of the truth to maintain. If your content fits in a prompt, and for most small business sites it does, this beats RAG on every axis that matters.

Guardrails: the boring part that makes it trustworthy

An assistant that invents a price or promises a deadline is worse than no assistant. The system prompt carries hard rules: only state facts from the prompt, never invent prices, deadlines, guarantees, or statistics, keep replies to a few sentences, and for anything specific, point warmly to the free discovery call. It also says plainly that it's an AI when asked. None of this is clever engineering, it's just deciding in advance what the bot must never do, and writing it down.

The hardening most tutorials skip

A public endpoint that spends API tokens on every request is a target, so the route defends itself before a single token is spent:

// Per-IP rate limit: 20 messages / 10 minutes
const WINDOW_MS = 10 * 60 * 1000;
const MAX_PER_WINDOW = 20;

// Strict input validation, reject anything odd
if (!Array.isArray(value) || value.length > 24) return null;
out.push({ role: m.role, content: m.content.slice(0, 2000) });
return out.slice(-12); // cap context: last 12 turns

In plain terms: each visitor gets 20 messages per 10 minutes, every message is validated for shape and capped at 2,000 characters, conversation context is trimmed to the last 12 turns, and replies are capped at 1,024 tokens. So the worst case, someone deliberately hammering the endpoint, is bounded and cheap. The endpoint also fails politely: if the API key is missing it returns a clean 503 instead of crashing, and if the stream breaks mid-reply the visitor gets an apology and our email address, not a spinner. All of it sits behind the same Content-Security-Policy the rest of the site ships with.

Streaming, because perceived speed is speed

The route streams tokens to the browser as the model produces them, through a plain ReadableStream, no websockets, no extra dependencies. If the visitor closes the widget mid-answer, the stream's cancel handler aborts the upstream request so we stop paying for tokens nobody will read. Small detail, but it's the difference between a demo and a thing you run in production.

What we'd tell a client

If your site's content fits in a prompt, a well-guarded, prompt-composed assistant is a weekend-sized project that genuinely helps visitors, and most of the real work is not the AI. It's the guardrails, the rate limiting, the input validation, and wiring the bot to your single source of truth so it never lies about you. That's the part we'd never skip.