AI Business Agent Console: a support agent that cites, asks, and waits
An AI support assistant added to an existing business system: answers from policy documents with citations, looks up orders through MCP tools, drafts replies — and never changes data without human approval.
Capability case built on synthetic data — not paid client work.
- Role
- Solo — product, design, architecture, full-stack and AI engineering, testing
- Stack
- Next.js 16TypeScriptMCP TypeScript SDKDeepSeekAlibaba Qwen embeddingsOpenAI / Gemini adaptersPostgreSQL + pgvectorDrizzleZodVitestDocker
Overview
- Challenge: a store's support team wants AI to look things up and draft replies, but is afraid it will invent policy or take actions on its own.
- Solution: an agent that answers only from cited policy, calls business tools through an MCP server, drafts replies, and pauses before any write until a different person approves the exact action.
- Results: a real end-to-end run with DeepSeek and Qwen embeddings, from question to approved ticket; 98 automated tests and 12 deterministic evaluation cases passing.
The challenge
The scenario is a fictional home-goods online store. The numbers are typical for a store this size and are assumptions, not client data.
- Team: 12 people, 3 in customer support, around 300 tickets a day — most about refunds, shipping and memberships.
- Today: agents switch between policy documents (which change often) and the order system. New hires quote outdated policy.
- What the owner asked for: "I want AI to gather the facts and draft the reply, but it must never refund on its own or make up policy."
That turns into five acceptance criteria:
- Every conclusion links to the policy text and version it relied on.
- If the policy doesn't cover it, the answer is "insufficient evidence — hand to a human", never a guess.
- Refunds, tickets and other data changes need approval from a different person.
- Every run shows what it cost and how long it took.
- The model vendor can be swapped for price or compliance reasons.
What I built
A case-first workbench, not a chat box. The queue, the case under investigation, the approval card with its exact arguments, the step trace and provider readiness all sit on one screen.
Workbench: from wireframe to final UI
A bounded agent. It runs as a state machine, not an open loop: classify → retrieve policy → look up customer, order and subscription → analyze → draft → risk check → wait for approval or execute. Hard limits: 8 steps, 5 model calls, 6 tool calls, 60-second timeout, no automatic retry of writes.
Tools behind a real MCP boundary.
| Tool | Risk | Approval |
|---|---|---|
| Look up customer / order / subscription, search policy | Low | No |
| Draft customer email | Medium | No — drafts only, never sends |
| Create support ticket | High | Always |
Technical highlights
| What a client worries about | How the console handles it |
|---|---|
| Hallucination | Citation IDs are assigned before generation; the model can only cite retrieved text; low evidence returns "insufficient evidence" |
| Unwanted actions | Tools are risk-tiered; writes need a signed, one-time approval bound to the exact arguments |
| Permissions and data isolation | Workspace identity is injected by the system — the model can never pass it — and roles are checked before every tool call |
| Prompt injection | Retrieved documents are treated as untrusted; instruction-like text is flagged and can't expand tool access |
| Privacy | Logs redact emails, phone numbers, tokens and API keys |
| Runaway cost | Step, call and time budgets; tokens and cost recorded per run |
| Vendor lock-in | Separate generation and embedding contracts; DeepSeek, Qwen, OpenAI and Gemini adapters |
Two defects only showed up against real providers — strict MCP output validation rejected internal metadata, and DeepSeek's thinking mode could use up the token budget without an answer. Both were fixed, which is why the live run mattered.
Quality & testing
| Layer | Result |
|---|---|
| Automated tests | 98 passing — tool schemas, approval replay and self-approval, risk policy, chunking, retrieval, redaction, provider adapters |
| Evaluations | 12 deterministic cases: retrieval, citation validity, tool choice, argument accuracy, approval, role denial, tenant isolation, injection |
| MCP protocol | Real Streamable HTTP path verified: web → MCP → approval → MCP → TICKET-0001 |
| Live providers | DeepSeek generation + Alibaba qwen3.7-text-embedding (1024-d) end to end; missing keys report not_run, never a made-up score |
Results & limits
| Original spec | Shipped in V1 |
|---|---|
| PDF, DOCX, Markdown, TXT up to 10 MB | UTF-8 .txt and .md up to 2 MB |
| Hybrid full-text + vector retrieval | Vector retrieval; pgvector schema ready for production |
| Four roles with row-level security | Fixed operator / approver identities in the demo |
| Live comparison of all providers | DeepSeek + Qwen verified live; OpenAI and Gemini built against the same contract |
The public demo runs on a deterministic demo provider so clicking costs nothing; the live-model run is shown in the video.
How I delivered it
| Step | What was produced |
|---|---|
| Requirements | One concrete support question, roles, tool risk table, acceptance criteria |
| Prototype | Page flow and wireframes for the workbench and the run / approval view |
| UI | Case-first layout, evidence next to every conclusion, approval beside its arguments |
| Architecture | Web app + independent MCP server, pluggable providers, pgvector |
| Build | Spec-first with AI coding agents; verified against real providers |
| Testing | Unit tests, deterministic evaluations, MCP protocol and live-provider runs |
| Launch | Two Docker services on Render (in progress) |