Memory and context layer for production copilots

The problem

A copilot was stuffing entire chat histories and large documents into every prompt. Costs spiked, latency drifted, and the model lost the thread on long conversations. There was no separation between what the user just said, what the user said last week, and what the product knew about them.

The solution

Designed a three-tier memory layer. Short-term working memory in a turn buffer, long-term semantic memory in a vector store keyed by user and tenant, and episodic memory as a structured summary written back to DynamoDB after each session. A context builder Lambda assembles a per-turn context window using a budget, recency, and relevance score, with explicit slots for user profile, tools, and retrieved knowledge. Eval pipeline regression-tests context quality on a frozen set.

Architecture

Stack

Outcomes