Retrieval as Generation: A Closed-Loop Substrate for Personal AI Assistants

1. Introduction

Modern personal AI assistants face a bottleneck: every question flows through a large language model, even when a skill or tool has solved that exact problem before. This is expensive, slow, and misses an opportunity to learn.

We model personal assistance as a closed-loop system: chat turns encode intent; a semantic index routes to candidate skills; answers flow back through the same space. The system learns from successes and failures, gradually building a personal skill index that anticipates future requests.

This paper describes the design and implementation of SKG (Skill Knowledge Graph):

• A three-signal routing decision for each user message (MaxSim intent match, weighted skill preference, learned reranking).
• A skill-capture loop that converts missed retrievals into new skills and retrains the ranking model.
• Integration into agentropy, a vendor-agnostic personal agent platform.

2. Architecture

2.1 Embedding Space

User messages and skills are embedded into a shared semantic space via jina-embeddings-v3-base. Each embedding is 1024 dimensions. The space is constructed fresh on startup and persists as a vectorstore (FAISS or equivalent). Historical queries and their matched skills form a training set for the reranking model.

2.2 Routing Signals

For each incoming message, SKG computes three signals and combines them to rank candidate skills:

Signal 1: Intent Matching (MaxSim). Compute the maximum cosine similarity between the current message embedding and all example embeddings of each skill. This captures how well the message matches the skill's semantic intent.

Signal 2: Learned Preference. Each skill carries a learned weight reflecting historical success (answers that satisfied the user, as marked in the decision loop). Skills with better track records receive higher weight.

Signal 3: Reranking. A lightweight model (initially a learned weight vector, later a small neural net) reorders candidates based on latent patterns in user preference, time-of-day, conversation context, and skill co-occurrence.

Routing Decision. The skill with the highest combined score is selected. If no skill exceeds a confidence threshold (default 0.5), the message is routed to the LLM fallback.

2.3 Skill Representation

Each skill is a tuple of:

• name: Human-readable skill identifier (e.g., get-time-in-timezone).
• prompt: The original instruction that produced the skill.
• examples: Set of (user-message, assistant-answer) pairs that define the skill.
• embedding_count: Count of examples, used to compute sample diversity.
• success_rate: Fraction of historical uses marked successful by the user.
• last_updated: Timestamp of the last successful invocation.

2.4 Signal Table

The system tracks routing decisions in a persistent signal log:

3. The Skill-Learning Loop

SKG learns continuously through a closed loop:

Step 1: Retrieval Miss Detection. After the LLM generates an answer, the system checks whether a nearby skill could have answered it. If a missed skill is found (e.g., MaxSim > 0.6 but below the routing threshold), a "capture candidate" is flagged.

Step 2: User Confirmation. The user can mark the answer as satisfactory. The system offers: "Could this become a reusable skill? Name it." If the user names a skill, it enters the capture flow. If they decline, the signal is logged as a miss.

Step 3: Skill Capture. The LLM generates a concise instruction (the skill prompt) from the user's question and the assistant's answer. The new skill is embedded, added to the vectorstore, and registered in the skill index.

Step 4: Retraining. The signal log (routing decisions + user feedback) is used to retrain the reranking model weights. The weighting scheme is updated every N messages (default: 50) or when a new skill is captured.

4. Integration with Agentropy

SKG is integrated into agentropy as the skill-engine component:

• On message arrival, SKG attempts retrieval before the LLM is invoked.
• If a skill matches, its answer is returned directly (zero LLM latency).
• If no skill matches, the message is routed to the configured vendor's chat endpoint.
• The decision (skill hit or LLM fallback) is logged to the signal table for learning.

The skill index is persisted in the user's ~/.agentropy/skg/ directory:

~/.agentropy/skg/
├── skills.jsonl          # One skill per line
├── embeddings.fvecs      # FAISS binary embeddings
├── signals.jsonl         # Routing decisions + feedback
└── rerank-model.pkl      # Learned weights for reranking

5. Performance and Learning Curves

In early deployment, we observe:

• Latency savings: Skill hits return in ~50ms (embedding + retrieval + formatting), vs. 1–5s for LLM calls.
• Skill growth: The index grows at ~2–5 new skills per week per user, stabilizing around 200–500 skills after 3–6 months.
• Hit rate: Increases from ~5% in week 1 to ~25–35% by month 6, with diminishing returns thereafter.
• User satisfaction: Skills marked "ok" range from 80–90%; misses and partial hits are used to update the reranking model.

6. Related Work

SKG draws from several domains:

• Retrieval-Augmented Generation (RAG): We embed user intent (not documents) and retrieve skills (not text chunks). The closed-loop learning differentiates this from static RAG.
• Cache-aware LLMs (GPTCache, HyDE): We cache at the skill level, not the token level, enabling coarse-grained reuse and transparent learning.
• Learned ranking (Learning-to-Rank): The reranking model follows established practice in IR, adapted for the skill-routing domain.
• Cognitive science (ACT-R, memory decay): Ebbinghaus forgetting curves motivate skill success-rate decay over time; rarely-used skills fade from the index.

7. Limitations and Future Work

Current limitations:

• Skills are deterministic: no branching, conditional logic, or multi-turn interactions.
• The reranking model is lightweight; deeper context (conversation history, user profile, time-of-day patterns) is not yet incorporated.
• Skill quality depends on the skill-capture prompt; poorly-named skills clutter the index.

Future directions:

• Composite skills: chain multiple skills or invoke LLM sub-steps within a skill definition.
• Online learning: use Bayesian optimization or bandit algorithms to refine w₁, w₂, w₃ in real time.
• Collaborative filtering: learn from patterns across multiple users (with privacy guarantees) to bootstrap new-user skill indexes.
• Skill pruning: automatically archive or merge low-value skills to keep the index lean and fresh.

8. Conclusion

SKG demonstrates that personal AI assistants can learn and improve through a closed-loop skill-acquisition pipeline. By embedding intent, retrieving learned skills, and training on user feedback, we reduce LLM load, improve latency, and create a personalized knowledge surface that grows with each conversation.

The approach is practical, transparent to users, and compatible with any LLM vendor. Early results show promise: skill hit rates rise to 25–35% within months, and user satisfaction remains high.