GitHub - machinarii/total-recall-catalog: A reference catalog of latest knowledge retrieval, memory & RAG systems

Catalog of AI Knowledge Retrieval, Memory & RAG Systems (Last updated 2025-04-13)

What This Is

A reference catalog mapping of AI knowledge systems that store, retrieve, and reason. The thesis: every AI knowledge system is solving a problem that biological memory already solved, just with different tradeoffs.

Humans remember through association (vector similarity), narrative (episodic memory), relationships (knowledge graphs), consolidation (sleep/dreaming), and forgetting (interference, decay). AI systems have converged on strikingly parallel architectures: vector databases for associative recall, conversation logs for episodes, graph RAG for relational reasoning, offline consolidation for memory pruning, and TTL/auto-expiry for managed forgetting.

This catalog exists because the landscape is fragmented across 100+ projects with overlapping names, unclear boundaries, and fast-changing codebases. It answers: what does each project actually do, what layer of the stack does it occupy, what hardware does it need, and how does it map to the cognitive function it's replacing.

Who This Is For

Anyone building or evaluating AI knowledge infrastructure — particularly local-first setups where hardware compatibility (Metal vs CUDA vs CPU) determines what's even possible.

How To Use It

Sections are ordered from low-level infrastructure (vector DBs, embedding servers) to high-level cognition (memory management, dreaming/consolidation). If you're building a stack, read bottom-up. If you're evaluating a specific project, find its category and compare within-section. The cognition mapping table at the bottom connects everything back to the human memory mechanisms each category replaces.

Cross-referenced against Awesome-Agent-Memory, Awesome-Memory-for-Agents, and Awesome-GraphRAG. GPU/platform data aligned to GPU Compute Platforms Breakdown.

GPU/Accelerator Legend

Platform terminology follows the six-layer stack model (see companion doc):

• ⭐ Stars: Approximate GitHub stars as of April 2026. Rounded to nearest K. Sourced from repo pages, search results, and Awesome lists. Counts change daily — treat as order-of-magnitude indicators, not exact figures.

Key distinction: Metal = native GPU shaders (fast, no PyTorch) vs MPS = PyTorch routing through Metal Performance Shaders (convenient, overhead). Your Ollama/llama.cpp stack uses Metal, not MPS.

Other Columns

• Updated: Last release as of April 2026. ~ = approximate.
• Deploy: Self = self-hosted | Cloud = managed service | Both = either

1. VECTOR DATABASES & SIMILARITY SEARCH

Purpose: Store and search high-dimensional vectors (embeddings) by similarity. The foundation layer — everything else retrieves from here.

Note: Vector DBs themselves rarely need GPU. The GPU matters for the embedding model that feeds vectors into them. FAISS is the exception — it can use CUDA for the similarity search itself.

2. RAG FRAMEWORKS

Purpose: Orchestrate the full retrieve-then-generate pipeline. Chunking, embedding, retrieval, prompt assembly, LLM call. Most are orchestrators that call external model servers.

Note: Most RAG frameworks are orchestrators — they call external LLM/embedding APIs and don't run models themselves. "CPU" means the framework runs on CPU; GPU usage depends on the model server it calls (Ollama, vLLM, etc.). DSPy is distinctive: it's a programmatic optimization framework (compile prompts/weights, not hand-tune) that treats RAG as composable modules. STORM is a knowledge curation system that uses multi-agent conversations to produce long-form cited articles from web or local sources.

3. GRAPH RAG & KNOWLEDGE GRAPHS

Purpose: Add entity-relationship structure on top of vector retrieval. Answers questions that require connecting dots across documents ("who reported to whom during which project").

Note: LightRAG MPS issue — default HF embedding code checks torch.cuda.is_available() and falls back to CPU, ignoring MPS. Patch: add elif torch.backends.mps.is_available() check. Not needed when routing embeddings through Ollama (which uses Metal natively).

4. SPECIALIZED RETRIEVAL & SEARCH

Purpose: Retrieval engines that solve specific problems — hybrid search, code search, cache-augmented generation, or token-level reranking.

5. DOCUMENT PREPROCESSING & INGESTION

Purpose: Convert raw files (PDFs, HTML, images, web pages) into clean, chunked text that vector DBs and RAG frameworks can ingest. Garbage in, garbage out — this layer determines retrieval quality.

6. RERANKING & RETRIEVAL REFINEMENT

Purpose: Take the top-N candidates from initial retrieval and re-score them with a more expensive model. Bridges the gap between "vaguely relevant" (vector search) and "actually useful" (cross-encoder scoring).

zerank-2 on your stack: Runs via llama.cpp on host Mac Studio → Metal backend → M4 Max GPU. The reranker is served at 127.0.0.1:8090 and ClawRAG calls it via HTTP after ChromaDB returns initial candidates.

7. EMBEDDING MODELS & INFERENCE SERVERS

Purpose: Run LLMs and embedding models locally or serve them via API. The compute layer. This is where GPU acceleration (Metal, CUDA) actually matters.

Metal vs MPS here: Ollama and llama.cpp use Metal (native shaders, no PyTorch, fastest path). sentence-transformers and FlagEmbedding use MPS (PyTorch → Metal Performance Shaders → Metal, convenient but slower). vLLM has no Apple Silicon support — CUDA only.

Your inference path: Ollama → llama.cpp → Metal shaders → M4 Max GPU cores (no PyTorch, no MPS, no MLX in the critical path)

8. AGENT MEMORY — UNIVERSAL / MULTI-TIER

Purpose: Give AI agents persistent memory across sessions. Extract facts from conversations, store them durably, retrieve when relevant. The "remember me" layer.

Note: Memory frameworks are almost universally CPU-only. They store/retrieve/manage memories and call external LLM APIs for extraction/reasoning. GPU usage is delegated to whatever model server they're configured to use (Ollama, OpenAI, etc.).

9. AGENT MEMORY — CONVERSATION & EPISODIC

Purpose: Track conversation history, temporal facts, and episodic sequences. Optimized for "what happened when" rather than "find similar content."

10. AGENT MEMORY — CODING & WORKSPACE

Purpose: Persist context for coding agents (Claude Code, OpenClaw). Remember project state, decisions, and working patterns across sessions.

11. ACADEMIC MEMORY SYSTEMS (Papers with Code)

Purpose: Research systems from papers — not production-ready but represent where the field is heading. Many introduce novel memory architectures that later get adopted by production frameworks.

12. CONTEXT & TOKEN MANAGEMENT

Purpose: Manage the LLM context window — the bottleneck. KV cache reuse, token compression, context trimming. The AI equivalent of managing working memory capacity.

13. DREAMING & CONSOLIDATION

Purpose: Offline memory processing between sessions. Consolidate, compress, prune, and promote knowledge. Named after the biological process it mimics.

14. ORCHESTRATION & ROUTING

Purpose: Route tasks to the right agent, model, or tool. The executive function layer — knows what to do without containing the knowledge itself.

15. EVALUATION & BENCHMARKING

Purpose: Measure whether your RAG/memory system actually works. Retrieval precision, generation faithfulness, memory recall accuracy.

16. META-RESOURCES

COGNITION MAP: Human → AI