Persistent memory NGT Memory for LLM

NGT Memory: Persistent Memory for LLMs with Association Graph and User Profile

NGT Memory implements persistent memory for large language models without external storage. The module stores user facts across sessions, retrieves relevant context, and injects it into the prompt. Three retrieval mechanisms ensure accuracy: cosine similarity of embeddings, a Hebbian associative graph, and hierarchical consolidation.

Cosine similarity compares the query embedding with stored facts. The graph links concepts that activate together: mentioning a diet with a query about restaurants pulls up the relevant fact. Consolidation promotes frequently used facts into long-term memory, displacing rarely requested ones.

Retrieval time is 2–3 ms on CPU. Main delays come from the OpenAI API: embeddings ~700 ms, generation ~800–1500 ms.

Setup and API

The module runs via Docker:

git clone https://github.com/ngt-memory/ngt-memory
cd ngt-memory
cp .env.example .env  # OPENAI_API_KEY
docker-compose up -d

Client code:

import httpx

client = httpx.Client(base_url="http://localhost:9190")

# Storing facts
client.post("/chat", json={
    "message": "I'm a vegetarian and live in Moscow.",
    "session_id": "user_42"
})

# Retrieval via session
r = client.post("/chat", json={
    "message": "What should I eat?",
    "session_id": "user_42"
})
print(r.json()["response"])  # Vegetarian restaurants in Moscow
print(r.json()["memories_count"])  # 2

Endpoints: /chat, /store, /retrieve, /session/reset, /health.

Structured User Profile

The system extracts slots from messages: age, city, diet, allergies. Data is injected into the system prompt with priority:

[USER PROFILE — structured facts, highest priority]
  - age: 30
  - city: "Moscow"
  - diet: "vegetarian"
[END USER PROFILE]

[MEMORY CONTEXT — verified facts about this user]
  1. [0.91] I'm a vegetarian and live in Moscow.
[END MEMORY CONTEXT]

Fragment stitching: "I'm" + "30" + "years old" → "I'm 30 years old" → age=30 (confidence 0.6).

Conflict resolution: Prevents decreasing age without the phrase "I was mistaken." Correction mode — 60 seconds.

Quality Filter and Architecture

The filter weeds out junk:

• Pure numbers, special characters, single words
• Less than 6 letters
• Assistant responses to user junk

Architecture:

• POST /chat
• OpenAI text-embedding-3-small (~700 ms)
• Profile extraction (regex)
• NGT Memory retrieve (cosine + graph, 2–3 ms)
• OpenAI gpt-4.1-nano (~800–1500 ms)
• Filter → store

Stack: FastAPI, AsyncOpenAI, Pydantic. Storage in RAM of a single process.

Experiment Results

Exp 44: Factual accuracy with memory 2.44/3 (+100% vs memoryless 1.22/3).

Exp 48: A/B test, 94% memory wins (17/18), average score 0.889 vs 0.056.

Exp 49: 51/54 (94%) on edge cases: cross-language, stitching, conflicts.

Performance (5000 facts):

| Operation | Throughput | p50 |

|-----------|------------|-----|

| store() | 3450/s | 0.29 ms |

| retrieve()| 150/s | 6.3 ms |

End-to-end: memory 2.5 ms, embeddings 764 ms.

Scaling Challenges

• Multi-worker: each worker has its own SessionStore. Solution: --workers 1 or Redis.
• Soft prompt: model ignored facts. Strengthened: "Treat every fact as absolute truth."
• Regex bugs: I'm allergic → name= allergic. Blacklist 25+ words.

Key Takeaways

• Combining cosine + graph + consolidation achieves 94% on edge tests.
• Structured profile with prompt priority doubles accuracy.
• Quality filter prevents search degradation from junk.
• Memory latency 2–3 ms, not a bottleneck.
• Open source, BSL 1.1, Docker-ready.

— Editorial Team