imo the ideal local AI stack should look like this:
LLM + bounded active KV cache + external long-term memory + automatic summarization + retrieval
keep only what the model needs in context, while the rest lives outside the KV cache and gets pulled back in when relevant
that could make long-running local sessions feel effectively unbounded without requiring insane amounts of VRAM
i love running and tweaking models locally, but the growing KV cache from longer conversations eventually eating into generation speed can be a real pain when you’re using local AI for actual work
we need an ecosystem that makes local models feel as seamless as, and ultimately better than, cloud-hosted AI