A sovereign alternative to AnythingLLM

AnythingLLM does a lot, and does it well

AnythingLLM is one of the better things to come out of the self-hosted AI wave. It's an all-in-one application from Mintplex-Labs that you run on your own hardware: Docker, bare metal, a cloud VM, or the one-click desktop app for Mac, Windows, and Linux. You point it at a model, drop in documents, and get a private, ChatGPT-style interface over your own files. It's MIT-licensed and genuinely pleasant to use.

Its strengths are real. You organise documents into workspaces, each with its own isolated set of files, so a legal team and a marketing team can share one instance without seeing each other's material. It connects to almost any model you like: OpenAI, Anthropic, Google Gemini, Mistral, and Groq on the API side, or Ollama, LM Studio, and LocalAI if you want everything running locally. It has agents that can browse the web and call tools, a no-code agent builder, multi-user controls, and a developer API. If you want a polished team hub for chatting with a large, mixed pile of documents, AnythingLLM is a strong, honest choice.

So this isn't the usual cloud-vs-local comparison. AnythingLLM is already self-hosted. The interesting difference is architectural: how each tool turns your documents into something a model can answer from, and what that costs you over time.

The vector-database tax

When you add a document to an AnythingLLM workspace, you "save and embed" it. Under the hood, the text is chunked, passed through an embedding model, and stored as vectors in a vector database: LanceDB by default, or Pinecone, Chroma, Weaviate, Qdrant, Milvus, PGVector, and others if you scale up. At query time your question is embedded too, and the system retrieves the chunks whose vectors sit closest to it. This is classic RAG, and it works.

It also carries costs that are easy to underestimate until you live with them:

• You host the vector store. It's another stateful service to run, back up, secure, and keep alive. The convenient default (LanceDB) is fine on a laptop; a serious multi-user deployment usually means standing up and maintaining a standalone vector database server.
• The index drifts from the documents. The embeddings are a derived copy of your files. When a document changes (a contract gets a new clause, a spec gets revised), you have to re-embed it. Forget to, and the model confidently answers from the stale version. The index is a second source of truth you now have to keep in sync.
• Re-embedding is recurring work. Switch embedding models, change chunk settings, or upgrade the app, and your existing vectors may need to be rebuilt. The cost isn't one-time; it recurs every time the inputs move.
• Retrieval is fuzzy by design. Nearest-neighbour search returns what's semantically similar, which is excellent for "find me anything about X" and weak for "what is the exact figure in row 12." If the right chunk doesn't rank in the top-k, the model never sees it, and you rarely get told it was missed.

None of this makes AnythingLLM bad. It's the inherent price of the embeddings-plus-vector-DB approach. We went deeper into that tradeoff in RAG without a vector database.

How pdf2okf approaches it differently

pdf2okf starts from a different premise: keep the knowledge as readable text, and skip the vector layer entirely.

It converts each PDF into an OKF-compatible knowledge bundle: structured Markdown concept files following Google's Open Knowledge Format (Google's open standard; pdf2okf is compatible with it, not its author). Both the concepts and the figures in the PDF are extracted. There's no embedding step and no index to host: instead, an agent greps the markdown concept files directly. Because nothing derived has to be kept in sync, when a document changes you just replace the file: no re-embedding, no stale vectors, nothing to rebuild. The difference between OKF and a vector database is the heart of it.

That changes a few things in practice:

• The workspace is portable. An OKFZ bundle is just files. You build it once, version it in git, move it between machines, or hand it to a colleague, with no separate index shipped alongside, because there isn't one.
• Answers can be deterministic and cited. When a question needs an exact number, code counts it and the model reports the result, rather than hoping the right chunk surfaced from a top-k search. Every answer points back to its source, so it's auditable.
• It runs where your data is. Everything works as on-device AI through a local model, or via your own key (BYOK) if you'd rather call an API you control. Nothing has to leave your machine.

If the no-upload angle is what matters most to you, the same logic applied to consumer tools is in the local ChatPDF alternative.

When to pick which

This isn't a case where one tool wins everything, and it would be dishonest to pretend otherwise.

Choose AnythingLLM when you have a large, heterogeneous corpus (thousands of mixed documents, web sources, codebases) and you want broad semantic search across all of it from a single, polished team UI. Its multi-user controls, agent builder, and huge list of model and vector-DB backends make it an excellent all-in-one knowledge hub. For "ask anything across everything," vector retrieval is exactly the right tool.

Choose pdf2okf when your documents are bounded and structured, and the answers have to be exact, cited, and portable: contracts, financial filings, technical specs, regulatory PDFs. When a wrong number is worse than no answer, deterministic counting beats fuzzy retrieval, and a portable bundle with no index to maintain beats a vector store you have to keep in sync.

Both can keep your data on your own hardware. The real question is what you want the knowledge to be: a searchable vector index, or a portable, auditable set of files. pdf2okf is in private build, so join the waitlist if the second description fits your documents.