A useful AI system usually needs more than a model. It needs a way to read data, find the right context, run reliably, and fit into somebody’s work. These ten repositories cover different parts of that job. This is a curated starting point, not a popularity ranking or a claim that every project needs all ten.
Start with the job, then choose the tool
For a first local assistant, begin with a model runtime. For answers from documents, add document processing, retrieval, and evaluation. For an agent that takes action, think about state, permissions, and review before adding more integrations.
The repository links below are the original project sources checked on 9 September 2026. The notes describe software capabilities, not a security audit or a benchmark. Review the current release and license before adopting a tool; downloaded model weights and hosted services can have separate terms.
The shortlist
Ollama
Local models · MIT
Run supported models and connect applications to a model interface. A practical entry point when you want to explore a local assistant.
Before you choose Decide which model fits your hardware, and check whether your chosen setup uses local or cloud inference.
Original repositoryllama.cpp
Model runtime · MIT
Run language and vision-language models through a C/C++ inference engine with support for different hardware backends.
Before you choose Useful when you want more control over model files, quantisation, and runtime settings.
Original repositoryvLLM
Model serving · Apache-2.0
Serve language models with an engine focused on throughput and memory efficiency.
Before you choose Check supported hardware and model compatibility. Test your real concurrency and latency requirements before choosing a deployment.
Original repositoryLangGraph
Agent workflows · MIT
Coordinate stateful, long-running agent workflows. Useful when a task needs explicit steps, saved state, or human review.
Before you choose Draw the workflow first. Define when it can retry, when it must stop, and which actions require approval.
Original repositoryLlamaIndex
Data & agents · MIT
Connect data to language-model applications through an open-source framework for retrieval and agent workflows.
Before you choose Distinguish the open-source framework from separately offered hosted parsing and enterprise services.
Original repositoryHaystack
AI pipelines · Apache-2.0
Compose retrieval, routing, and generation into modular Python pipelines and agent workflows.
Before you choose A candidate when you want to inspect the stages of a search or question-answering pipeline independently.
Original repositoryQdrant
Vector search · Apache-2.0
Store and search vectors with associated metadata. Useful for finding related document chunks and filtering search results.
Before you choose Design document permissions and metadata filters before you connect sensitive data.
Original repositoryDocling
Document processing · MIT
Convert documents into structured representations for downstream processing and AI applications.
Before you choose Inspect extraction quality on your own tables, scans, and layouts. A broken extraction becomes broken context.
Original repositorySentence Transformers
Retrieval & reranking · Apache-2.0
Work with embedding and reranker models for semantic retrieval and relevance scoring.
Before you choose Compare retrieval quality on representative questions. A reranker adds another processing step, so measure its benefit.
Original repositoryTransformers
Model development · Apache-2.0
Use model architectures for text, vision, audio, and multimodal inference or training.
Before you choose Check the individual model’s license and resource requirements. A library’s license does not grant rights to every model.
Original repositoryA small starting stack
For a document-assistant prototype, one possible combination is Ollama for model inference, Docling for document conversion, Qdrant for retrieval, and Sentence Transformers for embeddings or reranking. This is an architectural suggestion, not a tested compatibility matrix. You may need fewer components, or an orchestration framework to connect them.
Start with a handful of documents and questions you can check yourself. Include a question the documents cannot answer. Expand the system only after you can explain why it answered, where the evidence came from, and what it does when evidence is missing.
What could this make easier for you?
Bring the problem. We can work out which tools belong in the solution.
Discuss your idea ↗