# Local and self-hosted model runtimes

Software for running models on your own machine or servers: inference engines, model servers and desktop applications. Most offer an OpenAI-compatible endpoint that agent frameworks can call.

39 tools, in alphabetical order. Nothing is ranked by popularity.

- [AnythingLLM](https://modelmoot.com/tools/anythingllm.md): A desktop and Docker application for chatting with documents and running agents, using local or hosted models. (Mintplex Labs; open source, hosted service)
- [BentoML](https://modelmoot.com/tools/bentoml.md): A Python framework for packaging models and services into deployable containers, with a hosted cloud. (BentoML; open source, hosted service)
- [BitNet](https://modelmoot.com/tools/bitnet.md): An inference framework from Microsoft for running 1-bit language models on CPUs and GPUs, built on llama.cpp. (Microsoft; open source)
- [Docker Model Runner](https://modelmoot.com/tools/docker-model-runner.md): A Docker feature that pulls and runs models from registries on the local machine with an OpenAI-compatible API. (Docker; open source)
- [exo](https://modelmoot.com/tools/exo.md): Software that connects several local devices into one cluster that runs large models together. (exo labs; open source)
- [GPT4All](https://modelmoot.com/tools/gpt4all.md): A desktop application and library for running language models locally on CPUs and GPUs, with local document chat. (Nomic; open source)
- [GPUStack](https://modelmoot.com/tools/gpustack.md): A manager for GPU clusters that deploys and serves models with vLLM, SGLang and other engines. (GPUStack; open source)
- [Jan](https://modelmoot.com/tools/jan.md): An open source desktop application that runs language models locally and can connect to remote providers, with a local API server. (Menlo Research; open source)
- [KoboldCpp](https://modelmoot.com/tools/koboldcpp.md): A single-file program that runs GGUF models locally with a web interface and several API styles. (LostRuins; open source)
- [KServe](https://modelmoot.com/tools/kserve.md): A Kubernetes custom resource framework for serving predictive and generative models, with autoscaling. (KServe project; open source)
- [Lemonade](https://modelmoot.com/tools/lemonade.md): A local server that runs models on AMD GPUs and NPUs and other hardware, with an OpenAI-compatible API. (Lemonade SDK; open source)
- [LitServe](https://modelmoot.com/tools/litserve.md): A Python framework for building custom model inference servers on FastAPI, with batching and GPU autoscaling. (Lightning AI; open source)
- [llama-swap](https://modelmoot.com/tools/llama-swap.md): A proxy that starts and swaps local model servers on demand behind one OpenAI-compatible endpoint. (mostlygeek; open source)
- [llama.cpp](https://modelmoot.com/tools/llama-cpp.md): A C and C++ library and server for running language models in the GGUF format on CPUs and many GPUs. (ggml-org; open source)
- [llamafile](https://modelmoot.com/tools/llamafile.md): A way of distributing and running a model as one executable file that works across several operating systems. (Mozilla.ai; open source)
- [llm-d](https://modelmoot.com/tools/llm-d.md): A Kubernetes-native framework for distributed model serving, built on vLLM, with cache-aware routing. (llm-d project; open source)
- [LM Studio](https://modelmoot.com/tools/lm-studio.md): A desktop application and local server for downloading and running language models, with a model catalogue. (LM Studio; not open source)
- [LMCache](https://modelmoot.com/tools/lmcache.md): A KV cache layer for serving engines that stores and reuses cached attention states across requests and nodes. (LMCache project; open source)
- [LMDeploy](https://modelmoot.com/tools/lmdeploy.md): A toolkit for compressing, deploying and serving language models, with two inference engines. (Shanghai AI Laboratory; open source)
- [LocalAI](https://modelmoot.com/tools/localai.md): A self-hosted server that runs language, image, speech and embedding models behind OpenAI-compatible APIs. (LocalAI; open source)
- [MLC LLM](https://modelmoot.com/tools/mlc-llm.md): A compiler-based engine for deploying language models on many devices, including phones, browsers and GPUs. (MLC; open source)
- [MLX](https://modelmoot.com/tools/mlx.md): An array framework from Apple machine learning research for Apple silicon, with Python, Swift, C++ and C APIs. (Apple; open source)
- [MLX LM](https://modelmoot.com/tools/mlx-lm.md): A Python package for running and fine-tuning language models on Apple silicon with the MLX framework. (Apple; open source)
- [NVIDIA Dynamo](https://modelmoot.com/tools/nvidia-dynamo.md): A distributed inference framework that spreads serving of large models over many GPUs, with disaggregated prefill and decode. (NVIDIA; open source)
- [Ollama](https://modelmoot.com/tools/ollama.md): A tool that downloads and runs language models locally, with a command line, a REST API and an OpenAI-compatible endpoint. (Ollama; open source, hosted service)
- [Open WebUI](https://modelmoot.com/tools/open-webui.md): A self-hosted web interface for models that connects to Ollama and OpenAI-compatible servers, with RAG and tools. (Open WebUI; not open source)
- [OpenVINO GenAI](https://modelmoot.com/tools/openvino-genai.md): A C++ and Python library from Intel for running generative models efficiently on Intel CPUs, GPUs and NPUs. (Intel; open source)
- [RamaLama](https://modelmoot.com/tools/ramalama.md): A command-line tool that runs models in OCI containers using a runtime picked for the detected hardware. (Containers project; open source)
- [Ray Serve](https://modelmoot.com/tools/ray-serve.md): A model serving library on Ray that scales Python services across a cluster, with an LLM serving module. (Anyscale; open source, hosted service)
- [SGLang](https://modelmoot.com/tools/sglang.md): A serving framework for language and multimodal models with a fast runtime and a structured generation language. (SGLang project; open source)
- [Speaches](https://modelmoot.com/tools/speaches.md): An OpenAI-API-compatible server for speech-to-text and text-to-speech that runs locally. (Speaches contributors; open source)
- [Tabby](https://modelmoot.com/tools/tabby.md): A self-hosted AI coding assistant server with code completion and chat for editors, run on the organization own hardware. (TabbyML; open source)
- [TensorRT-LLM](https://modelmoot.com/tools/tensorrt-llm.md): A library from NVIDIA with a Python API for defining and running language models efficiently on NVIDIA GPUs. (NVIDIA; open source)
- [Text Embeddings Inference](https://modelmoot.com/tools/text-embeddings-inference.md): A server from Hugging Face for serving embedding, reranking and sequence classification models. (Hugging Face; open source)
- [Text Generation WebUI](https://modelmoot.com/tools/text-generation-webui.md): A local application with a web interface for running language models using several back ends, with an OpenAI-compatible API. (oobabooga; open source)
- [Triton Inference Server](https://modelmoot.com/tools/triton-inference-server.md): An inference server that serves models from several frameworks over HTTP and gRPC, with batching and ensembles. (NVIDIA; open source)
- [vLLM](https://modelmoot.com/tools/vllm.md): A high-throughput inference and serving engine for language models, with an OpenAI-compatible server. (vLLM project; open source)
- [whisper.cpp](https://modelmoot.com/tools/whisper-cpp.md): A C and C++ port of Whisper for running speech recognition on laptops, phones and servers without Python. (ggml-org; open source)
- [Xinference](https://modelmoot.com/tools/xinference.md): A server for deploying language, embedding, reranking, image and audio models locally or on a cluster. (Xorbits; open source)

HTML: https://modelmoot.com/tools/c/runtimes. All categories: https://modelmoot.com/tools.md
