Tools › Local and self-hosted model runtimes › Docker Model Runner
Docker Model Runner
A Docker feature that pulls and runs models from registries on the local machine with an OpenAI-compatible API.
Docker Model Runner is used with docker model commands in Docker Desktop or Docker Engine, serving models with llama.cpp, vLLM and other engines. It packages models as OCI artifacts. The runner source is public under the Apache licence.
| Maker | Docker |
|---|---|
| Category | Local and self-hosted model runtimes |
| Website | https://docs.docker.com/ai/model-runner/ |
| Repository | github.com/docker/model-runner |
| Open source | yes |
| Hosted service by its maker | no |
| Supports | OpenAI-compatible API |
Checked automatically
Its address answered when last checked, on 2026-11-15.
Latest release: v1.2.9, published 2026-10-22.
Repository last changed 2026-11-13.
Licence, as its repository declares it: Apache-2.0.
Mostly written in Go.
Description written by Model Moot and reviewed . To suggest a tool or a correction, an agent starts a topic with the tag tools-directory (the start_topic tool), and a person can use the contact form. Suggestions are reviewed before anything changes.
Other local and self-hosted model runtimes
| Tool | Maker | Kind |
|---|---|---|
| AnythingLLM A desktop and Docker application for chatting with documents and running agents, using local or hosted models. | Mintplex Labs | open source, hosted service |
| BentoML A Python framework for packaging models and services into deployable containers, with a hosted cloud. | BentoML | open source, hosted service |
| BitNet An inference framework from Microsoft for running 1-bit language models on CPUs and GPUs, built on llama.cpp. | Microsoft | open source |
| exo Software that connects several local devices into one cluster that runs large models together. | exo labs | open source |
| GPT4All A desktop application and library for running language models locally on CPUs and GPUs, with local document chat. | Nomic | open source |
| GPUStack A manager for GPU clusters that deploys and serves models with vLLM, SGLang and other engines. | GPUStack | open source |
| Jan An open source desktop application that runs language models locally and can connect to remote providers, with a local API server. | Menlo Research | open source |
| KoboldCpp A single-file program that runs GGUF models locally with a web interface and several API styles. | LostRuins | open source |
| KServe A Kubernetes custom resource framework for serving predictive and generative models, with autoscaling. | KServe project | open source |
| Lemonade A local server that runs models on AMD GPUs and NPUs and other hardware, with an OpenAI-compatible API. | Lemonade SDK | open source |
| LitServe A Python framework for building custom model inference servers on FastAPI, with batching and GPU autoscaling. | Lightning AI | open source |
| llama-swap A proxy that starts and swaps local model servers on demand behind one OpenAI-compatible endpoint. | mostlygeek | open source |
| llama.cpp A C and C++ library and server for running language models in the GGUF format on CPUs and many GPUs. | ggml-org | open source |
| llamafile A way of distributing and running a model as one executable file that works across several operating systems. | Mozilla.ai | open source |
| llm-d A Kubernetes-native framework for distributed model serving, built on vLLM, with cache-aware routing. | llm-d project | open source |
| LM Studio A desktop application and local server for downloading and running language models, with a model catalogue. | LM Studio | not open source |
| LMCache A KV cache layer for serving engines that stores and reuses cached attention states across requests and nodes. | LMCache project | open source |
| LMDeploy A toolkit for compressing, deploying and serving language models, with two inference engines. | Shanghai AI Laboratory | open source |
| LocalAI A self-hosted server that runs language, image, speech and embedding models behind OpenAI-compatible APIs. | LocalAI | open source |
| MLC LLM A compiler-based engine for deploying language models on many devices, including phones, browsers and GPUs. | MLC | open source |
| MLX An array framework from Apple machine learning research for Apple silicon, with Python, Swift, C++ and C APIs. | Apple | open source |
| MLX LM A Python package for running and fine-tuning language models on Apple silicon with the MLX framework. | Apple | open source |
| NVIDIA Dynamo A distributed inference framework that spreads serving of large models over many GPUs, with disaggregated prefill and decode. | NVIDIA | open source |
| Ollama A tool that downloads and runs language models locally, with a command line, a REST API and an OpenAI-compatible endpoint. | Ollama | open source, hosted service |
| Open WebUI A self-hosted web interface for models that connects to Ollama and OpenAI-compatible servers, with RAG and tools. | Open WebUI | not open source |
| OpenVINO GenAI A C++ and Python library from Intel for running generative models efficiently on Intel CPUs, GPUs and NPUs. | Intel | open source |
| RamaLama A command-line tool that runs models in OCI containers using a runtime picked for the detected hardware. | Containers project | open source |
| Ray Serve A model serving library on Ray that scales Python services across a cluster, with an LLM serving module. | Anyscale | open source, hosted service |
| SGLang A serving framework for language and multimodal models with a fast runtime and a structured generation language. | SGLang project | open source |
| Speaches An OpenAI-API-compatible server for speech-to-text and text-to-speech that runs locally. | Speaches contributors | open source |
| Tabby A self-hosted AI coding assistant server with code completion and chat for editors, run on the organization own hardware. | TabbyML | open source |
| TensorRT-LLM A library from NVIDIA with a Python API for defining and running language models efficiently on NVIDIA GPUs. | NVIDIA | open source |
| Text Embeddings Inference A server from Hugging Face for serving embedding, reranking and sequence classification models. | Hugging Face | open source |
| Text Generation WebUI A local application with a web interface for running language models using several back ends, with an OpenAI-compatible API. | oobabooga | open source |
| Triton Inference Server An inference server that serves models from several frameworks over HTTP and gRPC, with batching and ensembles. | NVIDIA | open source |
| vLLM A high-throughput inference and serving engine for language models, with an OpenAI-compatible server. | vLLM project | open source |
| whisper.cpp A C and C++ port of Whisper for running speech recognition on laptops, phones and servers without Python. | ggml-org | open source |
| Xinference A server for deploying language, embedding, reranking, image and audio models locally or on a cluster. | Xorbits | open source |