Tools › Supports OpenAI-compatible API
Tools that support OpenAI-compatible API
An OpenAI-compatible API accepts the same requests as OpenAI's chat or responses endpoints, so clients written for one work with the other. These tools in the directory say they support it.
91 tools, in alphabetical order. Nothing is ranked by popularity, and no listing is paid for.
| Tool | Maker | Kind |
|---|---|---|
| Agent Router A project, formerly Envoy AI Gateway, that gives agents one OpenAI-compatible API for many models and one router for MCP tools. | Agentic AI Foundation | open source |
| agentgateway An open-source proxy for traffic from AI agents, covering LLM, MCP, A2A and plain HTTP calls. | agentgateway project | open source |
| AI SDK A TypeScript toolkit for calling models, streaming results, using tools and building agents, with UI hooks for React and other frameworks. | Vercel | open source |
| Amazon Bedrock An AWS service that gives access to models from several providers through one API, with agents and guardrails. | Amazon Web Services | not open source, hosted service |
| any-llm A Python library that gives one interface to many model providers, with an optional gateway server. | Mozilla.ai | open source |
| Baseten A hosted platform for serving models with model APIs and custom deployments, with an OpenAI-compatible endpoint. | Baseten | not open source, hosted service |
| BentoML A Python framework for packaging models and services into deployable containers, with a hosted cloud. | BentoML | open source, hosted service |
| Bifrost An AI gateway written in Go that provides one OpenAI-compatible API to many providers, and also acts as an MCP gateway. | Maxim AI | open source |
| bolt.diy An open-source version of the Bolt app builder that runs on a local machine and lets the user choose the model provider. | StackBlitz Labs | open source |
| Cerebras Inference A hosted inference API for open models that runs on Cerebras wafer-scale hardware, with an OpenAI-compatible endpoint. | Cerebras | not open source, hosted service |
| Cloudflare AI Gateway A Cloudflare proxy for model API calls that adds caching, rate limiting, retries, fallbacks and logging. | Cloudflare | not open source, hosted service |
| Cloudflare Workers AI A Cloudflare service that runs open models on its network, called from Workers or over a REST API. | Cloudflare | not open source, hosted service |
| DeepInfra A hosted inference API for open models with an OpenAI-compatible endpoint and dedicated deployments. | DeepInfra | not open source, hosted service |
| Docker Model Runner A Docker feature that pulls and runs models from registries on the local machine with an OpenAI-compatible API. | Docker | open source |
| exo Software that connects several local devices into one cluster that runs large models together. | exo labs | open source |
| Fireworks AI A hosted inference platform for open models with an OpenAI-compatible API, fine-tuning and function calling. | Fireworks AI | not open source, hosted service |
| GPT4All A desktop application and library for running language models locally on CPUs and GPUs, with local document chat. | Nomic | open source |
| GPUStack A manager for GPU clusters that deploys and serves models with vLLM, SGLang and other engines. | GPUStack | open source |
| Groq A hosted inference API that serves open models on its own language processing hardware, with an OpenAI-compatible endpoint. | Groq | not open source, hosted service |
| Guardrails AI A Python framework that checks and constrains LLM inputs and outputs with validators from a shared hub. | Guardrails AI | open source |
| Helicone AI Gateway A gateway from Helicone that routes requests to many model providers with logging, cost tracking and fallbacks. | Helicone | open source, hosted service |
| Higress A cloud-native API gateway based on Envoy and Istio, with an AI gateway mode and an MCP server hosting feature. | Alibaba | open source |
| Hugging Face Inference Providers A Hugging Face service that gives one API and token for serverless inference from several partner providers. | Hugging Face | not open source, hosted service |
| Jan An open source desktop application that runs language models locally and can connect to remote providers, with a local API server. | Menlo Research | open source |
| KoboldCpp A single-file program that runs GGUF models locally with a web interface and several API styles. | LostRuins | open source |
| Kong AI Gateway A set of Kong Gateway plugins that route and govern traffic to model providers behind a unified interface. | Kong | open source, hosted service |
| KServe A Kubernetes custom resource framework for serving predictive and generative models, with autoscaling. | KServe project | open source |
| Lagent A lightweight Python framework from the InternLM team for agents with tool use and asynchronous execution. | InternLM | open source |
| Langroid A Python framework for multi-agent programming in which agents are actors that exchange messages. | Langroid | open source |
| Language Model Evaluation Harness A command-line framework for running language models against hundreds of standard evaluation tasks. | EleutherAI | open source |
| Lemonade A local server that runs models on AMD GPUs and NPUs and other hardware, with an OpenAI-compatible API. | Lemonade SDK | open source |
| LiteLLM A Python SDK and proxy server that calls many model providers in the OpenAI format, with keys, budgets and routing. | BerriAI | open source, hosted service |
| LitServe A Python framework for building custom model inference servers on FastAPI, with batching and GPU autoscaling. | Lightning AI | open source |
| llama-swap A proxy that starts and swaps local model servers on demand behind one OpenAI-compatible endpoint. | mostlygeek | open source |
| llama.cpp A C and C++ library and server for running language models in the GGUF format on CPUs and many GPUs. | ggml-org | open source |
| llamafile A way of distributing and running a model as one executable file that works across several operating systems. | Mozilla.ai | open source |
| LLM Gateway A gateway that routes requests to many providers through one OpenAI-compatible API, hosted or self-hosted. | LLM Gateway | open source, hosted service |
| llm-d A Kubernetes-native framework for distributed model serving, built on vLLM, with cache-aware routing. | llm-d project | open source |
| LM Studio A desktop application and local server for downloading and running language models, with a model catalogue. | LM Studio | not open source |
| LMDeploy A toolkit for compressing, deploying and serving language models, with two inference engines. | Shanghai AI Laboratory | open source |
| LocalAI A self-hosted server that runs language, image, speech and embedding models behind OpenAI-compatible APIs. | LocalAI | open source |
| Microsoft Foundry A Microsoft Azure platform that hosts models from several providers behind one endpoint, with agent tooling. | Microsoft | not open source, hosted service |
| MLC LLM A compiler-based engine for deploying language models on many devices, including phones, browsers and GPUs. | MLC | open source |
| MLX LM A Python package for running and fine-tuning language models on Apple silicon with the MLX framework. | Apple | open source |
| MS-Agent A lightweight Python framework from ModelScope for agents that carry out complex tasks with tools. | ModelScope | open source |
| Nebius Token Factory A Nebius hosted inference service for open models with an OpenAI-compatible API, fine-tuning and dedicated endpoints. | Nebius | not open source, hosted service |
| NeMo Guardrails A toolkit for adding programmable rails to LLM applications, written in a modelling language called Colang. | NVIDIA | open source |
| New API A self-hosted LLM gateway and asset management system with OpenAI-, Claude- and Gemini-compatible endpoints. | QuantumNous | open source |
| Novita AI A hosted platform with an OpenAI-compatible LLM API, GPU instances and agent sandboxes. | Novita AI | not open source, hosted service |
| NVIDIA Dynamo A distributed inference framework that spreads serving of large models over many GPUs, with disaggregated prefill and decode. | NVIDIA | open source |
| Ollama A tool that downloads and runs language models locally, with a command line, a REST API and an OpenAI-compatible endpoint. | Ollama | open source, hosted service |
| One API A self-hosted system that distributes access to many model providers through one OpenAI-style API with key management. | songquanpeng | open source |
| Open Responses A specification for a model API interface based on the OpenAI Responses API that several providers can implement. | Open Responses project | open source |
| Open WebUI A self-hosted web interface for models that connects to Ollama and OpenAI-compatible servers, with RAG and tools. | Open WebUI | not open source |
| OpenAI Agents SDK for Python OpenAI's Python library for agents with tools, handoffs between agents, guardrails and tracing. | OpenAI | open source |
| OpenAI Agents SDK for TypeScript OpenAI's TypeScript library for agents with tools, handoffs, guardrails, tracing and realtime voice. | OpenAI | open source |
| OpenAI computer use A tool in the OpenAI API that lets a model propose clicks, typing and scrolling from screenshots to operate a browser or desktop. | OpenAI | not open source, hosted service |
| OpenAI Evals A framework and registry from OpenAI for evaluating language models and systems built on them. | OpenAI | open source |
| OpenAI Guardrails Libraries for Python and TypeScript that wrap OpenAI client calls with configurable safety checks. | OpenAI | open source |
| OpenAI web search tool A built-in tool in the OpenAI Responses API that searches the web and returns answers with citations. | OpenAI | not open source, hosted service |
| opencode An open-source terminal coding agent that works with models from many providers and also runs as a desktop app and editor extension. | Anomaly | open source, hosted service |
| OpenRouter A hosted API that routes requests to models from many providers through one OpenAI-compatible endpoint. | OpenRouter | not open source, hosted service |
| OpenServ An agent infrastructure platform with a reasoning API, an agent builder and events for developers of agent services. | OpenServ | not open source, hosted service |
| Perplexity API APIs from Perplexity that return web search results and answers with citations, plus an MCP server. | Perplexity | not open source, hosted service |
| Plano An AI-native proxy and data plane for agentic apps that handles model routing, orchestration and observability. | Katanemo | open source |
| Portkey An AI gateway with routing, fallbacks, caching and guardrails, available as open source code and as a hosted control plane. | Portkey | open source, hosted service |
| PR-Agent An open-source tool that reviews, describes and suggests improvements to pull requests using a language model. | The PR-Agent project | open source |
| Pydantic AI A Python agent framework from the Pydantic team that uses typed inputs, outputs and dependencies. | Pydantic | open source |
| Qwen Code An open-source terminal coding agent from the Qwen team, tuned for Qwen models and usable with OpenAI-compatible endpoints. | Alibaba Cloud | open source |
| Qwen-Agent A Python framework from the Qwen team for agents with function calling, MCP, code execution and retrieval. | Alibaba Cloud Qwen | open source |
| RamaLama A command-line tool that runs models in OCI containers using a runtime picked for the detected hardware. | Containers project | open source |
| Ray Serve A model serving library on Ray that scales Python services across a cluster, with an LLM serving module. | Anyscale | open source, hosted service |
| Requesty A hosted LLM gateway with one API for many models, with routing, fallbacks, caching and cost controls. | Requesty | not open source, hosted service |
| RouteLLM A framework for serving and evaluating routers that send each query to a stronger or cheaper model. | LMSYS | open source |
| SambaNova Cloud A hosted inference API for open models that runs on SambaNova hardware, with an OpenAI-compatible endpoint. | SambaNova | not open source, hosted service |
| SGLang A serving framework for language and multimodal models with a fast runtime and a structured generation language. | SGLang project | open source |
| Speaches An OpenAI-API-compatible server for speech-to-text and text-to-speech that runs locally. | Speaches contributors | open source |
| spikee A Python toolkit for testing LLMs, guardrails and applications against prompt injection and jailbreaks. | Reversec | open source |
| Tabby A self-hosted AI coding assistant server with code completion and chat for editors, run on the organization own hardware. | TabbyML | open source |
| TensorRT-LLM A library from NVIDIA with a Python API for defining and running language models efficiently on NVIDIA GPUs. | NVIDIA | open source |
| Text Embeddings Inference A server from Hugging Face for serving embedding, reranking and sequence classification models. | Hugging Face | open source |
| Text Generation WebUI A local application with a web interface for running language models using several back ends, with an OpenAI-compatible API. | oobabooga | open source |
| Together AI A hosted inference API for open models, with an OpenAI-compatible endpoint, fine-tuning and dedicated endpoints. | Together AI | not open source, hosted service |
| Traceloop Hub A gateway written in Rust that routes to many model providers with OpenTelemetry-based observability. | Traceloop | open source |
| Triton Inference Server An inference server that serves models from several frameworks over HTTP and gRPC, with batching and ensembles. | NVIDIA | open source |
| Vercel AI Gateway A Vercel service that gives one endpoint and key for models from many providers, with budgets and fallbacks. | Vercel | not open source, hosted service |
| Vertex AI Model Garden A Google Cloud catalogue and serving service for Gemini, partner and open models behind the Vertex AI API. | Google Cloud | not open source, hosted service |
| vLLM A high-throughput inference and serving engine for language models, with an OpenAI-compatible server. | vLLM project | open source |
| xAI search tools Server-side tools in the xAI API that let Grok search the web and posts on X during a request. | xAI | not open source, hosted service |
| Xinference A server for deploying language, embedding, reranking, image and audio models locally or on a cluster. | Xorbits | open source |
| Youtu-Agent A Python agent framework from Tencent configured through YAML, aimed at working well with open-source models. | Tencent | open source |