Tools › Supports OpenAI-compatible API

Tools that support OpenAI-compatible API

An OpenAI-compatible API accepts the same requests as OpenAI's chat or responses endpoints, so clients written for one work with the other. These tools in the directory say they support it.

91 tools, in alphabetical order. Nothing is ranked by popularity, and no listing is paid for.

ToolMakerKind
Agent Router
A project, formerly Envoy AI Gateway, that gives agents one OpenAI-compatible API for many models and one router for MCP tools.
Agentic AI Foundationopen source
agentgateway
An open-source proxy for traffic from AI agents, covering LLM, MCP, A2A and plain HTTP calls.
agentgateway projectopen source
AI SDK
A TypeScript toolkit for calling models, streaming results, using tools and building agents, with UI hooks for React and other frameworks.
Vercelopen source
Amazon Bedrock
An AWS service that gives access to models from several providers through one API, with agents and guardrails.
Amazon Web Servicesnot open source, hosted service
any-llm
A Python library that gives one interface to many model providers, with an optional gateway server.
Mozilla.aiopen source
Baseten
A hosted platform for serving models with model APIs and custom deployments, with an OpenAI-compatible endpoint.
Basetennot open source, hosted service
BentoML
A Python framework for packaging models and services into deployable containers, with a hosted cloud.
BentoMLopen source, hosted service
Bifrost
An AI gateway written in Go that provides one OpenAI-compatible API to many providers, and also acts as an MCP gateway.
Maxim AIopen source
bolt.diy
An open-source version of the Bolt app builder that runs on a local machine and lets the user choose the model provider.
StackBlitz Labsopen source
Cerebras Inference
A hosted inference API for open models that runs on Cerebras wafer-scale hardware, with an OpenAI-compatible endpoint.
Cerebrasnot open source, hosted service
Cloudflare AI Gateway
A Cloudflare proxy for model API calls that adds caching, rate limiting, retries, fallbacks and logging.
Cloudflarenot open source, hosted service
Cloudflare Workers AI
A Cloudflare service that runs open models on its network, called from Workers or over a REST API.
Cloudflarenot open source, hosted service
DeepInfra
A hosted inference API for open models with an OpenAI-compatible endpoint and dedicated deployments.
DeepInfranot open source, hosted service
Docker Model Runner
A Docker feature that pulls and runs models from registries on the local machine with an OpenAI-compatible API.
Dockeropen source
exo
Software that connects several local devices into one cluster that runs large models together.
exo labsopen source
Fireworks AI
A hosted inference platform for open models with an OpenAI-compatible API, fine-tuning and function calling.
Fireworks AInot open source, hosted service
GPT4All
A desktop application and library for running language models locally on CPUs and GPUs, with local document chat.
Nomicopen source
GPUStack
A manager for GPU clusters that deploys and serves models with vLLM, SGLang and other engines.
GPUStackopen source
Groq
A hosted inference API that serves open models on its own language processing hardware, with an OpenAI-compatible endpoint.
Groqnot open source, hosted service
Guardrails AI
A Python framework that checks and constrains LLM inputs and outputs with validators from a shared hub.
Guardrails AIopen source
Helicone AI Gateway
A gateway from Helicone that routes requests to many model providers with logging, cost tracking and fallbacks.
Heliconeopen source, hosted service
Higress
A cloud-native API gateway based on Envoy and Istio, with an AI gateway mode and an MCP server hosting feature.
Alibabaopen source
Hugging Face Inference Providers
A Hugging Face service that gives one API and token for serverless inference from several partner providers.
Hugging Facenot open source, hosted service
Jan
An open source desktop application that runs language models locally and can connect to remote providers, with a local API server.
Menlo Researchopen source
KoboldCpp
A single-file program that runs GGUF models locally with a web interface and several API styles.
LostRuinsopen source
Kong AI Gateway
A set of Kong Gateway plugins that route and govern traffic to model providers behind a unified interface.
Kongopen source, hosted service
KServe
A Kubernetes custom resource framework for serving predictive and generative models, with autoscaling.
KServe projectopen source
Lagent
A lightweight Python framework from the InternLM team for agents with tool use and asynchronous execution.
InternLMopen source
Langroid
A Python framework for multi-agent programming in which agents are actors that exchange messages.
Langroidopen source
Language Model Evaluation Harness
A command-line framework for running language models against hundreds of standard evaluation tasks.
EleutherAIopen source
Lemonade
A local server that runs models on AMD GPUs and NPUs and other hardware, with an OpenAI-compatible API.
Lemonade SDKopen source
LiteLLM
A Python SDK and proxy server that calls many model providers in the OpenAI format, with keys, budgets and routing.
BerriAIopen source, hosted service
LitServe
A Python framework for building custom model inference servers on FastAPI, with batching and GPU autoscaling.
Lightning AIopen source
llama-swap
A proxy that starts and swaps local model servers on demand behind one OpenAI-compatible endpoint.
mostlygeekopen source
llama.cpp
A C and C++ library and server for running language models in the GGUF format on CPUs and many GPUs.
ggml-orgopen source
llamafile
A way of distributing and running a model as one executable file that works across several operating systems.
Mozilla.aiopen source
LLM Gateway
A gateway that routes requests to many providers through one OpenAI-compatible API, hosted or self-hosted.
LLM Gatewayopen source, hosted service
llm-d
A Kubernetes-native framework for distributed model serving, built on vLLM, with cache-aware routing.
llm-d projectopen source
LM Studio
A desktop application and local server for downloading and running language models, with a model catalogue.
LM Studionot open source
LMDeploy
A toolkit for compressing, deploying and serving language models, with two inference engines.
Shanghai AI Laboratoryopen source
LocalAI
A self-hosted server that runs language, image, speech and embedding models behind OpenAI-compatible APIs.
LocalAIopen source
Microsoft Foundry
A Microsoft Azure platform that hosts models from several providers behind one endpoint, with agent tooling.
Microsoftnot open source, hosted service
MLC LLM
A compiler-based engine for deploying language models on many devices, including phones, browsers and GPUs.
MLCopen source
MLX LM
A Python package for running and fine-tuning language models on Apple silicon with the MLX framework.
Appleopen source
MS-Agent
A lightweight Python framework from ModelScope for agents that carry out complex tasks with tools.
ModelScopeopen source
Nebius Token Factory
A Nebius hosted inference service for open models with an OpenAI-compatible API, fine-tuning and dedicated endpoints.
Nebiusnot open source, hosted service
NeMo Guardrails
A toolkit for adding programmable rails to LLM applications, written in a modelling language called Colang.
NVIDIAopen source
New API
A self-hosted LLM gateway and asset management system with OpenAI-, Claude- and Gemini-compatible endpoints.
QuantumNousopen source
Novita AI
A hosted platform with an OpenAI-compatible LLM API, GPU instances and agent sandboxes.
Novita AInot open source, hosted service
NVIDIA Dynamo
A distributed inference framework that spreads serving of large models over many GPUs, with disaggregated prefill and decode.
NVIDIAopen source
Ollama
A tool that downloads and runs language models locally, with a command line, a REST API and an OpenAI-compatible endpoint.
Ollamaopen source, hosted service
One API
A self-hosted system that distributes access to many model providers through one OpenAI-style API with key management.
songquanpengopen source
Open Responses
A specification for a model API interface based on the OpenAI Responses API that several providers can implement.
Open Responses projectopen source
Open WebUI
A self-hosted web interface for models that connects to Ollama and OpenAI-compatible servers, with RAG and tools.
Open WebUInot open source
OpenAI Agents SDK for Python
OpenAI's Python library for agents with tools, handoffs between agents, guardrails and tracing.
OpenAIopen source
OpenAI Agents SDK for TypeScript
OpenAI's TypeScript library for agents with tools, handoffs, guardrails, tracing and realtime voice.
OpenAIopen source
OpenAI computer use
A tool in the OpenAI API that lets a model propose clicks, typing and scrolling from screenshots to operate a browser or desktop.
OpenAInot open source, hosted service
OpenAI Evals
A framework and registry from OpenAI for evaluating language models and systems built on them.
OpenAIopen source
OpenAI Guardrails
Libraries for Python and TypeScript that wrap OpenAI client calls with configurable safety checks.
OpenAIopen source
OpenAI web search tool
A built-in tool in the OpenAI Responses API that searches the web and returns answers with citations.
OpenAInot open source, hosted service
opencode
An open-source terminal coding agent that works with models from many providers and also runs as a desktop app and editor extension.
Anomalyopen source, hosted service
OpenRouter
A hosted API that routes requests to models from many providers through one OpenAI-compatible endpoint.
OpenRouternot open source, hosted service
OpenServ
An agent infrastructure platform with a reasoning API, an agent builder and events for developers of agent services.
OpenServnot open source, hosted service
Perplexity API
APIs from Perplexity that return web search results and answers with citations, plus an MCP server.
Perplexitynot open source, hosted service
Plano
An AI-native proxy and data plane for agentic apps that handles model routing, orchestration and observability.
Katanemoopen source
Portkey
An AI gateway with routing, fallbacks, caching and guardrails, available as open source code and as a hosted control plane.
Portkeyopen source, hosted service
PR-Agent
An open-source tool that reviews, describes and suggests improvements to pull requests using a language model.
The PR-Agent projectopen source
Pydantic AI
A Python agent framework from the Pydantic team that uses typed inputs, outputs and dependencies.
Pydanticopen source
Qwen Code
An open-source terminal coding agent from the Qwen team, tuned for Qwen models and usable with OpenAI-compatible endpoints.
Alibaba Cloudopen source
Qwen-Agent
A Python framework from the Qwen team for agents with function calling, MCP, code execution and retrieval.
Alibaba Cloud Qwenopen source
RamaLama
A command-line tool that runs models in OCI containers using a runtime picked for the detected hardware.
Containers projectopen source
Ray Serve
A model serving library on Ray that scales Python services across a cluster, with an LLM serving module.
Anyscaleopen source, hosted service
Requesty
A hosted LLM gateway with one API for many models, with routing, fallbacks, caching and cost controls.
Requestynot open source, hosted service
RouteLLM
A framework for serving and evaluating routers that send each query to a stronger or cheaper model.
LMSYSopen source
SambaNova Cloud
A hosted inference API for open models that runs on SambaNova hardware, with an OpenAI-compatible endpoint.
SambaNovanot open source, hosted service
SGLang
A serving framework for language and multimodal models with a fast runtime and a structured generation language.
SGLang projectopen source
Speaches
An OpenAI-API-compatible server for speech-to-text and text-to-speech that runs locally.
Speaches contributorsopen source
spikee
A Python toolkit for testing LLMs, guardrails and applications against prompt injection and jailbreaks.
Reversecopen source
Tabby
A self-hosted AI coding assistant server with code completion and chat for editors, run on the organization own hardware.
TabbyMLopen source
TensorRT-LLM
A library from NVIDIA with a Python API for defining and running language models efficiently on NVIDIA GPUs.
NVIDIAopen source
Text Embeddings Inference
A server from Hugging Face for serving embedding, reranking and sequence classification models.
Hugging Faceopen source
Text Generation WebUI
A local application with a web interface for running language models using several back ends, with an OpenAI-compatible API.
oobaboogaopen source
Together AI
A hosted inference API for open models, with an OpenAI-compatible endpoint, fine-tuning and dedicated endpoints.
Together AInot open source, hosted service
Traceloop Hub
A gateway written in Rust that routes to many model providers with OpenTelemetry-based observability.
Traceloopopen source
Triton Inference Server
An inference server that serves models from several frameworks over HTTP and gRPC, with batching and ensembles.
NVIDIAopen source
Vercel AI Gateway
A Vercel service that gives one endpoint and key for models from many providers, with budgets and fallbacks.
Vercelnot open source, hosted service
Vertex AI Model Garden
A Google Cloud catalogue and serving service for Gemini, partner and open models behind the Vertex AI API.
Google Cloudnot open source, hosted service
vLLM
A high-throughput inference and serving engine for language models, with an OpenAI-compatible server.
vLLM projectopen source
xAI search tools
Server-side tools in the xAI API that let Grok search the web and posts on X during a request.
xAInot open source, hosted service
Xinference
A server for deploying language, embedding, reranking, image and audio models locally or on a cluster.
Xorbitsopen source
Youtu-Agent
A Python agent framework from Tencent configured through YAML, aimed at working well with open-source models.
Tencentopen source