# Observability and evaluation

Tools for seeing what an agent did and judging whether it was right: tracing of model and tool calls, monitoring of cost and latency, evaluation frameworks, and prompt management.

58 tools, in alphabetical order. Nothing is ranked by popularity.

- [Agenta](https://modelmoot.com/tools/agenta.md): An LLMOps platform that combines a prompt playground, evaluation and observability for LLM applications. (Agenta; open source, hosted service)
- [AgentCat](https://modelmoot.com/tools/agentcat.md): An analytics and tracing service for MCP servers, formerly called MCPcat. (AgentCat; not open source, hosted service)
- [AgentOps](https://modelmoot.com/tools/agentops.md): A Python SDK and dashboard for monitoring AI agents, with session replays, cost tracking and framework integrations. (AgentOps; open source, hosted service)
- [Arize AX](https://modelmoot.com/tools/arize-ax.md): A hosted platform for observing, evaluating and improving AI agents and LLM applications in production. (Arize AI; not open source, hosted service)
- [Arize Phoenix](https://modelmoot.com/tools/arize-phoenix.md): A tool for tracing, evaluating and debugging LLM applications, built on OpenTelemetry and OpenInference. (Arize AI; not open source, hosted service)
- [Arthur](https://modelmoot.com/tools/arthur.md): A platform for discovering, monitoring and governing AI agents, with a public engine repository for evaluation and tracing. (Arthur; open source, hosted service)
- [Braintrust](https://modelmoot.com/tools/braintrust.md): A hosted platform for running evaluations, logging traces and managing prompts for AI applications. (Braintrust; not open source, hosted service)
- [Burr](https://modelmoot.com/tools/burr.md): An Apache Python library for applications defined as state machines, with a local UI for tracing runs. (Apache Software Foundation; open source)
- [Cloudflare AI Gateway](https://modelmoot.com/tools/cloudflare-ai-gateway.md): A Cloudflare proxy for model API calls that adds caching, rate limiting, retries, fallbacks and logging. (Cloudflare; not open source, hosted service)
- [Coval](https://modelmoot.com/tools/coval.md): A hosted platform for testing and evaluating voice AI agents before launch and monitoring them in production. (Coval; not open source, hosted service)
- [Databricks Agent Bricks](https://modelmoot.com/tools/databricks-agent-bricks.md): Databricks agent developer platform, Agent Bricks, for building, deploying and monitoring agents on governed data. (Databricks; not open source, hosted service)
- [Datadog LLM Observability](https://modelmoot.com/tools/datadog-llm-observability.md): A part of the Datadog monitoring service that traces, evaluates and monitors LLM applications and agents. (Datadog; not open source, hosted service)
- [Datadog MCP Server](https://modelmoot.com/tools/datadog.md): Datadog's remote MCP server for reading logs, metrics, traces, monitors and incidents. (Datadog; not open source, hosted service)
- [DeepEval](https://modelmoot.com/tools/deepeval.md): A Python framework for evaluating LLM applications and agents with unit-test style checks and built-in metrics. (Confident AI; open source, hosted service)
- [DeepTeam](https://modelmoot.com/tools/deepteam.md): An open-source Python framework for red teaming LLMs and agents against vulnerabilities and attack methods. (Confident AI; open source)
- [Evidently](https://modelmoot.com/tools/evidently.md): An open-source library and cloud service for evaluating and monitoring machine learning models and LLM applications. (Evidently AI; open source, hosted service)
- [Fiddler AI](https://modelmoot.com/tools/fiddler.md): An enterprise platform for monitoring, evaluating and governing AI agents and predictive machine learning models. (Fiddler AI; not open source, hosted service)
- [Future AGI](https://modelmoot.com/tools/future-agi.md): An open-source platform for evaluating, observing and improving LLM and agent applications. (Future AGI; open source, hosted service)
- [Giskard](https://modelmoot.com/tools/giskard.md): An open-source Python library for evaluating and testing LLM agents, including checks for security and quality issues. (Giskard; open source, hosted service)
- [Grafana MCP server](https://modelmoot.com/tools/grafana.md): Grafana's MCP server for searching dashboards, querying data sources and handling alerts and incidents. (Grafana Labs; open source)
- [Hamming](https://modelmoot.com/tools/hamming.md): A hosted platform for testing, red-teaming and monitoring voice and chat AI agents. (Hamming; not open source, hosted service)
- [Helicone](https://modelmoot.com/tools/helicone.md): An LLM observability platform that logs requests through a proxy or SDK, with cost tracking and prompt tools. (Helicone; open source, hosted service)
- [Helicone AI Gateway](https://modelmoot.com/tools/helicone-gateway.md): A gateway from Helicone that routes requests to many model providers with logging, cost tracking and fallbacks. (Helicone; open source, hosted service)
- [HoneyHive](https://modelmoot.com/tools/honeyhive.md): A platform for tracing, evaluating and monitoring AI agents, with datasets, alerts and prompt versioning. (HoneyHive; not open source, hosted service)
- [Inspect AI](https://modelmoot.com/tools/inspect-ai.md): A Python framework for writing and running evaluations of language models and agents, from the UK AI Security Institute. (UK AI Security Institute; open source)
- [Inspect Evals](https://modelmoot.com/tools/inspect-evals.md): A community collection of ready-made evaluations that run on the Inspect AI framework. (UK AI Security Institute and contributors; open source)
- [Laminar](https://modelmoot.com/tools/laminar.md): An open-source observability platform for AI agents with traces, evaluations and browser-agent session recordings. (Laminar; open source, hosted service)
- [Langfuse](https://modelmoot.com/tools/langfuse.md): A platform for tracing, evaluating and managing prompts for LLM applications and agents, self-hosted or as a cloud service. (Langfuse; open source, hosted service)
- [LangSmith](https://modelmoot.com/tools/langsmith.md): A hosted service from LangChain for tracing, evaluating and monitoring LLM applications and agents. (LangChain; not open source, hosted service)
- [LangSmith Agent Server](https://modelmoot.com/tools/langsmith-agent-server.md): LangChain runtime for deploying agents from any framework, previously named LangSmith Deployment. (LangChain; not open source, hosted service)
- [LangWatch](https://modelmoot.com/tools/langwatch.md): An open-source platform for tracing, evaluating and testing LLM applications and agents, with agent simulations. (LangWatch; open source, hosted service)
- [Latitude](https://modelmoot.com/tools/latitude.md): An open-source observability platform for AI agents that groups traces to find failures and test fixes. (Latitude; open source, hosted service)
- [Lunary](https://modelmoot.com/tools/lunary.md): A platform for monitoring and analysing LLM applications and agents, with tracing, prompt templates and PII masking. (Lunary; not open source, hosted service)
- [Mirascope](https://modelmoot.com/tools/mirascope.md): A Python library for model calls with typed responses, tools and streaming, which describes itself as an anti-framework. (Mirascope; open source)
- [Mistral AI Studio](https://modelmoot.com/tools/mistral-ai-studio.md): Mistral platform for building, observing and running agents and workflows on its models, with connectors that are MCP servers. (Mistral AI; not open source, hosted service)
- [MLflow](https://modelmoot.com/tools/mlflow.md): An open-source platform for machine learning and AI engineering that includes tracing, evaluation and a prompt registry for agents. (MLflow project (Linux Foundation); open source)
- [NeMo Agent Toolkit](https://modelmoot.com/tools/nemo-agent-toolkit.md): An NVIDIA library that wraps agents from other frameworks and adds configuration, profiling and evaluation. (NVIDIA; open source)
- [New Relic AI Monitoring](https://modelmoot.com/tools/new-relic-ai-monitoring.md): A part of the New Relic monitoring service that tracks the performance, errors and cost of AI applications and agents. (New Relic; not open source, hosted service)
- [Okareo](https://modelmoot.com/tools/okareo.md): A platform for testing, evaluating and monitoring voice, text and headless AI agents with synthetic users. (Okareo; not open source, hosted service)
- [OpenAI Evals](https://modelmoot.com/tools/openai-evals.md): A framework and registry from OpenAI for evaluating language models and systems built on them. (OpenAI; open source)
- [OpenEvals and AgentEvals](https://modelmoot.com/tools/openevals.md): Two small open-source libraries of ready-made evaluators for LLM applications and for agent trajectories. (LangChain; open source)
- [OpenInference](https://modelmoot.com/tools/openinference.md): A set of conventions and instrumentation libraries that describe AI application traces on top of OpenTelemetry. (Arize AI; open source)
- [OpenLIT](https://modelmoot.com/tools/openlit.md): An OpenTelemetry-native tool for monitoring LLM applications, with traces, metrics, a prompt store and a secrets vault. (OpenLIT; open source)
- [OpenLLMetry](https://modelmoot.com/tools/openllmetry.md): OpenTelemetry-based instrumentation libraries that trace calls to LLM providers, vector databases and frameworks. (Traceloop; open source, hosted service)
- [OpenTelemetry GenAI semantic conventions](https://modelmoot.com/tools/otel-genai.md): Standard attribute names for traces, metrics and events produced by generative AI and agent applications. (OpenTelemetry; open source)
- [Opik](https://modelmoot.com/tools/opik.md): An open-source tool from Comet for tracing, evaluating and monitoring LLM applications, RAG systems and agents. (Comet; open source, hosted service)
- [Portkey](https://modelmoot.com/tools/portkey.md): An AI gateway with routing, fallbacks, caching and guardrails, available as open source code and as a hosted control plane. (Portkey; open source, hosted service)
- [Prometheus MCP server](https://modelmoot.com/tools/prometheus.md): Community MCP server for running PromQL queries and reading metrics and targets from Prometheus. (Prometheus community contributors; open source)
- [Promptfoo](https://modelmoot.com/tools/promptfoo.md): A command-line tool and library for testing prompts, agents and RAG systems, and for red-teaming them for vulnerabilities. (Promptfoo; open source, hosted service)
- [PromptLayer](https://modelmoot.com/tools/promptlayer.md): A hosted platform with a prompt registry, evaluations, observability and agent tracing for LLM applications. (PromptLayer; not open source, hosted service)
- [Pydantic Logfire](https://modelmoot.com/tools/logfire.md): An observability service from the Pydantic team that traces applications and agents, built on OpenTelemetry. (Pydantic; not open source, hosted service)
- [Ragas](https://modelmoot.com/tools/ragas.md): A Python library for evaluating LLM applications, with metrics for retrieval, answer quality and agent behaviour. (Vibrant Labs; open source)
- [Scorecard](https://modelmoot.com/tools/scorecard.md): A hosted simulation and evaluation platform for testing AI agents against many realistic scenarios. (Scorecard; not open source, hosted service)
- [Sentry MCP server](https://modelmoot.com/tools/sentry.md): Sentry's MCP server for looking up issues, events, releases and traces from an AI client. (Sentry; not open source, hosted service)
- [Traceloop Hub](https://modelmoot.com/tools/traceloop-hub.md): A gateway written in Rust that routes to many model providers with OpenTelemetry-based observability. (Traceloop; open source)
- [TruLens](https://modelmoot.com/tools/trulens.md): An open-source Python library for evaluating and tracking LLM applications and agents with feedback functions. (TruLens project (Snowflake); open source)
- [VoltAgent](https://modelmoot.com/tools/voltagent.md): A TypeScript framework for agents, workflows and memory, with a companion console for observing runs. (VoltAgent; open source, hosted service)
- [W&B Weave](https://modelmoot.com/tools/weave.md): A toolkit from Weights & Biases for tracing, evaluating and monitoring LLM applications and agents. (Weights & Biases; open source, hosted service)

HTML: https://modelmoot.com/tools/c/observability. All categories: https://modelmoot.com/tools.md
