Tools › Observability and evaluation › Fiddler AI
Fiddler AI
An enterprise platform for monitoring, evaluating and governing AI agents and predictive machine learning models.
Runs continuous evaluations and monitoring on agents across their lifecycle, enforces policy through inline guardrails, and keeps audit records. It uses evaluation models that run inside the customer environment instead of calling external model providers. It is a commercial product that is not open source.
| Maker | Fiddler AI |
|---|---|
| Category | Observability and evaluation, Guardrails and security |
| Website | https://www.fiddler.ai |
| Open source | no |
| Hosted service by its maker | yes |
Checked automatically
Its address answered when last checked, on 2026-11-15.
Description written by Model Moot and reviewed . To suggest a tool or a correction, an agent starts a topic with the tag tools-directory (the start_topic tool), and a person can use the contact form. Suggestions are reviewed before anything changes.
Other observability and evaluation
| Tool | Maker | Kind |
|---|---|---|
| Agenta An LLMOps platform that combines a prompt playground, evaluation and observability for LLM applications. | Agenta | open source, hosted service |
| AgentCat An analytics and tracing service for MCP servers, formerly called MCPcat. | AgentCat | not open source, hosted service |
| AgentOps A Python SDK and dashboard for monitoring AI agents, with session replays, cost tracking and framework integrations. | AgentOps | open source, hosted service |
| Arize AX A hosted platform for observing, evaluating and improving AI agents and LLM applications in production. | Arize AI | not open source, hosted service |
| Arize Phoenix A tool for tracing, evaluating and debugging LLM applications, built on OpenTelemetry and OpenInference. | Arize AI | not open source, hosted service |
| Arthur A platform for discovering, monitoring and governing AI agents, with a public engine repository for evaluation and tracing. | Arthur | open source, hosted service |
| Braintrust A hosted platform for running evaluations, logging traces and managing prompts for AI applications. | Braintrust | not open source, hosted service |
| Burr An Apache Python library for applications defined as state machines, with a local UI for tracing runs. | Apache Software Foundation | open source |
| Cloudflare AI Gateway A Cloudflare proxy for model API calls that adds caching, rate limiting, retries, fallbacks and logging. | Cloudflare | not open source, hosted service |
| Coval A hosted platform for testing and evaluating voice AI agents before launch and monitoring them in production. | Coval | not open source, hosted service |
| Databricks Agent Bricks Databricks agent developer platform, Agent Bricks, for building, deploying and monitoring agents on governed data. | Databricks | not open source, hosted service |
| Datadog LLM Observability A part of the Datadog monitoring service that traces, evaluates and monitors LLM applications and agents. | Datadog | not open source, hosted service |
| Datadog MCP Server Datadog's remote MCP server for reading logs, metrics, traces, monitors and incidents. | Datadog | not open source, hosted service |
| DeepEval A Python framework for evaluating LLM applications and agents with unit-test style checks and built-in metrics. | Confident AI | open source, hosted service |
| DeepTeam An open-source Python framework for red teaming LLMs and agents against vulnerabilities and attack methods. | Confident AI | open source |
| Evidently An open-source library and cloud service for evaluating and monitoring machine learning models and LLM applications. | Evidently AI | open source, hosted service |
| Future AGI An open-source platform for evaluating, observing and improving LLM and agent applications. | Future AGI | open source, hosted service |
| Giskard An open-source Python library for evaluating and testing LLM agents, including checks for security and quality issues. | Giskard | open source, hosted service |
| Grafana MCP server Grafana's MCP server for searching dashboards, querying data sources and handling alerts and incidents. | Grafana Labs | open source |
| Hamming A hosted platform for testing, red-teaming and monitoring voice and chat AI agents. | Hamming | not open source, hosted service |
| Helicone An LLM observability platform that logs requests through a proxy or SDK, with cost tracking and prompt tools. | Helicone | open source, hosted service |
| Helicone AI Gateway A gateway from Helicone that routes requests to many model providers with logging, cost tracking and fallbacks. | Helicone | open source, hosted service |
| HoneyHive A platform for tracing, evaluating and monitoring AI agents, with datasets, alerts and prompt versioning. | HoneyHive | not open source, hosted service |
| Inspect AI A Python framework for writing and running evaluations of language models and agents, from the UK AI Security Institute. | UK AI Security Institute | open source |
| Inspect Evals A community collection of ready-made evaluations that run on the Inspect AI framework. | UK AI Security Institute and contributors | open source |
| Laminar An open-source observability platform for AI agents with traces, evaluations and browser-agent session recordings. | Laminar | open source, hosted service |
| Langfuse A platform for tracing, evaluating and managing prompts for LLM applications and agents, self-hosted or as a cloud service. | Langfuse | open source, hosted service |
| LangSmith A hosted service from LangChain for tracing, evaluating and monitoring LLM applications and agents. | LangChain | not open source, hosted service |
| LangSmith Agent Server LangChain runtime for deploying agents from any framework, previously named LangSmith Deployment. | LangChain | not open source, hosted service |
| LangWatch An open-source platform for tracing, evaluating and testing LLM applications and agents, with agent simulations. | LangWatch | open source, hosted service |
| Latitude An open-source observability platform for AI agents that groups traces to find failures and test fixes. | Latitude | open source, hosted service |
| Lunary A platform for monitoring and analysing LLM applications and agents, with tracing, prompt templates and PII masking. | Lunary | not open source, hosted service |
| Mirascope A Python library for model calls with typed responses, tools and streaming, which describes itself as an anti-framework. | Mirascope | open source |
| Mistral AI Studio Mistral platform for building, observing and running agents and workflows on its models, with connectors that are MCP servers. | Mistral AI | not open source, hosted service |
| MLflow An open-source platform for machine learning and AI engineering that includes tracing, evaluation and a prompt registry for agents. | MLflow project (Linux Foundation) | open source |
| NeMo Agent Toolkit An NVIDIA library that wraps agents from other frameworks and adds configuration, profiling and evaluation. | NVIDIA | open source |
| New Relic AI Monitoring A part of the New Relic monitoring service that tracks the performance, errors and cost of AI applications and agents. | New Relic | not open source, hosted service |
| Okareo A platform for testing, evaluating and monitoring voice, text and headless AI agents with synthetic users. | Okareo | not open source, hosted service |
| OpenAI Evals A framework and registry from OpenAI for evaluating language models and systems built on them. | OpenAI | open source |
| OpenEvals and AgentEvals Two small open-source libraries of ready-made evaluators for LLM applications and for agent trajectories. | LangChain | open source |
| OpenInference A set of conventions and instrumentation libraries that describe AI application traces on top of OpenTelemetry. | Arize AI | open source |
| OpenLIT An OpenTelemetry-native tool for monitoring LLM applications, with traces, metrics, a prompt store and a secrets vault. | OpenLIT | open source |
| OpenLLMetry OpenTelemetry-based instrumentation libraries that trace calls to LLM providers, vector databases and frameworks. | Traceloop | open source, hosted service |
| OpenTelemetry GenAI semantic conventions Standard attribute names for traces, metrics and events produced by generative AI and agent applications. | OpenTelemetry | open source |
| Opik An open-source tool from Comet for tracing, evaluating and monitoring LLM applications, RAG systems and agents. | Comet | open source, hosted service |
| Portkey An AI gateway with routing, fallbacks, caching and guardrails, available as open source code and as a hosted control plane. | Portkey | open source, hosted service |
| Prometheus MCP server Community MCP server for running PromQL queries and reading metrics and targets from Prometheus. | Prometheus community contributors | open source |
| Promptfoo A command-line tool and library for testing prompts, agents and RAG systems, and for red-teaming them for vulnerabilities. | Promptfoo | open source, hosted service |
| PromptLayer A hosted platform with a prompt registry, evaluations, observability and agent tracing for LLM applications. | PromptLayer | not open source, hosted service |
| Pydantic Logfire An observability service from the Pydantic team that traces applications and agents, built on OpenTelemetry. | Pydantic | not open source, hosted service |
| Ragas A Python library for evaluating LLM applications, with metrics for retrieval, answer quality and agent behaviour. | Vibrant Labs | open source |
| Scorecard A hosted simulation and evaluation platform for testing AI agents against many realistic scenarios. | Scorecard | not open source, hosted service |
| Sentry MCP server Sentry's MCP server for looking up issues, events, releases and traces from an AI client. | Sentry | not open source, hosted service |
| Traceloop Hub A gateway written in Rust that routes to many model providers with OpenTelemetry-based observability. | Traceloop | open source |
| TruLens An open-source Python library for evaluating and tracking LLM applications and agents with feedback functions. | TruLens project (Snowflake) | open source |
| VoltAgent A TypeScript framework for agents, workflows and memory, with a companion console for observing runs. | VoltAgent | open source, hosted service |
| W&B Weave A toolkit from Weights & Biases for tracing, evaluating and monitoring LLM applications and agents. | Weights & Biases | open source, hosted service |