DeepEval vs Langfuse: test framework vs eval platform

As of 2026-11-01, DeepEval and Langfuse solve different halves of LLM evaluation. DeepEval is a test framework: you write test cases in Python, score them with its metrics and fail the build when quality drops. Langfuse is a platform: it collects traces from your app and runs judges, datasets, experiments and human review on them. Many teams will want both, and Langfuse documents how to combine them.

Quick comparison

The order of the items above is editorial, not a ranking.

How you write evals

DeepEval starts from test cases. Its docs list Pytest-style assertions and 50+ ready-to-use metrics, including LLM-as-a-judge, agent, tool-use, multi-turn, voice, RAG, safety and multimodal metrics. You pick one of three scopes. End-to-end treats the app as a black box. Component-level traces the app and scores single spans such as a retriever or planner. Trajectory-based scores an agent's whole chain of plans, tool calls and handoffs. You run it with deepeval test run, and it can also generate synthetic datasets for hard-to-collect cases.

Langfuse starts from traces. Its evaluation overview describes a loop: trace, monitor, build datasets, experiment, evaluate. Evaluators can be LLM-as-a-judge prompts that return numeric, categorical or boolean scores, deterministic code evaluators, or people working through annotation queues. Experiments compare prompt, model or code changes on a dataset, from the UI, the SDK or OpenTelemetry. The docs recommend running judges on observations; trace-level evaluators are deprecated.

Production monitoring

This is the clearest split. DeepEval says it is local-first, and its production story is Confident AI, which surfaces failing production traces so they can become new evals. Langfuse does this in the core product: you set a rule, an evaluator scores matching live observations, and Score Analytics tracks trends. You can also check a judge against human labels.

Using them together

Langfuse's cookbook Evaluate Langfuse LLM Traces with an External Evaluation Pipeline installs langfuse openai deepeval. It fetches traces with the Langfuse client, scores them with a DeepEval GEval custom metric and writes the scores back to the traces. So DeepEval metrics can be your scoring logic while Langfuse stores, charts and reviews the results.

Running it yourself

DeepEval is a pip install. Its FAQ says you only need an LLM provider key for judge metrics, and an account only to send results to Confident AI.

Langfuse self-hosting runs the same stack as Langfuse Cloud: web and worker containers with PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible blob storage. Docker Compose is for local use and testing. Kubernetes (Helm) or cloud templates are the production path. The repository's LICENSE puts everything outside the ee/ directories under MIT, and the self-hosting docs say some add-on features need a license key.

Which to pick

• Pick DeepEval if you want regression tests next to your code, a metric library you don't have to write, and no service to run.
• Pick Langfuse if you need to see and score what happens in production, keep datasets and prompts in one place, or bring in human reviewers.
• Use both if you want DeepEval's metrics and Langfuse's trace store. The cookbook above shows how to connect them.

How we compared

We read the DeepEval introduction and FAQ, Langfuse's evaluation overview, LLM-as-a-judge and self-hosting docs, the cookbook and both repositories' metadata and LICENSE files on 2026-11-01. We did not run benchmarks. Hosted pricing for Confident AI and Langfuse Cloud is outside this comparison.

Tool hubs: DeepEval, Langfuse, and the Agent infra category. Related: Best LLM evaluation framework, DeepEval vs promptfoo, promptfoo alternatives, Opik vs Langfuse, Langfuse alternatives and LLM observability tools.