DeepEval adds an evaluation setup wizard and voice metrics
Updated DeepEval
Summary
The setup-v0.1.0 release adds deepeval setup, transcription and sound-glitch metrics, plus audio input support for evaluation models.
- Tool
- DeepEval
- Category
- Agent infra
- Date
- Open source
- Yes
- License
- Apache-2.0
- Pricing
- Open-source framework; hosted Confident AI and model-provider charges are separate.
DeepEval published the setup-v0.1.0 release on October 16, 2026. This is the repository’s setup release tag, not a claim that the main Python package has reset to version 0.1.0.
What changed
The release adds an evaluation setup wizard and the deepeval setup command. Its changelog also includes transcription-accuracy and sound-glitch metrics, user transcript support in voice connectors, audio input for evaluation models and audio support in ConversationalGEval.
For existing test suites, the notes describe forwarding a pulled dataset’s version with test runs, accepting metadata and tags in expectations, and exposing verdict information. Other fixes address phone-call lifecycle handling, voice detectors and model registry compatibility.
Where it fits
These features help teams configure evaluations and inspect voice-agent outputs. They do not establish that a particular voice model performs well: teams still need representative test cases, suitable evaluation models and review of automated verdicts. This is a release-note summary, not a hands-on benchmark.
License and cost
The release license is Apache-2.0. Running the framework does not make paid evaluation-model calls or hosted Confident AI services free. The cited release does not specify a hosted allowance.
See the DeepEval hub and agent infrastructure.
Questions
What is DeepEval?
Its docs say "DeepEval is an open-source LLM evaluation framework for LLM applications."
Is DeepEval open source?
Yes. The GitHub API reports the license as Apache-2.0 (Apache License 2.0), and PyPI also lists Apache-2.0.
How is DeepEval used?
Its docs say you can "Unit test LLM outputs with Pytest-style assertions" and run end-to-end, trajectory-based, and component-level evals. The quickstart installs DeepEval, creates a test case, and runs it with deepeval test run.
Filed on Agent Radar. Claims in this note are limited to the sources above.