Skip to content
Agent Radar

promptfoo 0.125.0 makes Claude Agent and Codex SDKs opt-in

Updated promptfoo

Summary

promptfoo 0.125.0 (October 23, 2026) makes the Claude Agent, Codex and jks-js packages opt-in, adds a safe mode and reports flaky repeated tests.

Tool
promptfoo
Category
Agent infra
Date
Open source
Yes
License
MIT
Pricing
Not stated in the sources we cited.
Sources

promptfoo published 0.125.0 on October 23, 2026, its first minor release since 0.124.1 on October 8. It carries two breaking changes, both about dependencies that ordinary installs no longer pull in, plus new eval features. A 0.125.1 patch followed on October 25 with a single fix that restores the project's Discord invite link.

Release facts

FactDetail
Productpromptfoo, an open-source CLI and library for evaluating and red-teaming LLM apps
Version0.125.0, followed by 0.125.1
Release dateOctober 23, 2026 (0.125.1 on October 25)
LicenseMIT, per the GitHub API and npm
Node.js>=22.22.0, per the npm package

Breaking changes

  • Claude Agent and Codex SDKs are opt-in. They become optional peer dependencies, so a normal install no longer brings in those SDKs and their native CLIs. If you use either provider, install the SDK next to promptfoo. According to PR 11134, the supported ranges are Claude Agent SDK ^0.3.273 and Codex SDK ^0.156.1, and implicit Codex grading is only selected when a compatible SDK is present.
  • jks-js is opt-in. Java KeyStore authentication now needs jks-js installed separately. The PR says this removes jks-js and its node-forge dependency from default installs; teams can also convert keystores to PEM or PFX with Java's keytool.

What's new

  • Safe mode. --safe-mode, PROMPTFOO_SAFE_MODE=true or commandLineOptions.safeMode: true blocks string-based JavaScript assertions and transforms. PR 11382 is explicit that this is not a sandbox: executable providers, config modules, Python and Ruby assertions and HTTP parsers still run.
  • Repeat stability. Repeated evals report which test, variable, provider and prompt groups mixed passes and failures. The CLI lists up to five unstable groups, and JSON exports hold every group under results.repeatStability.
  • Cache-hit disclosure. The eval summary shows how many rows came from the response cache, so cached rows are no longer mistaken for fresh measurements.
  • New assertions. rouge-l and rouge-s, trace-span filtering by attributes, and optional Unicode normalization for equals and contains.
  • Provider-only environment overrides for applications that embed promptfoo and want to pass credentials without exposing them to templates or subprocesses.
  • Providers. Current Gemini media models, portable OpenAI config schemas and templated request metadata for the A2A provider.

Questions

Why did my Claude Agent or Codex provider break after upgrading promptfoo? Since 0.125.0 those SDKs are optional peer dependencies. Install the Claude Agent SDK (^0.3.273) or Codex SDK (^0.156.1) alongside promptfoo.

Does promptfoo safe mode make untrusted configs safe to run? No. It only blocks inline JavaScript assertions and transforms. The pull request states that other execution paths stay enabled.

Should I install 0.125.0 or 0.125.1? 0.125.1. Its only change is the Discord invite fix, so the features are those of 0.125.0.

More on the promptfoo hub, the previous promptfoo 0.124.1 launch, DeepEval vs promptfoo and promptfoo alternatives.

Alternatives

promptfoo compared with its alternatives
ToolLicensePricing note
promptfooThis launchpromptfoo is an open-source CLI and library for evaluating and red-teaming LLM apps.
MITNot stated
DeepEvalDeepEval is an Apache-2.0 open-source Python framework for evaluating LLM applications and agents.
Apache-2.0Not stated
LangfuseOpen source platform for tracing, evaluating and improving AI agents and LLM applications.
MITNot stated
OpikOpen-source platform for tracing, evaluating and optimizing LLM applications and agents.
Apache-2.0Not stated

More from promptfoo

View all

Questions

What is promptfoo?

Its docs say "promptfoo is an open-source CLI and library for evaluating and red-teaming LLM apps."

Is promptfoo open source?

Yes. Its docs describe it as open-source, and the GitHub API and npm both report the MIT license.

How is promptfoo used?

Its docs say it can be used "as a CLI, library, or in CI/CD". The workflow is to define test cases, configure the evaluation with prompts, test cases and API providers, run it, then analyze results.

Filed on Agent Radar. Claims in this note are limited to the sources above.