llm (CLI)
Use LLMs from the command line and pipe anything into them.
Test and compare LLM prompts and models with assertions, locally or in CI.
No votes yet. Tried it? Be the first to say.
One vote per person. No account needed.
promptfoo runs your prompts against test cases across one or more models, scores the outputs with assertions (exact, regex, model-graded) and shows a side-by-side comparison. It also does red-team scanning.
Eyeballing prompt changes in a playground
You ship features built on LLM prompts and want regressions caught before users see them.
Describe prompts, model providers and test cases in a simple YAML file.
OpenAI, Anthropic, Google, Azure, Bedrock, local models via Ollama, or any HTTP endpoint, including your own app.
Deterministic checks (contains, regex, is-json, JavaScript) and model-graded ones (llm-rubric, factuality, similarity).
A web viewer shows every prompt × model × test case side by side, with pass/fail and cost.
Run evals in CI and fail the build when quality regresses.
Generates adversarial inputs to probe for prompt injection, jailbreaks and data leakage.
You write a promptfooconfig.yaml listing prompts (with {{variables}}), the providers to run them against, and test cases that fill in variables and assert things about the output. promptfoo eval runs the full matrix, caches responses, scores every assertion and stores the results. promptfoo view opens a local web UI to browse and compare them.
description: Support reply summariser
prompts:
- "Summarise this customer message in one sentence for an agent: {{message}}"
providers:
- openai:gpt-5-mini
- ollama:chat:llama3.2
tests:
- vars:
message: "My invoice from March was charged twice and I need a refund."
assert:
- type: icontains
value: refund
- type: javascript
value: output.split('.').filter(Boolean).length <= 2
- type: llm-rubric
value: Mentions the duplicate charge and is a single sentencenpx promptfoo@latest eval
npx promptfoo@latest viewnpx promptfoo@latest init
npx promptfoo@latest eval
npx promptfoo@latest view An eval framework from OpenAI, more research-oriented.
Pytest-style LLM testing for Python projects.
The UK AI Security Institute's framework, strong for agentic and safety evals.
Hosted platforms that combine evals with tracing and dataset management.
Other tools that share topics with promptfoo.
Use LLMs from the command line and pipe anything into them.
An offline API client that stores collections as plain text files in your repo.
A coding-agent skill that turns generic, AI-looking UI into a designed interface.