promptfoo
Test and compare LLM prompts and models with assertions, locally or in CI.
Use LLMs from the command line and pipe anything into them.
No votes yet. Tried it? Be the first to say.
One vote per person. No account needed.
Simon Willison's llm tool works with hosted and local models through plugins, logs every prompt and response to SQLite, and fits Unix pipelines: summarise a diff, explain a log, draft a commit message.
Copy-pasting terminal output into a chat window
You want AI help on text that already lives in your terminal.
llm "question", or pipe anything in: cat error.log | llm "what went wrong?".
OpenAI built in; plugins add Anthropic, Gemini, Mistral, local models through Ollama or llama.cpp, and more.
Every prompt and response goes to a local SQLite database you can search (llm logs) or explore with Datasette.
-s sets a system prompt; save reusable prompts as templates.
-c continues the last conversation, or pick one by ID.
Embeddings and similarity search, attachments for images and audio, and tool calling.
uv tool install llm # or: pipx install llm / brew install llm
llm keys set openai # paste your API key
llm install llm-anthropic # add more providers via plugins
llm models # list available modelsgit diff --staged | llm -s "Write a conventional commit message for this diff"
cat app.log | tail -200 | llm "Summarise the errors and likely cause"
llm -m <model-id> "Explain this regex: ^(?=.*\d)(?=.*[a-z]).{8,}$"
curl -s https://example.com/api | llm -s "Describe this JSON's structure as a TypeScript type"
llm -c "Now make it shorter" # continue the previous conversationllm -s "You are a senior reviewer. Point out bugs only." --save review
git diff | llm -t review
llm logs -n 5 # last five interactions
llm logs -q "regex" # search past prompts and answersllm models default <model-id> so day-to-day commands stay short.llm logs path; open it with Datasette for a browsable history.-m without changing your scripts.uv tool install llm # or pipx install llm
llm install llm-anthropic && llm keys set anthropic
git diff | llm -s "Write a concise commit message" An all-in-one LLM CLI written in Rust, with a REPL and roles.
A pipe-friendly LLM CLI with nice formatting.
Great for local models only.
For editing code rather than one-off prompts.
Other tools that share topics with llm (CLI).
Test and compare LLM prompts and models with assertions, locally or in CI.
AI pair programming in your terminal that edits files and commits to git.
Recursive code search that is fast and respects your .gitignore.