1. Golden Dataset
5 entriesA fixed set of real queries, edge cases and expert answers. Every prompt version runs the same tests, so scores are comparable over time.
2. Exact Match
3 entriesDeterministic checks for answers with one right output: equals, regex and JSON schema. Fast and cheap.
3. LLM-as-a-Judge
5 entriesA strong model grades open-ended answers against a clear rubric for accuracy, completeness and tone. Calibrate it on human labels.
4. Human Review
2 entriesExperts label samples to catch what metrics miss. Hard cases go back into the golden dataset.
5. Regression Tests
3 entriesRe-run the eval suite on every prompt or model change. If the score drops below the baseline, block the release.
6. Hallucination Checks
5 entriesSplit the answer into claims and verify each one against the source. Two of three claims supported means the answer is two-thirds faithful.
7. A/B Testing
2 entriesShip two versions to real users and compare outcomes, not opinions.
8. Online Monitoring
2 entriesScore live traffic continuously for quality, latency and cost. Get alerted when quality drifts.
9. Popular Eval Tools
9 entriesNine tools that cover most of the methods above. Release facts are from each project's GitHub page on 20 November 2026.
Frequently Asked Questions
What are the main methods for testing LLMs?
Nine are common: a golden dataset, exact match, LLM-as-a-judge, human review, regression tests, hallucination checks, A/B testing, online monitoring and a tool to run them. Each catches a different kind of failure, so teams combine several instead of relying on one score.
How do I build and maintain a golden dataset?
Collect real queries, edge cases and expert answers, and run every prompt or model version against the same set. When human review or production monitoring finds a hard case, add it to the dataset so it is tested from then on.
When should I use exact match, LLM-as-a-judge or human review?
Use exact match when there is one right output, such as an equals check, a regex or a JSON schema, because it is fast and cheap. Use an LLM judge with a clear rubric for open-ended answers, and calibrate it on human labels. Use expert human review on samples to catch what automated metrics miss.
How do I test an LLM for hallucinations?
Split the answer into individual claims and check each against the source. If two of three claims are supported, the answer is two-thirds faithful. Ragas and DeepEval offer faithfulness metrics, and FActScore and SelfCheckGPT are research approaches to the same problem.
How do regression tests, A/B tests and online monitoring fit together?
Regression tests run offline on every change and block a release if the score drops below the baseline. A/B tests ship two versions to real users and compare outcomes. Online monitoring scores live traffic for quality, latency and cost and alerts you when quality drifts.