Skip to the content
AI and Cybersecurity Learning and Practice Center Test automation Open the Learning Center

Glossary

LLM-as-a-judge

Using a strong model to grade answers: it picks the better of two, scores one, or compares one with a reference. In the 2023 study that named it, GPT-4 as a judge agreed with people more than 80% of the time, as often as people agree with each other.

Judges have known biases toward an answer's position, its length and their own answers. A judge also reads the text it grades, so it can be prompt-injected: in the JudgeDeceiver research, a crafted sequence inside one candidate answer made the judge pick it.

Open the glossary in the Learning Center