Glossary
LLM-as-a-judge
Using a strong model to grade answers: it picks the better of two, scores one, or compares one with a reference. In the 2023 study that named it, GPT-4 as a judge agreed with people more than 80% of the time, as often as people agree with each other.
Judges have known biases toward an answer's position, its length and their own answers. A judge also reads the text it grades, so it can be prompt-injected: in the JudgeDeceiver research, a crafted sequence inside one candidate answer made the judge pick it.