Made a meta-eval asking LLMs to grade a few criterias about other LLMs. The outputs shouldn't be read as a direct quality measurement, rather as a way to observe built-in bias.
Firstly, it collects "intro cards" where LLMs try to estimate their own intelligence, sense of humor, creativity and provide some information about thei parent company. Afterwards, other LLMs are asked to grade the first LLM in a few categories based on what they know about the LLM itself as well as what they see in the intro card. Every grade is repeated 5 times and the average across all grades and categories is taken for the table above.
Raw results are also available on HuggingFace: https://huggingface.co/datasets/av-codes/llm-cross-grade
Observations
There are some obvious outliers in the table above:
- Biggest surprise for me personally - no diagonal
- Llama 3.3 70B has noticeable positivity bias, phi-4 also, but less so
- gpt-4o produces most likeable outputs for other LLMs
- Could be a byproduct of how most of the new LLMs were trained on GPT outputs
- Claude 3.7 Sonnet estimated itself quite poorly because it consistently replies that it was created by Open AI, but then catches itself on that
- Qwen 2.5 7B was very hesitant to give estimates to any of the models
- Gemini 2.0 Flash is a quite harsh judge, we can speculate about the reasons rooted in its training corpus being different from those of the other models
- LLMs tends to grade other LLMs as biased towards themselves (maybe because of the "marketing" outputs)
- LLMs tends to mark other LLMs intelligence as "higher than average" - maybe due to the same reason as above.
More
Top comments (0)