Polish EQ-Bench Leaderboard

Leaderboard was created as part of an open-science project SpeakLeash.org

Polish Emotional Intelligence Benchmark for LLMs

Help us develop Polish Large Language Model Bielik by using Arena.

We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2024/016951.

Model	Params	Benchmark Score	Percentage Questions Parseable	Error
TeeZee/Bielik-SOLAR-LIKE-10.7B-Instruct-v0.1	46.7	59.579532163742684	51.461988304093566	141.0 questions were parseable (min is 83%)

Model	Params	Benchmark Score	Percentage Questions Parseable	Error
mistralai/Mistral-Large-Instruct-2407	123	78.07	100.00
meta-llama/Meta-Llama-3.1-405B-Instruct-FP8	405	77.23	100.00
gpt-4o-2024-08-06	nan	75.15	100.00
gpt-4-turbo-2024-04-09	nan	74.59	95.91
meta-llama/Meta-Llama-3.1-70B-Instruct	70	72.53	100.00
Qwen/Qwen2-72B-Instruct	72	71.23	98.83
meta-llama/Meta-Llama-3-70B-Instruct	70	71.21	100.00
gpt-4o-mini-2024-07-18	nan	71.15	100.00
speakleash/Bielik-11B-v2.3-Instruct	11	70.86	100.00
alpindale/WizardLM-2-8x22B	141	69.56	100.00
speakleash/Bielik-11B-v2.2-Instruct	11	69.05	100.00
Qwen/Qwen2-72B	72	68.93	98.83
speakleash/Bielik-11B-v2.0-Instruct	11	68.24	100.00
Qwen/Qwen1.5-72B-Chat	72	68.03	100.00
mistralai/Mixtral-8x22B-Instruct-v0.1	141	67.63	100.00
speakleash/Bielik-11B-v2.1-Instruct	11	60.07	90.64
Qwen/Qwen1.5-32B-Chat	32	59.63	98.25
openchat/openchat-3.5-0106-gemma	7	59.58	99.42
openchat/openchat-3.5-0106-gemma	7	59.58	99.42
openchat/openchat-3.5-0106-gemma	7	59.41	98.83
openchat/openchat-3.5-0106-gemma	7	59.41	98.83
CohereForAI/aya-23-35B	35	58.41	100.00
openchat/openchat-3.5-0106-gemma	7	57.93	98.83
gpt-3.5-turbo	nan	57.70	100.00
Qwen/Qwen2-57B-A14B-Instruct	57	57.64	100.00
mistralai/Mixtral-8x7B-Instruct-v0.1	47	57.61	98.25
CohereForAI/c4ai-command-r-v01	35	56.43	100.00
microsoft/Phi-3-medium-4k-instruct	14	56.40	98.83
upstage/SOLAR-10.7B-Instruct-v1.0	11	55.21	95.91
NousResearch/Hermes-2-Theta-Llama-3-8B	8	54.88	100.00
mlabonne/NeuralDaredevil-8B-abliterated	8	54.74	100.00
NousResearch/Hermes-2-Pro-Llama-3-8B	8	54.57	100.00
upstage/SOLAR-10.7B-Instruct-v1.0	11	54.33	94.74
Qwen/Qwen1.5-32B	32	54.03	99.42
Qwen/Qwen2-7B-Instruct	7	53.74	100.00
Qwen/Qwen2-7B-Instruct	7	53.08	100.00
google/recurrentgemma-9b-it	9	52.82	100.00
Qwen/Qwen1.5-72B	72	51.44	95.32
berkeley-nest/Starling-LM-7B-alpha	7	49.63	100.00
NousResearch/Nous-Hermes-2-SOLAR-10.7B	11	49.27	98.83
openchat/openchat-3.5-1210	7	49.04	100.00
lex-hue/Delexa-7b	7	48.46	98.83
Qwen/Qwen1.5-14B-Chat	14	47.96	93.57
NousResearch/Nous-Hermes-2-SOLAR-10.7B	11	47.66	98.83
mistralai/Mistral-7B-Instruct-v0.2	7	47.02	88.30
meta-llama/Meta-Llama-3-8B-Instruct	8	46.53	100.00
01-ai/Yi-1.5-9B-Chat	9	46.50	95.32
01-ai/Yi-1.5-34B-Chat	34	46.32	100.00
meta-llama/Meta-Llama-3-8B-Instruct	8	46.27	100.00
berkeley-nest/Starling-LM-7B-alpha	7	46.26	100.00
mistralai/Mistral-7B-Instruct-v0.2	7	45.86	86.55
CohereForAI/aya-23-8B	8	45.43	100.00
openchat/openchat-3.5-0106	7	45.42	99.42
openchat/openchat-3.5-0106	7	45.42	99.42
openchat/openchat-3.5-0106	7	45.42	99.42
mistralai/Mistral-7B-Instruct-v0.3	7	45.21	100.00
Remek/Kruk-7B-SP-001	7	44.44	100.00
openchat/openchat-3.5-0106	7	43.81	100.00
Nexusflow/Starling-LM-7B-beta	7	43.78	97.08
Remek/OpenChat3.5-0106-Spichlerz-Bocian	7	42.84	97.08
tiiuae/falcon-11B	11	42.41	100.00
Nexusflow/Starling-LM-7B-beta	7	41.76	92.98
Remek/OpenChat3.5-0106-Spichlerz-Inst-001	7	41.60	100.00
internlm/internlm2-chat-7b-sft	7	41.38	99.42
internlm/internlm2-chat-7b	7	39.53	98.83
nvidia/Llama3-ChatQA-1.5-8B	8	39.36	97.08
meta-llama/Meta-Llama-3-70B	70	39.09	84.80
teknium/OpenHermes-2.5-Mistral-7B	7	37.48	100.00
internlm/internlm2-chat-20b	20	36.31	99.42
Qwen/Qwen2-7B	7	35.51	97.08
teknium/OpenHermes-13B	13	34.91	94.74
TeeZee/Bielik-SOLAR-LIKE-10.7B-Instruct-v0.1	11	34.17	95.91
speakleash/Bielik-7B-Instruct-v0.1	7	31.26	91.23
speakleash/Bielik-7B-Instruct-v0.1	7	29.77	87.13
Remek/Llama-3-8B-Omnibus-1-PL-v01-INSTRUCT	8	26.63	100.00
microsoft/Phi-3-mini-4k-instruct	4	26.08	92.98
Voicelab/trurl-2-13b-academic	13	24.56	94.74
Qwen/Qwen1.5-7B-Chat	7	23.98	95.91
Qwen/Qwen1.5-7B	7	20.95	90.64
google/gemma-1.1-2b-it	2	16.47	100.00
Qwen/Qwen2-1.5B-Instruct	2	14.79	96.49
internlm/internlm2-chat-1_8b	2	12.13	87.72
01-ai/Yi-1.5-6B-Chat	6	4.89	94.15
openchat/openchat-3.6-8b-20240522	8	0.00	99.42
Voicelab/trurl-2-7b	7	nan	82.46	141.0 questions were parseable (min is 83%)
speakleash/Bielik-7B-v0.1	7	nan	81.29	139.0 questions were parseable (min is 83%)
microsoft/WizardLM-2-7B	7	nan	80.12	137.0 questions were parseable (min is 83%)
Qwen/Qwen2-0.5B-Instruct	0	nan	73.10	125.0 questions were parseable (min is 83%)
meta-llama/Llama-2-7b-chat-hf	7	nan	67.84	116.0 questions were parseable (min is 83%)
HuggingFaceH4/zephyr-7b-alpha	7	nan	57.89	99.0 questions were parseable (min is 83%)
HuggingFaceH4/zephyr-7b-beta	7	nan	51.46	88.0 questions were parseable (min is 83%)
lmsys/vicuna-13b-v1.5	13	nan	49.12	84.0 questions were parseable (min is 83%)
mistralai/Mistral-7B-v0.1	7	nan	38.01	65.0 questions were parseable (min is 83%)
Qwen/Qwen2-1.5B	2	nan	31.58	54.0 questions were parseable (min is 83%)
mistralai/Mixtral-8x22B-v0.1	141	nan	29.24	50.0 questions were parseable (min is 83%)
alpindale/Mistral-7B-v0.2-hf	7	nan	26.32	45.0 questions were parseable (min is 83%)
internlm/internlm2-7b	7	nan	25.15	43.0 questions were parseable (min is 83%)
TinyLlama/TinyLlama-1.1B-Chat-v1.0	1	nan	21.05	36.0 questions were parseable (min is 83%)
mistralai/Mixtral-8x22B-v0.1	141	nan	19.88	34.0 questions were parseable (min is 83%)
Qwen/Qwen2-0.5B	0	nan	10.53	18.0 questions were parseable (min is 83%)
mistralai/Mistral-7B-v0.3	7	nan	8.19	14.0 questions were parseable (min is 83%)
mistralai/Mixtral-8x7B-v0.1	47	nan	5.85	10.0 questions were parseable (min is 83%)
internlm/internlm2-base-7b	7	nan	3.51	6.0 questions were parseable (min is 83%)
internlm/internlm2-20b	20	nan	2.34	4.0 questions were parseable (min is 83%)
01-ai/Yi-1.5-6B	6	nan	0.58	1.0 questions were parseable (min is 83%)
meta-llama/Llama-2-7b-hf	7	nan	0.58	1.0 questions were parseable (min is 83%)
szymonrucinski/Curie-7B-v1	7	nan	0.58	1.0 questions were parseable (min is 83%)
01-ai/Yi-1.5-9B	9	nan	0.58	1.0 questions were parseable (min is 83%)
upstage/SOLAR-10.7B-v1.0	11	nan	0.58	1.0 questions were parseable (min is 83%)
Azurro/APT3-275M-Base	0	nan	0.00	0.0 questions were parseable (min is 83%)
Azurro/APT3-1B-Base	1	nan	0.00	0.0 questions were parseable (min is 83%)
OPI-PG/Qra-1b	1	nan	0.00	0.0 questions were parseable (min is 83%)
sdadas/polish-gpt2-xl	2	nan	0.00	0.0 questions were parseable (min is 83%)
internlm/internlm2-1_8b	2	nan	0.00	0.0 questions were parseable (min is 83%)
microsoft/phi-2	3	nan	0.00	0.0 questions were parseable (min is 83%)
ssmits/Falcon2-5.5B-Polish	6	nan	0.00	0.0 questions were parseable (min is 83%)
google/gemma-7b	7	nan	0.00	0.0 questions were parseable (min is 83%)
google/gemma-7b-it	7	nan	0.00	0.0 questions were parseable (min is 83%)
OPI-PG/Qra-7b	7	nan	0.00	0.0 questions were parseable (min is 83%)
RWKV/HF_v5-Eagle-7B	7	nan	0.00	0.0 questions were parseable (min is 83%)
RWKV/v5-Eagle-7B-HF	7	nan	0.00	0.0 questions were parseable (min is 83%)
meta-llama/Meta-Llama-3-8B	8	nan	0.00	0.0 questions were parseable (min is 83%)

Plot

Authors:

Automatic translation: Remigiusz Kinas
Translation proofreading and localization: Maria Filipkowska, Zuzanna Dabić
Preparing dataset: Kacper Milan
Running benchmark and leaderboard: Krzysztof Wróbel

Based on: EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models, Samuel J. Paech, 2023