Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance

Rabab Alkhalifa; Hsuvas Borkakoty; Romain Deveaud; Alaa El-Ebshihy; Luis Espinosa-Anke; Tobias Fink; Petra Galuščáková; Gabriela Gonzalez-Saez; Lorraine Goeuriot; David Iommi; Maria Liakata; Harish Tayyar Madabushi; Pablo Medina-Alias; Philippe Mulhem; Florina Piroi; Martin Popel; Arkaitz Zubiaga

doi:10.1007/978-3-031-71908-0_10

Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance

Rabab Alkhalifa
, Hsuvas Borkakoty
, Romain Deveaud
, Alaa El-Ebshihy
, Luis Espinosa-Anke
, Tobias Fink
, Petra Galuščáková
, Gabriela Gonzalez-Saez
, Lorraine Goeuriot
, David Iommi
, Maria Liakata
, Harish Tayyar Madabushi
, Pablo Medina-Alias
, Philippe Mulhem
, Florina Piroi^*
, Martin Popel
, Arkaitz Zubiaga

^*Corresponding author for this work

Computer Engineering Department (CE)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

9 Scopus citations

Abstract

We describe the second edition of the LongEval CLEF 2024 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

Original language	English
Title of host publication	Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings
Editors	Lorraine Goeuriot, Philippe Mulhem, Georges Quénot, Didier Schwab, Giorgio Maria Di Nunzio, Guglielmo Faggioli, Nicola Ferro, Laure Soulier, Petra Galuščáková, Alba García Seco de Herrera
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	208-230
Number of pages	23
ISBN (Print)	9783031719073
DOIs	https://doi.org/10.1007/978-3-031-71908-0_10
State	Published - 2024
Event	15th International Conference of the CLEF Association, CLEF 2024 - Grenoble, France Duration: 9 Sep 2024 → 12 Sep 2024

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14959 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	15th International Conference of the CLEF Association, CLEF 2024
Country/Territory	France
City	Grenoble
Period	9/09/24 → 12/09/24

Keywords

Evaluation
Information Retrieval
Temporal Generalisability
Temporal Persistence
Text Classification

Access to Document

10.1007/978-3-031-71908-0_10

Cite this

Alkhalifa, R., Borkakoty, H., Deveaud, R., El-Ebshihy, A., Espinosa-Anke, L., Fink, T., Galuščáková, P., Gonzalez-Saez, G., Goeuriot, L., Iommi, D., Liakata, M., Madabushi, H. T., Medina-Alias, P., Mulhem, P., Piroi, F., Popel, M., & Zubiaga, A. (2024). Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. In L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, G. M. Di Nunzio, G. Faggioli, N. Ferro, L. Soulier, P. Galuščáková, & A. García Seco de Herrera (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings (pp. 208-230). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14959 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-71908-0_10

Alkhalifa, Rabab ; Borkakoty, Hsuvas ; Deveaud, Romain et al. / Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings. editor / Lorraine Goeuriot ; Philippe Mulhem ; Georges Quénot ; Didier Schwab ; Giorgio Maria Di Nunzio ; Guglielmo Faggioli ; Nicola Ferro ; Laure Soulier ; Petra Galuščáková ; Alba García Seco de Herrera. Springer Science and Business Media Deutschland GmbH, 2024. pp. 208-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{9c8599fccbf3456ba70e3323f4ddfc7a,

title = "Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance",

abstract = "We describe the second edition of the LongEval CLEF 2024 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.",

keywords = "Evaluation, Information Retrieval, Temporal Generalisability, Temporal Persistence, Text Classification",

author = "Rabab Alkhalifa and Hsuvas Borkakoty and Romain Deveaud and Alaa El-Ebshihy and Luis Espinosa-Anke and Tobias Fink and Petra Galu{\v s}{\v c}{\'a}kov{\'a} and Gabriela Gonzalez-Saez and Lorraine Goeuriot and David Iommi and Maria Liakata and Madabushi, \{Harish Tayyar\} and Pablo Medina-Alias and Philippe Mulhem and Florina Piroi and Martin Popel and Arkaitz Zubiaga",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 15th International Conference of the CLEF Association, CLEF 2024 ; Conference date: 09-09-2024 Through 12-09-2024",

year = "2024",

doi = "10.1007/978-3-031-71908-0\_10",

language = "English",

isbn = "9783031719073",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "208--230",

editor = "Lorraine Goeuriot and Philippe Mulhem and Georges Qu{\'e}not and Didier Schwab and \{Di Nunzio\}, \{Giorgio Maria\} and Guglielmo Faggioli and Nicola Ferro and Laure Soulier and Petra Galu{\v s}{\v c}{\'a}kov{\'a} and \{Garc{\'i}a Seco de Herrera\}, Alba",

booktitle = "Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings",

}

Alkhalifa, R, Borkakoty, H, Deveaud, R, El-Ebshihy, A, Espinosa-Anke, L, Fink, T, Galuščáková, P, Gonzalez-Saez, G, Goeuriot, L, Iommi, D, Liakata, M, Madabushi, HT, Medina-Alias, P, Mulhem, P, Piroi, F, Popel, M & Zubiaga, A 2024, Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. in L Goeuriot, P Mulhem, G Quénot, D Schwab, GM Di Nunzio, G Faggioli, N Ferro, L Soulier, P Galuščáková & A García Seco de Herrera (eds), Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14959 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 208-230, 15th International Conference of the CLEF Association, CLEF 2024, Grenoble, France, 9/09/24. https://doi.org/10.1007/978-3-031-71908-0_10

Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. / Alkhalifa, Rabab; Borkakoty, Hsuvas; Deveaud, Romain et al.
Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings. ed. / Lorraine Goeuriot; Philippe Mulhem; Georges Quénot; Didier Schwab; Giorgio Maria Di Nunzio; Guglielmo Faggioli; Nicola Ferro; Laure Soulier; Petra Galuščáková; Alba García Seco de Herrera. Springer Science and Business Media Deutschland GmbH, 2024. p. 208-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14959 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance

AU - Alkhalifa, Rabab

AU - Borkakoty, Hsuvas

AU - Deveaud, Romain

AU - El-Ebshihy, Alaa

AU - Espinosa-Anke, Luis

AU - Fink, Tobias

AU - Galuščáková, Petra

AU - Gonzalez-Saez, Gabriela

AU - Goeuriot, Lorraine

AU - Iommi, David

AU - Liakata, Maria

AU - Madabushi, Harish Tayyar

AU - Medina-Alias, Pablo

AU - Mulhem, Philippe

AU - Piroi, Florina

AU - Popel, Martin

AU - Zubiaga, Arkaitz

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024

Y1 - 2024

N2 - We describe the second edition of the LongEval CLEF 2024 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

AB - We describe the second edition of the LongEval CLEF 2024 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

KW - Evaluation

KW - Information Retrieval

KW - Temporal Generalisability

KW - Temporal Persistence

KW - Text Classification

UR - https://www.scopus.com/pages/publications/85201644331

U2 - 10.1007/978-3-031-71908-0_10

DO - 10.1007/978-3-031-71908-0_10

M3 - Conference contribution

AN - SCOPUS:85201644331

SN - 9783031719073

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 208

EP - 230

BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings

A2 - Goeuriot, Lorraine

A2 - Mulhem, Philippe

A2 - Quénot, Georges

A2 - Schwab, Didier

A2 - Di Nunzio, Giorgio Maria

A2 - Faggioli, Guglielmo

A2 - Ferro, Nicola

A2 - Soulier, Laure

A2 - Galuščáková, Petra

A2 - García Seco de Herrera, Alba

PB - Springer Science and Business Media Deutschland GmbH

T2 - 15th International Conference of the CLEF Association, CLEF 2024

Y2 - 9 September 2024 through 12 September 2024

ER -

Alkhalifa R, Borkakoty H, Deveaud R, El-Ebshihy A, Espinosa-Anke L, Fink T et al. Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. In Goeuriot L, Mulhem P, Quénot G, Schwab D, Di Nunzio GM, Faggioli G, Ferro N, Soulier L, Galuščáková P, García Seco de Herrera A, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction - 15th International Conference of the CLEF Association, CLEF 2024, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 208-230. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-71908-0_10