Last modifications of this page were done on 15th of March.
Classification across the eight emotional classes provided: anger, happiness, sadness, fear, surprise, contempt, disgust, and neutral state. The test set for the challenge has a balanced distribution across the emotional categories.
Prediction of the emotional attributes for arousal (calm to active), valence (negative to positive), and dominance (weak to strong). The emotional attributes are annotated with continuous values between 1 and 7 for each dimension.
Release of training data January 20th, 2024
Submission of the results opens February 7th, 2024
Submission of the results closes March 1st, 2024
Notification of the final results March 1st, 2024
Deadline to submit a paper for the Challenge,
describing the proposed methods March 22nd, 2024
Notification of acceptance April 15th, 2024
Speaker Odyssey 18 - 21 June, 2024
Special Session for the Challenge June 20th, 2024
Carlos Busso University of Texas at Dallas
Berrak Sisman University of Texas at Dallas
Ali Salman University of Texas at Dallas
Lucas Goncalves University of Texas at Dallas
Abinay Reddy University of Texas at Dallas
Laureano Moro Velazquez Johns Hopkins University
Thomas Thebaud Johns Hopkins University
Leibny Paola Garcia Johns Hopkins University
Najim Dehak Johns Hopkins University
There is no registration fee.
To register your team and obtain the full dataset, complete and send the following academic license to msp-lab@utdallas.edu.
This includes your signature at the end of the third page
Only one group per laboratory will be allowed. Participant groups enrolled several times using different names will be disqualified.
Participants can submit their papers to our special session.
You can submit your results there.
On the two tabs, you can see a leaderboard that is automatically updated after submission.
After you submit, you get an email back with the results.
Each team can submit only three times for each task.
You can see under the csv files that we expect to receive:
The letters must be {A, C, D, F, H, N, S, U}.
Any other letter will create an error.
A=Anger, C= Contempt, D= Disgust, F= Fear, H= Happiness, N= Neutral, S= Sadness, U= Surprise.
We decided to use the Macro-F1 score for task 1.
The values are from 1 to 7.
Values outside this range will create an error.
Arousal (1 calm, 7 active), Valence (1 negative, 7 positive), and Dominance (1 weak vs. 7 strong).
We will use the concordance correlation coefficient (CCC) for task 2. The ranking will be based on the average CCC across the three emotional attributes
The training set has 68,119 speaking turns. The development set has 19,815 speaking segments from 454 speakers. The test set comprises 2,347 unique segments from 187 speakers, for which the labels have not been made publicly available. The segments for the test set have been curated to maintain a balanced representation based on primary categorical emotions. The protocol used to collect this corpus is described in this paper:
Award Certification will be given together with $1.000 for each task for the top-performing team.
To qualify for the award, a paper needs to be submitted and accepted to the conference.
The Emotion Recognition Challenge is a special session at Odyssey 2024. Please attend Odyssey 2024 and come to our poster to listen to the challenge summary.
Participating teams whose paper got accepted will also present their work there, and papers will be published in the workshop’s proceedings.
As our team members are from different universities, we understand everyone needs a license. Do we need to provide a license from each university, or is one from the registrant sufficient?
Yes, every institution should have a license.
Should the license be signed?
Yes, at the end of the third page.
The license and under clause 6 require researchers to send unpublished papers to UTD. Is this really needed?
No. The first 3 pages are a standard data transfer agreement signed by several US institutions to facilitate the sharing of data (all the legal terms are already agreed). We cannot change any of these three pages. Any amendments to these first three pages are made in the attachments. If you check attachment 2, you will read:
Pursuant to Section 6, Page 1, Recipient is not required to send publications or public disclosures to Provider for comment. Recipient can make publicly available the results of the Project without prior review or comment by Provider."
Therefore, you do not need to send papers to UTD.
Would it be allowed to use public models, such as wav2vec and emotion2vec, for generating audio representations? Additionally, if transcripts are not provided, is it allowed to use ASR models to obtain the transcripts?
General audio representations yes.
General models trained for emotions no.
Is it allowed to use a pre-trained Speech-self-supervised model for fine-tuning/feature extraction within the MSP-Podcast corpus?
Yes
Can we use pre-trained text generative models such as ChatGPT?
Yes
Would it be allowed to use additional training data in addition to the provided training set?
No, for fairness.
Is it allowed to use additional datasets for training (publicly available and described in the submitted solution) except MSP-Podcast corpus?
No, for fairness.
Would it be allowed to annotate the provided data with additional labels by ourselves, such as gender?
No. (BTW, gender is included in the data so you can use it).
Are there any "No agreement" emotion class in the test3 ?
No, the test has the 8 emotions and it is balanced across classes.
Is the test a part of the MSP-Podcast corpus? (If so, and if the usage of the Speech-self-supervised model is allowed, we must not use those that saw MSP-Podcast corpus during training).
The test set for the challenge is the test3 set in the MSP-Podcast that first appears in version 1.11. Labels for this set have never been released.
Is the data you as organizers provide will be different to the original MSP-PODCAST corpus?
The data does not have test 1 and test 2, samples without speaker id have been removed to make the task speaker independent.
Is data augmentation allowed?
Yes.
The use of augmentation datasets such as RIRS or MUSAN is also allowed.
Can an Automatic Speech Recognition system be used to retrieve and use linguistic information from the audio?
Yes, ASR and text based approaches can be used.
Is there a platform to submit the model? How will be the evaluation proceed?
You only need to submit the results there, using the formating described under the "submission" section.
On the two tabs, you can see a leaderboard that is automatically updated after submission.
After you submit, you get an email back with the results.
What are the metrics that will be used?
We decided to use the Macro-F1 score for task 1, and the concordance correlation coefficient (CCC) for task 2.
What is the file format expected for the submission?
We are expecting .csv files with the labels or values for each file of the test set.
Sample submission files are available for each task under the "submission" section.
Can you confirm that we have to submit our result between the 20th of February and 1st March, and not before the 20th ?
You can start submitting very soon (you will receive an email from us).
The last day that you can submit is March 1 AoE.
Is there a maximum number of submission and if yes, how many ?
Yes 3
Will we have access to the teams ranking and score during the submission period ?
You can see the leaderboard. The table should update very soon after you submit your answer.
https://lab-msp.com/MSP-Podcast_Competition/leaderboard.php
Is there an official baseline?
Yes, we ran a simple baseline, detailed in this document.
Do you have more questions? Please get in touch with us at this address: msp-lab@utdallas.edu