Emotional Conversation Generation Challenge

Call for Participation

In recent years, there has been a rising tendency in AI research to enhance Human-Computer Interaction by humanizing machines. However, to create a robot capable of acting and talking with a user at the human level requires the robot to understand human cognitive behaviors, while one of the most important human behaviors is expressing and understanding emotions and affects. As a vital part of human intelligence, emotional intelligence is defined as the ability to perceive, integrate, understand, and regulate emotions. Though a variety of models have been proposed for conversation generation from large-scale social data, it is still quite challenging (and yet to be addressed) to generate emotional responses.

In this challenge, participants are expected to generate Chinese responses that are not only appropriate in content but also adequate in emotion, which is quite important for building an empathic chatting machine. For instance, if user says “My cat died yesterday”, the most appropriate response may be “It’s so sad, so sorry to hear that” to express sadness, but also could be “Bad things always happen, I hope you will be happy soon”to express comfort.

Task Definition

This task is defined as follows: Given a Chinese post X = (x1 , x2 , · · · , xn ), and a user-specified emotion category of the response to be generated, the goal is to generate a response Y = (y1,y2,··· ,ym) that is coherent with the emotion category. The emotion categories are {Anger, Disgust, Happiness, Like, Sadness, Other}, the same as defined in the NLPCC Emotion Classification Challenge.

Each team can submit at most TWO runs.

Dataset Description

The dataset is constructed from Weibo posts and replies/comments. More than 1 million Weibo post-response pairs will be provided to participants for training their models. To ensure fair comparison, NO additional training data will be allowed for conversation generation, but participants can use other data to train supplementary classifiers for emotion classification, for instance. Such details should be reported in the submission of results.

The test dataset consists of about 5000 posts while 100~200 of the posts will be manually assessed, and for each post, at most 3 emotion classes will be manually specified to indicate the emotion class of a generated response. Participating systems should generate a response for each emotion class. Note that participants should generate responses for all posts with appropriate emotion classes. Which part of the posts will be manually checked is unknown to participants for fair comparison.

The dataset will include labels of each post and response. These labels are for reference only, and they are obtained by a simple classifier that is based on a bidirectional LSTM model. The classifier was trained on the data from the NLPCC Emotion Classification Challenge. The accuracy of our classifier for six-way classification is about 64%. In other words, the emotion label of these data is NOISY. Participants are encouraged to implement the emotion classifier by themselves and with their own data, but all details must be reported and all resources should be accessible to the community to let other researchers reproduce their results. Note that no additional data is permitted to train the generation model.

The correspondence between the label and the emotion class can be seen as follows:

0:Other

1: Like

2: Sadness

3: Disgust

4: Anger

5: Happiness


The training dataset looks like: [[[post,post_label], [response,response_label]],[[post,post_label],[response,response_label]],…]. There are about 1,110,000 pairs in the training data.

*** Submission Format ***

Each participant should name their file as this: TEAMID_RUNID_EGG.txt.

TEAMID: the identifier of your team, just alphabetical letters.

RUNID: a digit number as 1,2, at most two runs can be accepted.

The file should use an array of JSON objects to format the data:

"[{post:"post1_runs_here",emo:"emotion label of response1",res:"response1 runs here"},{post:"post2_runs_here",emo:"emotion label of response2",res:"response2 runs here"} ]" .

NOTE0: emo denotes the emotion class of the response to be generated; res denotes the response.

NOTE1: you must use JSON API to generate the data, with UTF-8 encodings.

NOTE2: the emotion class must be coherent with what we defined previously.

NOTE3: please send your results to task4@nlpcc2017.info before June 7, 23:59 Beijing Time.

How to Participate

Please fill out the registration form and send it to the coordinator Fang Liu(刘芳) by email (contact@nlpcc2017.info) before April 30, 2017. If you have any question about the shared tasks, please do not hesitate to contact us by email.

For more details, refer to the official page of the NLPCC Challenge 2017.

Important dates

2017/3/6:announcement of shared tasks and call for participation;

2017/3/31:release of detailed task guidelines & sample data release;

2017/4/30:registration deadline;

2017/5/30:test data release;

2017/6/5:participants’ results submission deadline;

2017/6/15:evaluation results release and call for system reports and conference papers;

2017/7/15:conference paper submission deadline (only for shared tasks);

2017/8/20:conference paper accept/reject notification;

2017/9/5:camera-ready paper submission deadline;

2017/11/11~12:NLPCC 2017 main conference.

Download Data

The training data can be download here.

The test data is available now: download here.

Further, we provide the same data with the corresponding emotion class: download here. The emotion class is decided by our emotion classifier, just for your reference.




Note that the use of the data comforms to Copy Rights as claimed below.

Copy Rights

The dataset refers to the training and test post-response pairs we provide to participants.

Copy rights of the dataset are all reserved and belonging to the task organizers. Participants can only use these data for the challenge purpose. Commercial use of the datasets are prohibited.

For other academic use, please cite our work:

Hao Zhou, Minlie Huang, Xiaoyan Zhu, Bing Liu. Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory. arXiv:1704.01074.

Contact

A/Prof. Minlie Huang

Dept. of Computer Science, Tsinghua University

aihuang~AT~tsinghua~DOT~edu~DOT~cn