Multimodal Dialogue Systems (Summer 2019)

Block course

Time & Location: TBA, indicatively September-October

Teacher: Dr Volha Petukhova

*** Announcements***

Kick-off: 10.05.2019 (Friday) at 15:00 in 1.12 (Seminarraum) C 7.3


Introduction slides: PDF

Suitable for: CoLi, CS and CuK

Registration system: CLOSED

Please note that the registration system does not send any confirmation e-mails. The number of slots is limited. Therefore, it is advised to register early.


We plan to hold a first planing meeting early in the semester. For the actual seminar (doodle decision on time and papers) we will have a talk for each participant of 30 minutes followed by 10 minutes discussion (discussions participation will be also graded) . After the talk, the presenter has to prepare a short about 10 pages report and hand it in for grading. 

Grading: 40% based on the talk, 40% based on the report, 20% based on discussions participation.

Term paper:

  • LaTeX template for term papers (zip)
  • 11-point checklist for term papers (pdf)


Situated interaction

Understanding and generation of multimodal human dialogue behaviour

Social signals/affective computing

Multimodal dialogue modelling

Multimodal dialogue systems & applications

  *Each talk will be based on a research paper


1.  Barsalou, Lawrence W. “Situated conceptualization: theory and application.” Perceptual and Emotional Embodiment: Foundations of Embodied Cognition. Psychology Press: East Sussex (2015) PDF

2. Schmorrow, Dylan, Kay M. Stanney, Glenn Wilson, and Peter Young. “Augmented cognition in human-system interaction.” Handbook of human factors and ergonomics 3 (2005) PDF


3. Kulms, Philipp, and Stefan Kopp. “The effect of embodiment and competence on trust and cooperation in human–agent interaction.” In International Conference on Intelligent Virtual Agents, pp. 75-84. Springer, Cham, 2016. PDF

4. Traum, David, Stacy C. Marsella, Jonathan Gratch, Jina Lee, and Arno Hartholt. “Multi-party, multi-issue, multi-strategy negotiation for multi-modal virtual agents.” In International Workshop on Intelligent Virtual Agents, pp. 117-130. Springer, Berlin, Heidelberg, 2008. PDF

Social signals and affective computing

5. Vinciarelli, Alessandro, Maja Pantic, Dirk Heylen, Catherine Pelachaud, Isabella Poggi, Francesca D’Errico, and Marc Schroeder. “Bridging the gap between social animal and unsocial machine: A survey of social signal processing.” IEEE Transactions on Affective Computing 3, no. 1 (2012): 69-87  PDF

6. Gilmartin, Emer, Benjamin R. Cowan, Carl Vogel, and Nick Campbell. “Chunks in Multiparty Conversation—Building Blocks for Extended Social Talk.” In Proceedings of IWSDS, 2017 PDF

 7. Ginzburg, Jonathan, Ellen Breithholtz, Robin Cooper, Julian Hough, and Ye Tian. “Understanding laughter.” In Proceedings of the 20th Amsterdam Colloquium. 2015 PDF

 Multimodal fusion, dialogue modelling and management

8. S. Dham, A. Sharma, and A. Dhall. Depression scale recognition from audio, visual and text analysis, 2017 PDF

9. Lalanne, Denis, Laurence Nigay, Peter Robinson, Jean Vanderdonckt, and Jean-François Ladry. “Fusion engines for multimodal input: a survey.” In Proceedings of the 2009 international conference on Multimodal interfaces, pp. 153-160. ACM, 2009 PDF

10. Chen, Zheqian, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. “Dialogue act recognition via crf-attentive structured network.” In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 225-234. ACM, 2018. PDF

Multimodal dialogue systems & applications

11. DeVault, David, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila et al. “SimSensei Kiosk: A virtual human interviewer for healthcare decision support.” In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp. 1061-1068. International Foundation for Autonomous Agents and Multiagent Systems, 2014 PDF

12. Schroder, Marc, Elisabetta Bevacqua, Roddy Cowie, Florian Eyben, Hatice Gunes, Dirk Heylen, Mark Ter Maat et al. “Building autonomous sensitive artificial listeners.” IEEE Transactions on Affective Computing 3, no. 2 (2012): 165-183. PDF

For any questions, please send an email to:

Use subject tag: [MDS_2019]