Seminar: Multimodal Dialogue Systems

Summer Semester 2017 

Time & Location: Mo 16:15-17:45, C7.2 Seminarraum 

Teacher: Dr Volha Petukhova


*** Announcements***

Change location: Seminarraum C 7.2 (ground floor), room 1.12

First meeting Monday 8.05.2017 

Paper selection CLOSED!


Course registration is CLOSED 

Kick-off: PDF

Introduction slides: PDF


Suitable for: CoLi, CS and CuK


Please note that the registration system does not send any confirmation e-mails. The number of slots is limited. Therefore, it is advised to register early.


We plan to hold a first planing meeting early in the semester followed by 2 introduction meetings. For the actual seminar we will have a talk for each participant of 45 minutes followed by discussion (discussions parrticipation will be also graded) . After the talk, the presenter has to prepare a short about 10 pages report and hand it in for grading. 

Grading: 40% based on the talk, 40% based on the report, 20% based on discussions participation.

Term paper:

  • LaTeX template for term papers (zip)
  • 11-point checklist for term papers (pdf)


Understanding and generation of multimodal human dialogue behaviour

Situated interaction

Adaptive Dialogue Management

End-to-end dialogue system design


 *Each talk will be based on a research paper

 Situated interaction

1.  Bohus, D., Horvitz, E. (2010) On the Challenges and Opportunities of Physically Situated Dialog, in AAAI Fall Symposium on Dialog with Robots, Arlington, VA  PDF

2. Streitz, N. A., Röcker, C., Prante, T., Stenzel, R., van Alphen, D. (2003). Situated Interaction with Ambient Information: Facilitating Awareness and Communication in Ubiquitous Work Environments. In: D. Harris, V. Duffy, M. Smith, C. Stephanidis (Eds.): Human-Centred Computing: Cognitive, Social, and Ergonomic Aspects, New Jersey, Lawrence Erlbaum Publishers, pp. 133 – 137. PDF


3. Carolin Straßmann, Astrid Rosenthal von der Pütten, Ramin Yaghoubzadeh, Raffa-el Kaminski, and Nicole Krämer (2016) The effect of an intelligent virtual agent’s nonverbal behavior with regard to dominance and cooperativity. In: Intelligent Virtual Agents: 16th International Conference, IVA 2016, Los Angeles, CA, USA, September 20--23 PDF

4. Gupta, Rahul, et al. (2016) Laughter Valence Prediction in Motivational Interviewing based on Lexical and Acoustic Cues. Interspeech 2016, pp. 505-509 PDF

5.  Jonathan Herzig, Guy Feigenblat, Michal Shmueli-Scheuer, David Konopnicki, Anat Rafaeli, Daniel Altman and David Spivak (2016) Classifying Emotions in Customer Support Dialogues in Social Media. In: Proceedings of 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue PDF

Common sense knowledge learning and modelling

6. Henry Lieberman, Hugo Liu, Push Singh, and Barbara Barry. (2005) Beating Common Sense into Interactive Applications, AI Magazine. PDF

7. Elahe Rahimtoroghi, Ernesto Hernandez and Marilyn A Walker (2016) Learning Fine-Grained Knowledge about Contingent Relations between Everyday Events. In: Proceedings of the SIGDIAL 2016 Conference, pages 350–359, Los Angeles, USA PDF

Dialogue Behavior Understanding and Management

8. Turunen, Markku, and Jaakko Hakulinen (2001) Agent-based adaptive interaction and dialogue management architecture for speech applications." International Conference on Text, Speech and Dialogue. Springer Berlin Heidelberg PDF

9. Kennington, Casey, and David Schlangen (2016) Supporting Spoken Assistant Systems with a Graphical User Interface that Signals Incremental Understanding and Prediction State." 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue PDF

10. Lee, Sungjin, and Amanda Stent (2016) Task Lineages: Dialog State Tracking for Flexible Interaction. 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. PDF

11. Georgiladakis, Spiros, et al.  (2016) Root Cause Analysis of Miscommunication Hotspots in Spoken Dialogue Systems." Interspeech 2016 PDF

Prototyping, applications and end-to-end systems

12. Pulman, S. G., et al. (2010) How was your day? Proceedings of the 2010 Workshop on Companionable Dialogue Systems. Association for Computational Linguistics PDF

13. Stoyanchev, Svetlana, Pierre Lison, and Srinivas Bangalore (2016) Rapid Prototyping of Form-driven Dialogue Systems Using an Open-source Framework. 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue  PDF

14.  Kohei Ono et al. (2016) Toward Lexical Acquisition during Dialogues through Implicit Confirmation for Closed-Domain Chatbots. In Proceedings of the 2nd Workshop on Chatbots and Conversational Agent Technologies PDF

15. Tiancheng Zhao et al. (2016) The DialPort Portal: Grouping Diverse Types of Spoken Dialog Systems. In Proceedings of the 2nd Workshop on Chatbots and Conversational Agent Technologies PDF


For any questions, please send an email to:


Use subject tag: [MDS]