Multimedia Discourse Interaction
Official course page

The course is led by Lynda Hardman, and runs from 13th November 2019 until 31st January 2020.

NOTE: This web page is dynamic and will change throughout the course.

Table of contents

• Schedule
• Seed Publications
• Interesting things to present
• Assignments
• Useful Web sites
• Semantic Web resources
• Random useful links

Schedule

Wednesday 13th November 14:15-17:00

Who is the teacher? What is the course about?
What is your background? What do you want to do in the course? How does this course fit with your other courses?
When do we meet?
What do we need to keep track of? How will we keep track of it?
Who is in which group?
How can we communicate most efficiently? How do you get the most out of my time?
Do we want to use Slack for group communication.
Talk From Linked Data to Stories presented by Lynda

Homework for Wednesday 20th November.
Presentations and discussions on 5 papers
Provide 1 good point and 1 point for improvement on the research question, the evaluation method and the analysis for each of the 5 papers to be read.
Your answers are to help you think about how you will construct your own research question, decide on the evaluation method and the analysis in the next few weeks. You will need these for writing your final report for the course.

Friday 15th November 13:15-15:00

Canonical processes of semantically annotated media production Lynda Hardman, Željko Obrenović, Frank Nack, Brigitte Kerhervé, Kurt Piersol. Multimedia Systems. December 2008. 14(6) pp 327–340 Web site

Talks (to be) presented by Lynda
Ana Carina Palumbo User information needs for environmental opinion-forming and decision-making in linked-enriched video
Helen Boots-Blankers Identifying Aspects of Informative Videos that Indicate Objectivity: Is Seeing Believing? Slide 15, video fragment 1 , video fragment 2. MSc thesis Is Seeing Believing? Identifying Aspects of Informative Videos that Indicate Objectivity.

Wednesday 20th November 14:15-17:00

Decide in class on communal papers to read for Wednesday 27th November.
Before Friday November 29th come up with suggestions for 5 papers to read for Wednesday 4th December (and beyond). Put them in order of preference since you need only read 2 or 3 before 4th December.

Friday 22nd November 13:15-15:00

Potential project topics:
• Visualizations for Linked Brain Data
• Marc Geerards composition of video clips around a specific topic. Video clips together with topic & filmic annotations are available.
• Diana Guzijan, perception of cultural heritage
• National Geographic Making the Case for Nature

Wednesday 27th November 14:15-17:00

Presentations and discussions on 5 papers. Discuss what should be in initial project proposals

Friday 29th November 13:15-15:00

Work on initial project proposals. Feedback to be supplied by Lynda asap.

Wednesday 4th December 14:15-17:00

Second draft of individual project proposals submitted.
Presentations of initial project proposals (5 minutes presentation and 3 minutes discussion per group).
Use of time to be planned. Schedule presentations of papers and how they relate to project.

Friday 6th December 13:15-15:00

To be planned

Wednesday 11th December 15:15-17:00

First draft of structured set of writing tips. [Link to be proposed by group]
To be planned

Friday 13th December 13:15-16:00

To be planned

Wednesday 18th December 14:15-17:00

To be planned

Friday 20th December 13:15-15:00

Skip

Wednesday 8th January 14:15-17:00

Group project pilot studies.

Friday 10th January 13:15-15:00

Group project pilot studies.

Wednesday 15th January 14:15-17:00

To be planned

Friday 17th January 13:15-15:00

To be planned

Wednesday 22nd January 14:15-17:00

To be planned

Friday 24th January 13:15-15:00

Lynda away.
Draft reports submitted for feedback (by 25th January at the latest). This is optional - but recommended!

Wednesday 29th January 14:15-17:00

Final presentations 15 minutes per group - 1 minute for set-up, 12 minutes for the talk, 2 minutes for questions.

Friday 31st January

No room booked. Final report to be handed in.

Seed publications for the seminar

Papers to read for week 2 of course

• Vox Populi paper:Automatic generation of matter-of-opinion video documentaries, talk, demo and thesis by Stefano Bocconi. Interview With America documentary.
• Artequakt: Generating Tailored Biographies with Automatically Annotated Fragments from the Web, S. Kim, H. Alani, W. Hall, P. Lewis, D. Millard, N. Shadbolt, and M. Weal. In Semantic Authoring, Annotation and Knowledge Markup (SAAKM) 2002 Workshop at the 15th European Conference on Artificial Intelligence (ECAI 2002), Lyon, France.
• Link-enriched video: user information needs for environmental opinion-forming and decision-making slides in pdf, technical report and bad quality video by Ana Carina Palumbo. Her talk starts at 1hr 3mins in the video.
• Generation of abstract geometric art based on exact aesthetics, gestalt theory and graphic design principles, Michiel Kauw-A-Tjoe, CWI Tech report
• Genre Driven Multimedia Document Production by Means of Incremental Transformation Marc Nanard, Jocelyne Nanard, Peter R. King, Ludovic Gaillard, ACM Symposium on Document Engineering 2007: 111-120

Papers to read for week 3 of course

• Where story and media meet: computer generation of narrative discourse, Rémi Ronfard, Nicolas Szilas, CMN 2014: 164-176
• Towards Constructive Text, Diagram, and Layout Generation for Information Presentation, John Bateman, Thomas Kamps, Jörg Kleinz and Klaus Reichenberger, Computational Linguistics 27(3) pp409-449
• Can Computers Create Humor? Greame Ritchie. AI Magazine 30(3): 71-81 (2009)
• The canonical processes of a dramatized approach to information presentation. Vincenzo Lombardo, Fabrizio Nunnari, Rossana Damiano, Antonio Pizzo, Cristina Gena. Multimedia Systems (2008) 14:385-393
• Enriching news for supporting users' information needs using schema-driven classification of entities and relations, Viola Pinzi, MSc thesis, UvA 2014

Papers not yet read

• Towards Ontology-Driven Discourse: From Semantic Graphs to Multimedia Presentations, Joost Geurts, Stefano Bocconi, Jacco van Ossenbruggen, Lynda Hardman. ISWC, 2nd International Semantic Web Conference 2003, 597-612.
• Context Aware Guidance for Multimedia Authoring: Harmonizing Domain and Discourse Knowledge by Kateryna Falkovych. Further articles on the topic all Kateryna's publications at CWI.
• Generation of video documentaries from discourse structures, Cesare Rocchi and Massimo Zancanaro, in proceedings of the 9th European Workshop on Natural Language Generation. 2003
• Papers by Marc Cavazza.

Interesting things to present

• Introduction to the LinkedTV project, Lynda and Lilia Pérez Romero
• TV screen of LinkedTV News demo and second screen application
• LinkedTV News video demo. Lilia's USI project report LinkedTV News: designing a second screen companion for web-enriched news broadcasts.
• Short video explanation of entities (What is Metaweb?)
• Linked Data - Connect Distributed Data across the Web, The Linking Open Data cloud diagram and its statistics.

• How the emergence of the semantic web changes our approach to information architecture by Silver Oliver, BBC.
• Linking hypervideos to Web content, Raphaël Troncy
• EventMedia Live: Exploring Events Connections in Real-Time to Enhance Content
• Experiencing Events Through User-Generated Media by André Fialho.
• Meaningful presentations.
• The next 5,000 days of the Web
• Modes of persuasion
• Rhetorical Structure Theory
• Investigation of methods for user adapted visualisation of information in a hypermedia generation system by Jana Werner has a figure of different (simple) discourse structures on p37.

New HTML5 video technologies for the future of TV by Silvia Pfeiffer.
Papers on finding and filtering links by Michiel Hildebrand: Thesaurus-based search in large heterogeneous collections, Searching in semantically rich linked data: a case study in cultural heritage Wordnet allows you to traverse the class hierarchy using hyponyms and hypernyms (after finding the definition of a word, click on the underlined S). More information on the linking and filtering:

• for the die hards Towards a pattern science for the Semantic Web
• for those who want something that just works, tools are available, in particular the relationship finder.

Assignments

Everything in Google doc, including links to contributions

Useful Web Sites

• europeana.
• CHIP allows exploratory access to (cultural heritage) information.

These show interesting interfaces:

• openAcademia shows several visualizations of search results (tagcloud, topic graph, relation browser.
• OntoWiki follows the wiki style with faceted navigation, map and calendar visualization.
• Tabulator (Tim Berners-Lee) is available as a Firefox extension. Very simple interface, which effectively shows (and allows navigation) of the raw graph structure (not included in the survey).

These are more oriented to search.

• Swoogle, semantic web search engine.
• Squiggle is a simple but effective interface for recommendation and search in music.
• SemSearch allows simple semi-structured input queries.
• Freebase, an open shared database of the world's knowledge (not strictly semantic web, but similar goals).

Vox Populi, a system for generating video sequences based on argument structures that uses explicit representation of argumentation structure.

Semantic Web Resources

How Will We Interact with the Web of Data? by Tom Heath.
Semantic Web Challenge, in particular the 2008 winner paggr.

Wordnet is an existing English language thesaurus from Princeton that has been converted to RDF.
Dublin Core (DC), VRA Core are vocabularies for describing resources. VRA Core is specialised for works of visual culture as well as the images that document them.
SKOS, Simple Knowledge Organisation System, is a family of formal languages, built on RDF and RDFS, designed for the representation of structured controlled vocabularies, such as thesauri or classification schemes.
FOAF (Friend of a Friend) a means of creating links among people on the Web, also the FOAF project, in particular the explorer.

Random Useful Links

These are some links that may not be useful now for the course, but are useful resources about relevant topics.

• Seminal article on the notion of mechanised support for access to information. Quoted by both the hypertext and digital library communities.
  As We May Think by Vannevar Bush.
• Weaving the Web, the story of the Web from Tim Berners-Lee himself.
• "Understanding Comics" by Scott McCloud for a discussion of the whys and wherefors of text and images and mixes of the two.
• Inaugural speech Smart Style for Conveying Information at TU/e)

Web Design

• Cool examples of CSS.
• "Don't Make Me Think : A Common Sense Approach to Web Usability" by Steve Krug for a simple but fundamental approach to information design work.
• Usability for the Web: Designing Web Sites that Work, The Morgan Kaufmann Series in Interactive Technologies.
• Exhibit allows you to combine different interfaces into a single one. For example, a time and place visualization of American Presidents using both a map and a timeline.

Semantic Web

• "The Semantic Web: an Introduction" by Sean B. Palmer
• Handy Semantic Web reference cheat sheet. Click on the pdf logo for an A4 version.
• Article explaining the (many) connections and similarities of hypermedia research and the Semantic Web.
• Article about Semantic Web applications
  "Usability and the Semantic Web" by Anthony Jameson. Slides of talk
• Semantic Web application examples RDFa bookmarklets
• Discussion of whether entities should be able to be modelled at the class and instance level simultaneously.
  Representing Classes As Property Values on the Semantic Web

Facet Browsing

• Flamenco paper Faceted Metadata for Image Search and Browsing and demo.
• Longwell and demo
• Facet browsing at Ebay
• E-culture's /facet