“Gleimhaus” Museum – Speech Recognition based exhibition installation

Posted on

This project was part of my studies during the 5th and 6th semester at the Harz University of Applied Sciences. It was a cooperation between the University and the design agency “IdeenGut” for the literature museum “Gleimhaus” in Halberstadt, Germany.
This project was also supported by the state of Saxony-Anhalt as part of their program “Sachsen-Anhalt DIGITAL“.

UPDATE!

The exhibition has been opened at the “Gleimhaus” museum since April 2, 2019, as this was a consecutive project and got finished by the next group.

An article about the finished installation can be found here: Link

We worked in a group of 7 (6 students and our professor) to develop a speech recognition based system that allows visitors to interact with portraits of known minds of the Age of Enlightment, e. g. Schiller. In memoriam of Gleim the visitor shall be able to greet a portrait and start a “conversation” in wich he can ask questions about multiple topics and get a feeling of personal interaction.

Our goals:

  • The visitor can navigate through the menu via spoken commands
  • The visitor has multiple choices for a command
  • Visitors that don’t want to use the speech recognition can trigger the dialogs via a web-interface on a given tablet
  • The adressed portrait gets illuminated for visual feedback via serial communication
  • Audio comments are played for each portrait and topic in sourround sound

Worked on/Done so far

Programming:

  • Speech recognition(Swift):
    • Developed a modular structure for building menus
    • Integrated Apples NSSpeechRecognizer API
    • Implemented option of multiple choices for commands
  • Serial communication(Swift & Arduino):
    • Implemented attachable serial commands to the menu structure
    • Implemented function for sending a command to a connected Arduino that gets called from the speech recognizer
    • Wrote Arduino-sketch to control the lights with the given commands
  • Audio(Swift):
    • Implemented attachable audio files to the menu structure
    • Implemented function for playing and stop playing audio files via voice commands
  • Backend(Java):
    • Developeded a GUI-backend-application for building our menus, which can:
      • manage nodes and their attributes
      • add an “unlimited” amount of new nodes
      • remove existing nodes
      • save the menu-structure as xml-file
      • load menus from existing xml-files
  • Web Application(Javascript, AJAX, PHP):
    • Integrated functions for controlling the system via LAN into the website

Details

Github repository: Click here!

My tasks

 Lead Programming

Technology used

 XCode, Arduino Uno, Github, Trello

Context

 “Annual project” – 5th and 6th semester

Project leader

 Prof. Daniel Ackermann

Partner

 Design Agency IdeenGut, Gleimhaus museum

Persons involved

 Johanna Haupt, Mona Holtmann, Madita Lange, Ivo Max Müllner,Annabel Schewe, Niklas Trenkel