ADVANCe Augmented dialogue tool based on verbal and non-verbal behaviour computing

  • HES-SO Fribourg
  • Prof. Jean Hennebert
  • Beat Wolf
  • Julien Esseiva
  • Oussama Zayene
  • Machine learning
  • Video and audio processing
Our skills
  • Applied Machine Learning, building pre-production software including deep learning algorithms
  • Technology transfer to industrial partners
  • Institute of Complex Systems – iCoSys
  • CM Profiling
Swiss Innovation Agency – innosuisse under grant 31933.1 IP-ICT
01.03.2019 – 01.09.2020
Architecture overview

ADVANCe is an ongoing Innosuisse project that uses the recent advances in machine learning and deep learning in the domain of video processing and especially human behaviour detection to build a tool to assist recruiters during face to face or video (recruiting) interviews. The goal of the project is to assist the recruiters by giving them cues on verbal and non-verbal indices of the interviewee to enhance the dialogue interaction quality.

The project includes the creation of a dataset of videos of real life interview settings. Those videos are recorded using an intel 3D camera. The videos are annotated using a custom developed video annotation tool.

Using this dataset a SaaS tool is developed to assist recruiters either using a recorded video of the interview, or with a real time analysis of a video 3D video stream. This is achieved by using state of the art machine learning models to extract the verbal and non-verbal cues of the video and analyse them to establish a correlation between visual and audio cues.

The final product, commercialized by CM-Profiling can be found at