[PhD defence] 21/10/2024 - Timothée Dhaussy: "Proactive multimodal human-robot interaction in a hospital setting" (UPR LIA)

Mr Timothée DHAUSSY will publicly defend his thesis entitled: "L'interaction humain-robot multimodal proactive dans un cadre hospitalier", on Monday 21 October 2024.

Laboratory

UPR 4128 LIA - Avignon Computing Laboratory

Composition of the jury

MR FABRICE LEFEVRE	Avignon University	Thesis supervisor
Mr Julien PINQUIER	Paul Sabatier University	Rapporteur
Ms Aurélie CLODIC	LAAS-CNRS	Rapporteur
Bassam JABAIAN	Avignon University	Thesis co-supervisor
Ms Laurence DEVILLERS	Paris-Sorbonne University 4	Examiner
Mr Olivier ALATA	Jean Monnet University	Examiner

Summary

Human-Robot Interaction (HRI) is an interdisciplinary field of research in robotics and the social sciences. It aims to understand, design and evaluate the use of robots by humans. It represents an important issue for social robotics in the 21ᵉ century. In recent times, interest in companion robots capable of helping individuals in their daily lives and interacting with them has grown considerably. These robots, considered as social entities, have demonstrated their usefulness in the fields of healthcare and the psychological well-being of the elderly. Proactivity, or the ability to act proactively and autonomously, is an intrinsically human characteristic that enables us to actively influence our environment and circumstances, rather than passively reacting to them. In human-robot interactions, this proactivity is crucial because it enables robots to react more naturally and in a way that is adapted to users' needs. To anticipate user needs and take initiatives, the robot must understand its environment using its multimodal perceptions.

In this thesis, we focus on the creation of a proactive multimodal system for the social robot Pepper, intended for a hospital waiting room. To this end, we have developed a cognitive architecture for human-robot interaction, based on a continuous loop of perception, representation and decision. The perception flow is divided into two stages: firstly, the retrieval of data from the robot's sensors, and then their enrichment using refinement modules. A speaker diarisation refinement module, based on Bayesian modelling of the fusion of audio and visual perceptions by spatial coincidence, has been integrated. To enable proactive action, we designed a model analysing users' availability for interaction in a waiting room. The refined perceptions are then ordered and aligned to create a constantly updated representation of the environment. This image of the environment is then transmitted to the decision-making layer. There, an action planning module analyses the environmental data and develops action strategies by informing the action modules asynchronously. This ability to operate asynchronously means that the action planner can continue to look out for proactive opportunities provided by the scene, despite the operation of one of the action sub-modules, such as the speech module, which is responsible for holding a conversation with a user during an interaction. The whole system is implemented on ROS, enabling it to be adapted to various robotic supports.

This thesis presents the mechanisms required to create a proactive multimodal interaction system between a human and a robot. This system includes all the perception and action modules, as well as a global cognitive architecture for perception management. The whole system was tested in a controlled laboratory environment, as well as in real conditions at Broca hospital.

Keywords : interaction,proactive,hri,social robot