Multimodal Soft Sensor to Assess Food Doneness.

CREATE Lab, EPFL

October-December 2023

Humans perceive doneness based on numerous variables, from smells to colors and touch. This project asks how a robot can combine several of those signals to cook the perfect meal.

Approach

Combine several sensing modalities into a single robot fingertip form factor.
Mount the sensor on a robotic arm to use arm motion itself to enhance the modalities and differentiate textures.

Exploration of modalities

Stiffness, permittivity, conductivity, and temperature.
Vision and sound features including Laplacian, FFT, and PSD.

Prototype setup used to explore multiple sensing modalities on toast

Design and fabrication

Combination of the most relevant modalities in a compact package.
Soft pressure sensor cast in silicone.

Front view of the fabricated multimodal soft sensor

Arm-sensor combination

Sensor package mounted on a UR3 robotic arm.
Combination of arm movement and sensor modalities for greater distinction.

Multimodal sensor mounted on a UR3 interacting with toast

Results

Tested on toasts with different cooking times.
Clear distinction between cooking times with differences as small as 30 seconds.

Sensor used for Rapid Action Evaluation and Optimization. Paper presented at RoboSoft 2024 - DOI: 10.1109/RoboSoft60065.2024.10521945