eSensibility - Emotionserkennung im Auto von Morgen
Interaction. Emotion. Empathy.

Driver monitoring in the car – a new quality of interaction

"Imagine a car that says 'good morning' when you get in the car in the morning. You sit back, make yourself comfortable and say 'let's go' – and off you go. Your driver monitoring system does the rest. It tells you when it's time to take a break because your eyes are tired and your facial muscles are relaxed or you are staring blankly, lost in thought."

Timeofflightcamera Produkt

Autonomous driving

For a safe (semi-)autonomous journey, the reactivity of the driver is of the utmost importance, even when not actively driving the vehicle. It is thus necessary to identify the person and their ability to take over driving the vehicle. Assistance systems, such as that from KOSTAL, which process data from the vehicle interior and the surroundings in a complex system, will form the heart of autonomous driving in future. The KOSTAL driver monitoring camera focuses on the level of attention or distraction, which is captured by the KOSTAL camera technology using sensors and can be suitably influenced. Understimulation or fatigue can also be detected reliably by our camera with high uptime using appropriate analysis. 



The vehicle system can call up the appropriate escalation strategy by identifying driver distraction and/or understimulation (drowsiness detection & distraction warning) in a timely manner. If the driver displays signs of fatigue, for example, which are detected by the driver monitoring camera by analysing eyelid position, the vehicle  recommends service stations within the navigation, or overnight accommodation in the current surrounding area.
Augmented Reality HUD
In addition to assistance and safety functions, additional comfort functions benefit from the new sensors. In augmented reality applications, the real-life environment is, for example, overlapped with virtual contents via a heads-up display. The vehicle disk is the ideal display surface for digitally enriched augmented reality applications – from overlays of particular events in the area, through to virtual banner ads that can be linked with map services. Being able to continuously identify eye position is essential for perfectly bringing virtual reality contents into the driver's field of vision in terms of perspective. This is the only way that an image displayed in the heads-up display can be harmonised with the real world. The eye position necessary for this in x, y and z coordinates is determined by the absolute measurement driver monitoring camera. 
eSensibility Personalisierung


Driver identification using the driver monitoring camera enables comfort functions like the personalisation of driver settings. When you get in, the vehicle automatically configures itself by adjusting the seat position, interior lighting and playlist to the driver and their preferences. Tracking the movements of the mouth region facilitates binary voice recognition. As such, the camera can determine whether the driver is the active voice operator and relay this data to the vehicle system.

We are designing the 'new mobility HMI' of the future with KOSTAL's expertise in integration and technology. The megatrends 'autonomous driving' and 'shared mobility' guide us in this process.

Herr Dr. Naumann
Dr. Tino Naumann
Executive Vice President Business Field Mechatronic Modules, Switch Panels & Switches

What does the camera see?

  • High-resolution 2D and 3D camera systems recognise and track up to 49 significant facial characteristics. 
  • Supported by KOSTAL algorithms, the camera still generates reliable information even if the view is partially obscured, e.g. by a  mask or sunglasses. 
  • Even under difficult dynamic lighting conditions such as shadows or solar irradiation in the vehicle interior, the monitoring camera generates uninfluenced image data for information processing thanks to the 3D time-of-flight technology used.
  • The cameras can determine fatigue and/or distraction through the continual analysis of the eyelid position.
  • The camera processes the head position and the alignment of the driver and ascertains the driver's level of attention, determines the positioning of the heads-up display as well as the smart airbags.
  • It recognises the line of sight based on the pupil position and can specify the perfect position for augmented reality contents. 
  • It determines the driver's vocal activity based on mouth movements and can support the vehicle speech recognition system in differentiating voice commands from ambient noise.
  • The camera recognises the driver using driver identification and calls up personalisation functions such as seat adjustment and playlists. 
  • By observing facial characteristics, the system analyses the emotions and well-being of the driver and initiates comfort functions. 

Observing attentively, reacting intelligently


Driver monitoring with deep learning

Deep learning is a future-oriented technology in the area of artificial intelligence. Here, the neural structure of the human brain is taken in parts as an analogy to map large artificial neural networks. Deep learning goes hand in hand with the most successful scientific achievements in the area of image and voice recognition in recent years. Our engineers are already using self-developed deep-learning algorithms in the KOSTAL driver monitoring camera to identify and classify the driver's line of vision and emotions.


Interaction. Emotion. Empathy. 

By using deep-learning algorithms, the driver monitoring camera turns the car into an intelligent companion. Owing to attentive observation, the vehicle is able to trigger warnings and provide maintenance, services or assistance. The vehicle system can provide a playlist, introduce a relaxing scent in the air-conditioning system or switch the lighting scheme from sporty to relaxing based on analysing the driver's mood and depending on the personality profile.

The moment a special driver experience occurs, the camera takes a selfie of the driver and shares it with their preferred social network. There are almost no limits to the information processing opportunities.
Kamera liest Emotionen

Integration expertise

Integration is customer satisfaction. Together with our customers, we are working on solutions to optimally accommodate our camera technology in the vehicle, where it is indiscernible to your end customers.