TSU Faculty of Radiophysics students won Neurohackathon with a program for determining emotions by facial expressions. The program analyzes the results of monitoring, scientific studies involving people, psychotherapy sessions, and other situations.
- The neural network analyzes video and, relying on large amounts of data that were analyzed in the course of training, says that a person experiences such an emotion, - said Andrei Kostornoy, a student at the Faculty of Radiophysics, a team member. - At the jury, we presented an assistant program for the psychotherapist. It records a video session and tracks changes in behavior. When the doctor gives a certain stimulus (a question, anecdote, picture, provocation, sounds, or video), the patient changes their state, reacts to a change in facial expressions. At this point, the program detects the change and cuts a fragment of the video, which is delivered in real time to the doctor. The specialist has the opportunity to simplify additional analysis of the situation.
As the radiophysicist explained, this software and hardware complex can be used in medicine (psychology, telemedicine), scientific research (monitoring the state of the subjects during an experiment), economics, social work, and entertainment - wherever monitoring of various kinds is required.
- This idea can be applied, for example, as a mechanism of adaptive gaming, when the plot of the game adapts to the reaction of the player who plays it, - added Andrei Kostornoy. - To solve different tasks, in addition to facial expressions, it is necessary to add analysis of other data - tracking the duration and intensity of emotions, voice analysis, and electroencephalography analysis.
Students are planning to finalize the project by taking part in the accelerator AI & BIGDATA IN MEDICINE.
Third place in the hackathon was taken by the Faculty of Radiophysics students Nikita Panin and Rostislav Gursky, as part of another team. For the program to recognize emotions, they suggested ideas for collecting preliminary data, for example, during mandatory medical examinations in schools, monitoring the level of emotional state at work, in plastic surgery, and other areas. This will help to increase the database and, over time, reduce the error of the result.