My name is Amaya Becvar Weddle and I am the UX Research Manager at Immersion Corporation. I’m going to be speaking to you today about a case study we did investigating the emotional impacts of haptic-enhanced media. In the process, I’d like to share some of our discoveries about measuring emotion in user experiences, and my hope is that you will walk away with some insights and new perspectives on emotion in UX studies.I’ll plan to stop periodically throughout the talk and take questions that you may have – and I also have an interactive exercise planned for the end of the session.
How many of you know what haptics is? “Haptics” is the study of touch. In technology terms, it refers to tactile feedback technology which takes advantage of the sense of touch by applying forces, vibrations, or motions to the user.Haptic effects (also known as touch or tactile feedback) are produced by actuators, such as motors, which are built into devices to create vibrations. These actuators are combined with Immersion software to create haptic sensations, like the feel of a button “click” when you press a virtual button. Haptics provide a sense of realism and improve the user experience.
Rolling Ball Demo
The algorithm processes the audio signal from multimedia content playing on a mobile device and converts low frequency audio into haptic feedback in real-time. The resulting signal is then fed into a vibration actuator in a mobile handset, delivering haptic effects that can be felt in the hands of a user as they hold the device while viewing the screen. In this way, the generated tactile feedback is synchronized with the magnitude and frequency of low frequency audio signals. For example, processing the audio track of gunfire may trigger a series of sharp vibrations that synchronize with the loud cracks of the gun.
With the proliferation of mobile devices, how people consume media is changing dramatically. People are watching more and more television and video content on mobile devices. So the screens upon which people are commonly consuming this content are shrinking. Immersion was interested in whether we could add value to these experiences by adding another sensory modality to the video experience.
We realized in this context, that emotional response was a very important variable to investigate. So I’m going to tell you about how we studied this problem, our findings, and how they impacted our company strategy. I hope you’ll also be able to take away some ideas about how you might use some of these methods for studying emotional response in your own work.
Does realism influence emotional response?
Before we measure it, we need to define what it is we are talking about
One of the most common frameworks in the emotions field proposes that affective experiences are best characterized by two main dimensions: arousal and valence. The dimension of valence ranges from highly positive to highly negative, whereas the dimension of arousal ranges from calming or soothing to exciting or agitating.
As a measure of arousal level, we chose to use “phasic” skin conductance measurements. These are characterized with abrupt increases in the skin conductance having discrete “peaks,” a distinctive attack, and a decay signature. These peaks are generally referred to as Skin Conductance Responses (SCRs). Electrodermal activity was captured during the testing sessions.
As a measure of arousal level, we chose to use “phasic” skin conductance measurements. These are characterized with abrupt increases in the skin conductance having discrete “peaks,” a distinctive attack, and a decay signature. These peaks are generally referred to as Skin Conductance Responses (SCRs). Electrodermal activity was captured during the testing sessions.
As a measure of arousal level, we chose to use “phasic” skin conductance measurements. These are characterized with abrupt increases in the skin conductance having discrete “peaks,” a distinctive attack, and a decay signature. These peaks are generally referred to as Skin Conductance Responses (SCRs). Electrodermal activity was captured during the testing sessions.
Tried to make a natural environment where participants could relax as they might in a home or waiting areaThey were instructed to hold the phones in their hands and use the headphones
Participants were presented videos within a special app that played the video and timestamped the data for comparison to their biometric readings later.
Longitudinal designFirst session, we gave participants a training session where they experienced “Reverb” for the first time on all devices. They also were trained in how to use the equipment and the rating scales/surveys we had them fill out. They then worked independently through a series of 24 video segments of ~1 min. each, and did a series of ratings after each video. Phone order was randomized between participants.We did not use the tonic skin conductance level (SCL) as a variable in the analysis due to the experimental design. To avoid habituation to the video content as much as possible, we showed subjects the repeated video content with a week’s separation. Thus, we compared electrodermal measurements of the same subjects on two different days, and because the SCL can be influenced by long-term physiological states, hydration, skin dryness, and autonomic regulation. SCR are independent of SCL and therefore not affected by long-term influences that cannot be controlled for. Comparing the number of specific SCRs per subject presented a more reliable and replicable measure for comparing affective responses from the same subject on different days.
Longitudinal designFirst session, was 90 minutes onsite. We gave participants a training session where they experienced “Reverb” for the first time on all devices. They also were trained in how to use the equipment and the rating scales/surveys we had them fill out. They then worked independently through a series of 24 video segments of ~1 min. each, and did a series of ratings after each video. Phone order was randomized between participants.
Longitudinal designFirst session, was 90 minutes onsite. We gave participants a training session where they experienced “Reverb” for the first time on all devices. They also were trained in how to use the equipment and the rating scales/surveys we had them fill out. They then worked independently through a series of 24 video segments of ~1 min. each, and did a series of ratings after each video. Phone order was randomized between participants.
The level of involvement was measured by a modified version of the Wells R Scale [17], a research instrument used to assess audience involvement in advertising messages. The instrument asks participants to rate their level of immersion, or feeling of “being there” in the video, as well as their level of emotional involvement with the content. We asked participants to rate on a scale of 1-9 based on how much they agreed with the following statements: “I felt like I was there in the video, experiencing the situation;” and “I really got involved with the emotions provoked by the video.” Finally, we had participants give us a numerical report on the overall quality of experience (QoE) of each video, using these landmarks: 0 – bad, 50 – average, 100 – excellent. Participants were instructed to rate their overall experience while watching videos (i.e. not simply rating the quality of the haptic experience).
The waveforms from the EDA recordings were compared to recorded timestamps of when subjects were watching specific videos during each session. The number of SCRs observed during video viewing segments was calculated, using standard thresholds for counting a skin conductance response (SCR) as documented in the literature. We imported the raw data into MATLAB and used an open source toolbox to perform waveform extraction and quantify the number of specific SCRs. Next, a comparison was made between the average number of SCRs per subject recorded as they viewed haptic-enhanced videos versus those recorded while they watched control videos. We did not use the tonic skin conductance level (SCL) as a variable in the analysis due to the experimental design. To avoid habituation to the video content as much as possible, we showed subjects the repeated video content with a week’s separation. Thus, we compared electrodermal measurements of the same subjects on two different days, and because the SCL can be influenced by long-term physiological states, hydration, skin dryness, and autonomic regulation. SCR are independent of SCL and therefore not affected by long-term influences that cannot be controlled for. Comparing the number of specific SCRs per subject presented a more reliable and replicable measure for comparing affective responses from the same subject on different days.
Average skin conductance response rate for each condition. Error bars represent 95% confidence intervals. The skin conductance response rate was higher (M = 1.9, SD = 0.8) when participants were viewing haptic-enhanced videos as compared to those viewed without haptics (M=1.1, SD = 0.7); (UA=270, z=2.6, p<0.006). Student t-testThis indicates that participants were more physiologically aroused when watching video content with haptic enhancement. Many studies have reported that the arousal dimension of emotion is acritical factor contributing to the emotional enhancement effect on memory.It has also been shown as important in advertising persuasion. So showing that haptics can enhance the arousal dimension is interesting for those who are creating mobile ads.
We are seeing “2D” histograms of all the affect reports of all subjects for each of the two conditions. The number of ratings provided for that square of the affect grid are shown as a number and color intensity indicates the number of reports in that square. As you can see, the Reverb ratings were slightly up and to the right as compared to the Control ratings. Looking at the averages…(M = 6.3, SD = 1.3) compared to the control videos (M = 5.8, SD = 1.6). Although the effect size is relatively small, it is statistically significant (UA= 155.0, z=1.64, p < 0.05). Participants also reported feeling more emotionally aroused after watching audio-haptic enhanced videos (M = 6.1, SD = 1.6) compared to the control videos (M = 5.1, SD = 1.5). The difference in conditions is also statistically significant (UA= 325.5, z=2.63, p < 0.004).
(M = 6.3, SD = 1.3) compared to the control videos (M = 5.8, SD = 1.6). Although the effect size is relatively small, it is statistically significant (UA= 155.0, z=1.64, p < 0.05). Participants also reported feeling more emotionally aroused after watching audio-haptic enhanced videos (M = 6.1, SD = 1.6) compared to the control videos (M = 5.1, SD = 1.5). The difference in conditions is also statistically significant (UA= 325.5, z=2.63, p < 0.004).
(M = 6.3, SD = 1.3) compared to the control videos (M = 5.8, SD = 1.6). Although the effect size is relatively small, it is statistically significant (UA= 155.0, z=1.64, p < 0.05). Participants also reported feeling more emotionally aroused after watching audio-haptic enhanced videos (M = 6.1, SD = 1.6) compared to the control videos (M = 5.1, SD = 1.5). The difference in conditions is also statistically significant (UA= 325.5, z=2.63, p < 0.004).
(M = 6.3, SD = 1.3) compared to the control videos (M = 5.8, SD = 1.6). Although the effect size is relatively small, it is statistically significant (UA= 155.0, z=1.64, p < 0.05). Participants also reported feeling more emotionally aroused after watching audio-haptic enhanced videos (M = 6.1, SD = 1.6) compared to the control videos (M = 5.1, SD = 1.5). The difference in conditions is also statistically significant (UA= 325.5, z=2.63, p < 0.004).
The results indicate that participants reported feeling significantly more immersed in the situations portrayed in the haptic-enhanced videos (a feeling of “being there” in the video) (M = 6.8, SD = 1.7) versus the no haptic control videos (M = 5.6, 1.9); (UA= 109, z=2.79, p < 0.003). They also reported a higher level of emotional involvement with the emotions provoked by haptic-enhanced videos (M = 6.6, SD = 2.2) versus no haptic control videos (M = 5.8, SD = 1.6); (UA= 124.5, z=2.4, p < 0.008).
The mean QoE of the audio-haptic videos was consistently rated higher than no haptic control videos. Figure 7 depicts this pattern. This result is significant according to the Mann Whitney test (UA=93.5, z=2.87, p <0.002). As participants were instructed to rate their overall experience, small fluctuations in overall score pairs may be attributable to content preferences. When accompanied by audio-haptic enhancement, videos were rated an average 15% higher than the same videos without haptics.
