Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Eldercare Research: Cognitive Computing for an Aging Society
1. Eldercare Research
Cognitive Computing for Aging Society
Hiro Takagi
STSM, IBM Research - Tokyo
Aging Strategic Initiative lead, IBM Research
Human Computer Interaction area lead, IBM Research
Academy of Technology member
2. 62 87
28%
27%
Dementia 15% + MCI 13%
estimation by Japanese Government, 2012
of people over the age of 65 are
at risk of dementia.
People over the age of 65, 2017.
70%of people want to work over the
age of 70 (and forever).
Life expectancy of women in
1950 and 2015.
* numbers in Japan
8. Use case barrier
Psychological barrier
User interface
barrier
Price Barrier
Small fonts, small buttons
Don’t know how to operate
Fear, uncertainty
Cannot imagine use cases
Barriers for Senior Citizens to Start Using
Information Technologies
9. Target Actual
(average)
Fingertip angle + Parallax
Landed on
Took off
(px)
(px)
Touch duration + Tremor
Unintentional Motion
Accessibility: Touching a target location
10. Gesture Analysis for Novice Elderly Assist (2014)
Press release & images (Japanese only)
IBM Japan Press Release: http://bit.ly/1gavLuN
KDDI R&D Labs Press Release: http://bit.ly/1hK1B2K
KDDI Corporation Press Release: http://bit.ly/1f5zbTx
Provided appropriate guidance to
users based on user's level of
smartphone operation
proficiency (skill and operational
misstep) by analyzing
smartphone operation logs with
mathematical analysis
technology
13. Elderly Skill Matching System – J-scouter
Senior profile
datasets
(Free-form text)
Job / Activity
Descriptions
(Free-form text)
Find best
persons
Find best
jobs/activities
A Prime minister initiative
“Work-style transformation.”
“Senior workforce” is one of
focuses.
A senior worker with rich
experience in legal compliance is
working as an advisor in a start up
company
Supported by
Japan Science and Technology Agency
Senior HR company
10,000 seniors registered
from TV BS11
By Kyodo News
15. Japan Post Project
• Pilot with 1000 elderlies completed (Oct. 2015 – Oct. 2016)
• Medicine reminders, video phones, shopping assist, photo sharing, etc.
Video
21. IBM Accessibility Research
IoT, Health and Aging
Ecosystem Partners /
Enriched Data
Watson IoT & Health Platform
Sensors &
Devices
Client’s
Data
Personality Insights &
Social Interaction Data
Other Data Sources
Applications
Data &
Platforms
01
0110
0010
001001
Cognitive Services for Elders
Empowered Living
Empowered Social
Empowered Care
Knowledge
Reactor
“Infrastructure
Providers”
22.
23. • Voice-base natural user interface for daily watchover
• Analyze life patterns, feelings, interests and issues from daily conversation (e.g. cognitive decline detection)
• Share information with families and care givers
• Provide support that can keep the elderly self-sufficient
Conversation as Sensors
25. Intent of Each Questionnaire
Questionnaire Intent
“What did you have last night?” Dementia (Neuro Cognitive Disease) Risk
Assessment
“Did you go out today?” Activity checking
“You mentioned about gardening in diary. Are you
interested in community gardening?”
Recommendation of activities
”Which city do you want to travel?” Marketing
“The shop, you mentioned last week, is now on sale!” Promotion
“We delivered a package, but you were not there.
Please contact post office"
Notification (business related)
25
26. Emotion and Physical Condition Recognition
Voice data Features
988 acoustic features
for emotion recognition.
Intensity
Loudness
12 MFCC
Pitch (F0)
Probability of voicing, F0
envelope,
8 LSF,
Zero-Crossing Rate
Labeled
voice
dataset
ML-based
Regression
Estimation
Feeling good
or bad
40
0
100
Physical
condition
Ex: Good night
*Each circle represents one greeting.
*One participant data
Emotion
NegativePositive
Confusion FrustrationDelight Flow
Physical Condition
Emotion
Yorktown – Speech team
Tokyo Research
Happy
Sad
27. Dashboard for Family and Care Givers
27
Condition
Emotion (voice sound analysis)
Personality analysis (Watson
personality insights)
Daily activities
Eating habits
Dementia questionnaire scores
28. Dashboard Visualization Examples
28
“what did you do today?” = Variety of daily activities.
practice, songs, poems, musical instruments,
sessions, friends, music hall, …. House (largest), cleaning, washing, room,
study…..
Smaller variety and words are not so active.Wider variety, and words are active.
“what did you have today?” = Variety of nutrition
Wider variety, and healthy meals
Vegetables, salad, radish, juice, natto, miso, eggs,
….
Rice (largest), bento, sandwich, ….
Smaller variety and ready-made meals
29. Feature extraction of gait
• Gait speed & its variability
• Step frequency
• Stride time variability
• Step-length & its variability
• Foot swing velocity
• Stance and stride time
Detection
• Motor function
• Fall risks
• Cognitive decline
• Episodic memory
• Executive function
• Diseases
• Parkinson disease
• MCI
• Alzheimer’s disease
Gait and Cognitive Decline