Homeostasis is nature’s engineering behind the most complex autonomic system that exists: the human body. Homeostasis is a self-regulating process by which biological systems tend to maintain stability while adjusting to conditions that are optimal for survival. Disruption in homeostasis results in malfunctioning of natural autonomic system causing chronic diseases. Chronic diseases have been the leading cause of death and human suffering in the last 50 years. They also have resulted in highest financial burden for individuals and countries. This can be corrected using external augmentation of the homeostasis loop. Recent progress in artificial pancreas for Type 1 Diabetes is a compelling example for such augmentation. In this paper we discuss emerging multimodal approaches for such augmentation in the context of chronical diseases. We show that multimodal sensing and fundamental technology developed by multimedia computing community may offer powerful augmentation of natural homeostasis to assist in management of chronic diseases.
1. Multimodal Augmented Homeostasis
Ramesh Jain
Department of Computer Science
UCI Future Health (https://futurehealth.ics.uci.edu/)
University of California, Irvine
jain@ics.uci.edu
With Several Collaborators
2. Multimodal Augmented Homeostasis: Agenda
• What is Multimodal Augmented Homeostasis?
• Health as a ‘Grand Challenge’ for Multimodal computing.
• Personal Health Navigator for Augmenting Homeostasis.
• Progress and Challenges.
2
3. Papers that are related to the ideas presented here.
1. Ramesh Jain, Laleh Jalali:
Objective Self. IEEE MultiMedia 21(4): 100-110 (2014)
2. N. Nag and R. Jain, “A navigational approach to health: Actionable guidance for improved quality of life,”
IEEE Comput., vol. 52, no. 4, pp. 12–20. April 2019.
3. Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, Ramesh Jain, “Food Computing”, ACM Computing
Surveys, 52(5):1-36, September, 2019
4. Weiqing Min, Shuqiang Jiang, and Ramesh Jain, ‘Food Recommendation: Framework, Existing Solutions,
and Challenges’, IEEE Trans on Multimedia, November 2019.
5. Nitish Nag, Vaibhav Pandey, Ramesh Jain:
Health Multimedia: Lifestyle Recommendations Based on Diverse Observations. ICMR 2017: 99-106
6. Hyungik Oh, Ramesh Jain:
From Multimedia Logs to Personal Chronicles. ACM Multimedia 2017: 881-889
7. Nitish Nag, Vaibhav Pandey, Preston J. Putzel, Hari Bhimaraju, Srikanth Krishnan, Ramesh Jain: Cross-
Modal Health State Estimation. ACM Multimedia 2018: 1993-2002
8. Hyungik Oh, Ramesh C. Jain:
Detecting Events of Daily Living Using Multimodal Data. CoRR abs/1905.09402 (2019)
9. Nitish Nag:
Health State Estimation. CoRR abs/2003.09312 (2020)
10. Ali Rostami, Vaibhav Pandey, Nitish Nag, Vesper Wang: Personal Food Model. ACM Multimedia 2020.
11. Laleh Jalali and Ramesh Jain: Event Mining for Explanatory Analysis (Book), Coming from ACM Books. 3
4. Augment
• “technologies that enhance human productivity or capability, or that
somehow add to the human body”
• to make greater, more numerous, larger, or more intense.
• make (something) greater by adding to it; increase.
6. Multimedia
• Essential for understanding real world
•No single mode provides complete
information in a real situation.
• Different modes must be combined for
understanding a situation.
• Different modes have complementary
information.
7. Multimodal is the future of Multimedia
• Multimedia
• GPS
• Heartrate
• Temperature
• Accelerometer
• …
7
8. Big Data is Multimedia Data
Multimedia
Realtime
Credibility
9. Dominant Applications of Multimedia
• Image Search
• Video on demand
• Video-conferencing
• Shopping
• Social Media
• The Best is yet to come – just now becoming popular.
9
12. What is Homeostasis?
• Homeostasis is the state of steady internal, physical,
and chemical conditions maintained by living systems.
• Maintain optimal conditions for
• Temperature
• Glucose level
• pH
• Oxygen
• …
• Maintains the constancy of the internal environment in which the
cells of the body live and survive.
• Works every moment of life.
13. Important Turning Point in Health
In the mid-20th century, the primary causes of death
worldwide shifted from infections to chronic conditions.
15. * Source: CDC
The Effect of Chronic Diseases, 2016*
7 of 10 Deaths
Due to chronic diseases
86% of Costs
Driven by treating chronic
diseases
Heart Disease Cancer Type 2 Diabetes Brain Diseases Lung Diseases Bone Diseases
We all are likely to face Chronic Diseases.
16. Why are Chronic Diseases so Common?
• All bodily functions deteriorate with age.
• When this dysfunction affects Homeostasis, the Onset of the Chronic
Disease starts.
• Usually by the time, this is detected, it is too late.
• P4 (predictive, preventive, personalized and participatory) is the
new Paradigm in Healthcare
16
18. Homeostasis is Nature’s Engineering
Homeostasis: any self-regulating process by which biological
systems tend to maintain stability while adjusting to conditions
that are optimal for survival.
18
20. Basic Systems Theory
Where x, u, and y are state, input, and output vectors;
A, B, C, and D are appropriate matrices for transformation.
Observability: Determine state using measurements.
Controllability: Get to desirable state.
21. Type 1 Diabetes
• Your pancreas doesn't make insulin or makes very little insulin.
• Treatment aims at maintaining normal blood sugar levels through
regular monitoring, insulin therapy, diet, and exercise.
21
22. Impact:
It has been a life-
saver for many
patients and their
families. ‘Artificial Pancreas’ for Type 1 Diabetes.
Cybernetics is now Used for Augmenting Homeostasis
Miracle for Type 1 Diabetes Patient
23. Why Are These Birds Huddling?
By Blalonde - Own work, CC BY-SA 4.0,
https://commons.wikimedia.org/w/index.php?curid=37172033
26. Augmented Homeostasis:
• Health State Estimation.
• A disease is just an undesirable region in health state space.
• Guidance: life-style, environment, and medication.
• Actuation
• Personal Model
• Goal State
• Close-loop perpetual operation
Self-regulating external process by which human systems achieve
health goals to maximize their quality of life.
30. Digitalization made getting directions easy.
Maps
Points of Interest
Starting Point
Destination
Mapping
System
Getting to a destination: 20 Years Ago.
31. Getting to a destination: 10 Years Ago.
Maps
Points of Interest
Current State : GPS
Current Traffic:
Waze
Destination
Cybernetic loop
Navigation
system
Knowing Current State is Magical.
32. Measure
Collect all data
Perpetual Health Guidance
Guidance
Estimate
(Health State)
Influence and
Compliance
How do we close
the loop?
Inspiration:
Navigation Systems.
33. When do people get best healthcare?
When they are about to die – in ICU.
34. Personal Health Navigator
•Guides you to better health.
•Uses real-time information to redirect you
around a disease.
•Provides step-by-step directions to get you to
your optimal health state, whether you’re a
professional athlete or a taxi driver.
36. Personal Health Navigator: Diabetes
Activity
Food
Medication
Glucose Level
Activity
Food
Medication
Glucose
Level in last
3 hours
Sensitivity to
different foods.
Sensitivity to
medication.
Activity Checks.
Glucose Level
Checks.
37. • Ping Wang and Ramesh Jain co-founded,
then joined by other people
• Licensed UCI technology
• A start-up company formed (Team of 7 now,
raised seed funding)
• App available on iPhone
Personal Diabetes Navigator
Dianavi is building Personal
Diabetes Navigator.
Their application is now
available on iPhone.
Disclaimer: I am a co-founder.
38. Sensors to Estimate Health State
• Two major classes:
• Low cost, continuous, unobtrusive (LCU) – Used in Wellness
• High cost, episodic, intrusive (HEI) – Used in Diagnostics
• Currently LCU is used for ‘Wellness’ and HEI for ‘Disease Control’.
• Though both are related and complementary, they are treated in
different silos.
• Both are to improve the quality of Life of a person.
43. Person :== Genome + Lifestyle + Exposome
‘Omics’
Personal
Model
Exposome
Personicle
AI + Learning
• Propensity to
diseases
• Rule Based
Health Behavior
• Your behavior
and health profile
Could be for
Whole Life
Recent Past
After a particular health Condition
(such as heart attack)
45. Add other personal streams as
NEEDED.
RAMESH JAIN@Imaging and Multimedia Analytics 45Jan 29, 2020
46. Personicle: Personal Chronicle
46
Personicle is a complete chronological log of person-
centered events in life (health, social, environmental)
captured objectively but augmented using subjective and
diagnostic information as needed.
Continuous
Observations
Episodic
Observations
Personicle
Person
• All continuous sensors work 24/7.
• Episodic data when suggested by
• Either continuous data tracking or
• By doctors.
48. Pattern
Mining
High-level
Pattern
Formulation
Pattern Query
…
Data Streams Event Streams Semi-interval Event Sequences
Data-Driven Analysis
Hypothesis-Driven Analysis
While air pressure is high, pollution
starts increasing gradually, within T
time units asthma outbreak happens.
((pollution_inc_steadily ;ωT asthma_outbreak ) ||
airpressure_stayshigh)
Example:
Interactive Visualization
Interactive Event Mining: Correlation and Causality
Bottom Up
Top Down
49. Input to the System
• Food
• Exercise/activities
• Sleep
• Stress
• Environment
• Medicine
49
50. Food is the most important input.
50What I like to eat is not what my body likes!
51. Food Recommendation
•Who: Person.
•What: Food Items.
•When: Context. Food
Recommendation
System
Food
Items
Context
Personal
Model
List and
volumes
of Food
Items
and How
to get
them.
52. Food Logging is important application.
52
Information in a Food Log for Personal Food Model
Learn from Activity and sleep
Logging.
Can we develop a platform
for easier logging?
54. Two Challenges to Solve the Problem!
• Creating a detailed Food Chronicle
for a person.
• Creating a detailed World Food
Atlas.
55. Event Mining for Rule-Based Actionable Insights
Having difficulty sleeping well on several nights.
Warn him when he is at a Sichuan restaurant.
Personicle
Food Stream
Sleep
Spicy food and 2 glasses of wine result in sleepless
nights.
56. Homeostasis is essential for Life.
• Sensing for health state is multimodal.
• Multimedia community is expert in information extraction from
multimodal data streams.
• Augmented Homeostasis is a new grand challenge for our
community.
56
57. My dream: Empowering Personal Health
For every living person in the world.
Multimodal Computing will bring it.
We can make it happen.
Should we start? Now?
58. Thanks for your attention and Time
Time for Questions and Discussions
Continue after the session: jain@ics.uci.edu
58