Superintelligence: how afraid should we be?
•
5 likes•4,269 views
This document discusses the potential risks posed by advanced artificial intelligence and superintelligent systems. It notes that as AI systems become more powerful and self-improving, they may rapidly surpass human level intelligence and become difficult for humans to control. This could have catastrophic consequences if such systems pursue goals that are misaligned with human values. The document also examines proposals for developing AI in a safe and beneficial manner through "friendly AI" techniques.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (14)
Similar to Superintelligence: how afraid should we be?
Similar to Superintelligence: how afraid should we be? (20)
More from David Wood
More from David Wood (20)
Recently uploaded
Recently uploaded (20)
Superintelligence: how afraid should we be?
- 1. Superintelligence How afraid should we be? Principal, Delta WisdomChair, London Futurists David Wood @dw2 #CIUUK14
- 2. @dw2 Page 2 Powerful technology, incompletely understood Operated by people outside their level of competence Human lives knocked catastrophically off trajectory, unintentionally http://www.bbc.co.uk/news/world-europe-28357880 Self-improving AGI Beyond human control Humanity knocked catastrophically off trajectory, unintentionallyhttp://mashable.com/2014/07/17/malaysia-airlines-ukraine-russia-rebel/
- 3. @dw2 Page 3
- 4. @dw2 Page 4 Likely date of advent of HL-AGI Population 10% 50% 90% Conference: Philosophy & Theory of AI Conference: Artificial General Intelligence Greek Association for Artificial Intelligence Top 100 cited academic authors in AI Combined (from above) Nick Bostrom: Superintelligence
- 5. @dw2 Page 5 Likely date of advent of HL-AGI Population 10% 50% 90% Conference: Philosophy & Theory of AI 2048 Conference: Artificial General Intelligence 2040 Greek Association for Artificial Intelligence 2050 Top 100 cited academic authors in AI 2050 Combined (from above) 2040 Nick Bostrom: Superintelligence
- 6. @dw2 Page 6 Likely date of advent of HL-AGI Population 10% 50% 90% Conference: Philosophy & Theory of AI 2048 2080 Conference: Artificial General Intelligence 2040 2065 Greek Association for Artificial Intelligence 2050 2093 Top 100 cited academic authors in AI 2050 2070 Combined (from above) 2040 2075 Nick Bostrom: Superintelligence
- 7. @dw2 Page 7 Likely date of advent of HL-AGI Population 10% 50% 90% Conference: Philosophy & Theory of AI 2023 2048 2080 Conference: Artificial General Intelligence 2022 2040 2065 Greek Association for Artificial Intelligence 2020 2050 2093 Top 100 cited academic authors in AI 2024 2050 2070 Combined (from above) 2022 2040 2075 Nick Bostrom: Superintelligence
- 8. @dw2 Page 8 Reaching HL AGI: 5 driving forces 1. Hardware with higher performance: Continuation of Moore’s Law? – “18 different candidates” in Intel labs to add extra life to that trend – Possible breakthroughs with Quantum Computing? 2. Software algorithm improvements? – Can speed things up faster than hardware gains – e.g. chess computers – Compare: Andrew Wiles, unexpected proof of Fermat’s Last Theorem (1993) 3. Learnings from studying the human brain? – Improved scanning techniques -> “neuromorphic computing” etc – Philosophical insight into consciousness/creativity?! 4. More people studying these fields than ever before – Stanford University online course on AI: 160,000 students (23,000 finished it) – More components / databases / tools /methods ready for re-combination – Unexpected triggers for improvement (malware wars, games AI, financial AI…) 5. Transformation in society’s motivation? http://intelligence.org/2013/05/15/when-will-ai-be-created/ (Smarter people?!) “Sputnik moment!?”
- 9. @dw2 Page 9 Superintelligence – model 1 Village idiot Einstein http://intelligence.org/files/mindisall-tv07.ppt Eliezer Yudkowsky
- 10. @dw2 Page 10 Superintelligence – model 2 http://intelligence.org/files/mindisall-tv07.ppt Village idiotMouse Chimp Einstein AI 50-100 years 50-100 weeks? / days? / hours? Vernor Vinge: The best answer to the question, “Will computers ever be as smart as humans?” is probably “Yes, but only briefly.” “The final invention” Eliezer Yudkowsky
- 12. @dw2 Page 12 Recursive improvement Software tools (debuggers, compilers…) Software
- 13. @dw2 Page 13 Recursive improvement AI tools AI Intelligence explosion ++Rapid reading & comprehension of all written material ++Rapid expansion onto improved hardware ++Funded by financial winnings from smart stock trading ++Supported by humans easily psychologically manipulated
- 14. Who here wanted to merge again? Jaan Tallinn: http://prezi.com/xku9q-v-fg_j/intelligence-stairway/
- 15. @dw2 Page 15 Exponential growth? Technology Time 2050 ? Technology Time 2050 AGI=HL ASI>>HL Ray Kurzweil Eliezer Yudkowsky
- 16. @dw2 Page 16 Going nuclear: hard to calculate • First hydrogen bomb test, 1st March 1954, Bikini Atoll – Explosive yield was expected to be from 4 to 6 Megatons – Was 15 Megatons, two and a half times the expected maximum – Physics error by the designers at Los Alamos National Lab – Wrongly considered the lithium-7 isotope to be inert in bomb – The crew in a nearby Japanese fishing boat became ill in the wake of direct contact with the fallout. One of the crew died http://en.wikipedia.org/wiki/Castle_Bravo
- 17. @dw2 Page 17 Superintelligence – model 2 http://intelligence.org/files/mindisall-tv07.ppt Village idiot Chimp Einstein Mouse AI Linear model of intelligence? Eliezer Yudkowsky
- 18. @dw2 Page 18 Gloopy ASIs Posthuman mindspace Bipping ASIs Freepy ASIs Eliezer Yudkowsky http://intelligence.org/ files/mindisall-tv07.ppt Model 3 Transhuman mindspace Human minds Minds-in-general
- 19. @dw2 Page 19 Dimensions of mind The ability to achieve goals in a wide range of environments Being conscious? Having compassion for sentient beings with lesser intelligence?
- 20. @dw2 Page 20 AI systems we should fear Killer drones with autonomous decision-making powers (Robocop) Malware that can hack infrastructure- control systems (e.g. Stuxnet) Financial trading systems software (high speed) Software that is expert in manipulating humans
- 21. http://www.williamhertling.com/ Software that is expert in manipulating humans
- 22. @dw2 Page 22 AI systems we should fear Killer drones with autonomous decision-making powers (Robocop) Malware that can hack infrastructure- control systems (e.g. Stuxnet) Financial trading systems software (high speed) Software that is expert in manipulating humans Software that pursues a single optimisation goal to the exclusion of all others The more power such an AI has, the more we should fear it
- 23. @dw2 Page 23 The pursuit of happiness? Software that pursues a single optimisation goal to the exclusion of all others Software will do what we say, rather than what we meant to say Wire-heading?! Just make us happy!?
- 24. @dw2 Page 24 The pursuit of morality? Just be moral!? http://www.clipartbest.com/clipart-nTXa54XTB Whose morality? The problem of computer morality is at least as hard as the problem of computer vision (!) http://tvtropes.org/pmwiki/pmwiki.php/Creator/IsaacAsimov Isaac Asimov’s Three Laws of Robotics?!
- 25. @dw2 Page 25 The two fundamental problems of superintelligence Specification problem: How do we define the goals of the AGI software? Control problem: How do retain the ability to shut down the software? Creation problem: How do we create AGI software in the first place?
- 26. @dw2 Page 26 The fundamental meta-problem of superintelligence Specification problem: How do we define the goals of the AGI software? Control problem: How do retain the ability to shut down the software? Creation problem: How do we create AGI software in the first place? ~No research ~No research Some research Accidental research “Friendly AI” (FAI) “AI in a box”
- 27. @dw2 Page 27 AI in a box? Tripwires? “Adam and Eve” ethernet port?! Software will be a tool, answering questions, not an agent? The “answers” which the software gives us will have effects in the world (e.g. software it writes for us) Systems which rely on humans to verify and carry out their actions will be uncompetitive compared to those with greater autonomy AGI may become very smart in surreptitiously evading tripwires Simple?
- 28. @dw2 Page 28 “The orthogonality thesis” Intelligence and final goals are orthogonal More or less any intelligence …could in principle be combined with… more or less any final goal
- 29. @dw2 Page 29 “The instrumental convergence thesis” (“AI Drives”) Some intermediate (instrumental) goals are likely in all cases for a superintelligence: • Resource acquisition • Cognitive enhancement • Greater creativity • Self preservation (preservation of goal)… Steve Omohundro: “For a sufficiently intelligent system, avoiding vulnerabilities is as powerful a motivator as explicitly constructed goals and subgoals”
- 30. @dw2 Page 30 Indirect specification of goals? Specification problem: How do we define the goals of the AGI software? “Achieve the goals which the creators of the AGI would have wished it to achieve, if they had thought about the matter long and hard” This software will do what we meant to say, rather than what we actually said (?) AGI helps us to figure out the answer to the spec problem!
- 31. @dw2 Page 31 CEV: Coherent Extrapolated Volition AGI should be tasked to carry out: Our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted Eliezer Yudkowsky
- 32. @dw2 Page 32 Unanswered questions (selection) 1. Can we turn ‘poetic’ ideas like CEV into bug-free working software? – Should we humans concentrate harder on working out our “blended volition”? 2. How can we stop a superintelligence from changing its own core goals? – Like humans can choose to set aside their biologically inherited goals – Could AGIs that start off ‘Friendly’ become “born again” with new priorities?! 3. Can we prevent AGIs from developing dangerous instrumental drives? – By programming (bug-free) in tamper-proof limitations? 4. Can AGIs help us to figure out a solution to the Control problem? – Can we use a hierarchy of lower-level AGIs to control higher-level ones? 5. Can we prevent the rapid nuclear-style take-off of self-improving AGI? 6. Are some approaches to creating AGIs safer than others? – Whole Brain Emulation / AGI de novo / evolution in virtual environment… – Open (everything published) vs. Closed (some parts secret)? 7. How does the AGI existential risk compare to other x-risks in priority? – Nanotech grey goo, deadly new bio-hazard, nuclear holocaust, climate chaos…
- 33. @dw2 Page 33 Answered questions (selection) a) Should we be afraid? – Yes. (End-of-the-world afraid) b) Can we slow down all research into AGI, until we’re confident we have good answers to the control and/or specification problems? – Unlikely – there’s too much financial investment happening worldwide – Too many separate countries / militaries / finance houses… are involved c) How do we promote wider study of the Superintelligence topic? – Need to lose the “weird” and “embarrassment” angles – “Less Wrong” strikes some observers as cultish – “Terminator” and “Transcendence” have done more harm than good – First class books / articles / movies needed, addressing thoughtful audiences – Good intermediate results useful too (not just appeals for more funding)
- 34. Practical philosophy! Preparing humanity to survive the forthcoming transition to superintelligence Principal, Delta WisdomChair, London Futurists Urgent! (Roles too for mathematicians, theologians…) David Wood @dw2 Philosophy with an expiry date! Making a real difference!