Top Rated Pune Call Girls Ravet ⟟ 6297143586 ⟟ Call Me For Genuine Sex Servi...
Building a Thought Controlled Drone
1. Building a
Thought-Controlled Drone
Jim McKeeth, Embarcadero Technologies
jim.mckeeth@embarcadero.com
@JimMcKeeth
Presented at
InterDrone Conference
Sept 9th, 2015
2. About Jim McKeeth
• Lead World Wide Developer Evangelist
• Longtime developer
• Object Pascal, Java, C#, JavaScript, Objective-C, etc.
• Invented and patented swipe to unlock in 2000
• US Patent # 8352745 & 6766456, etc.
• Improvisational performer with ComedySportz Boise
• I am not a neuroscientist, but I find this all fascinating!
3. • Founded 1993
• Provides Architect, Database and Developer Tools
• 3.2 Million Customers including 97% of Fortune 2000
• 35+ Product Awards – Consistent Innovation
• 500+ Employees in 29 Countries
• www.embarcadero.com
4. Agenda
• Drone technical details
• Headset technical details
• Drone API
• Headset SDK
• Concerns, Challenges and the Future
5. More Information
• My email: jim.mckeeth@embarcadero.com
• My blog: delphi.org
• This Slide Deck
http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone
• Overview Slide Deck
http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input
• The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC
6. “In five years, science
fiction will be real!”
— Claus Torp Jensen (@ClausTorpJensen)
Senior Technical Staff Member and Chief Architect
API Economy at IBM (@ibmapimgt)
What will the world be like
in five years?
9. The Drone: Parrot AR.Drone 2.0
• Price: $299.99
• Runs Linux internally, but not user accessible
• Contains it’s own WiFi access point
• Connect via UDP & TCP
• Send control strings
• Maintain transmission rate to keep alive
• Can also receive status and video back
10. Completely Rebuildable
HD Battery $60
Gears+Shafts $15
Tools $20
Cross bars $30
Screws $7
Propellers $12
4x Engines $200
Bottom structure $60
Main board $110
Nav board $70
Charger $40
Outdoor haul $25
————————
Total $650
11. New contender: Parrot Bebop
• Price: $499.99
• Lots of upgrades, especially camera
• Also contains own WiFi access point
• Whole new SDK
• iOS, Android & Linux
• Published Nov 2014
• Incompatible with AR.Drone 2.0
12. SDK vs. API
• Parrot AR.Drone 2.0 provides both an API and SDK (Bebop has just an
SDK at this point.)
• An SDK is a software library that does additional work locally - makes it
easier to handle video streaming, etc.
• An API is a specification of how to connect and control.
• SDK works with specific tools and platforms.
• Typically SDK uses API.
• My examples use the API so it works across platforms.
14. Electroencephalography (EEG)
• Measures electrical activity along the
scalp.
• Detects voltage fluctuations resulting from
ionic current flows within neurons of brain.
Image source: https://commons.wikimedia.org/wiki/File:EEG_cap.jpg
15. Neural Oscillation / Brain WavesBeta
15-30Hz
Waking Consciousness and Reasoning
Alpha
7-14Hz
Deep Relaxation
Delta
4Hz
Deep Sleep
Theta
4-7Hz
Light Meditation And Sleeping
Mu
8-13Hz
Voluntary Movement
Gamma
30-100Hz
Conscious Perception
Waveforms by Hugo Gamboa
16. NeuroSky’s MindWave (also in XWave headsets)
• EEG with 1 dry electrode
• Measures “attention,” “meditation” and eye-blinks
• Bluetooth communication
• SDK for iOS, Android, PC & Mac
• $129.99 to $79.95 retail neurosky.com
• Hacking: http://makezine.com/2011/05/10/behind-
the-brain-blinker/
17. InteraXon’s Muse
• 7 dry EEG sensors via Bluetooth
• Focuses on “brain fitness”
• 4 hour battery
• Android, Mac & Windows SDK now,
iOS coming soon
• Works with Windows, OS X, iOS, Android, etc.
• $299 at choosemuse.com
18. OpenBCI
• Open source hardware & software
• Bluetooth, Arduino, EEG
• 8 wet Electrodes per board (daisy chainable)
• Full open EEG & BCI platform
• SDK provides EEG data
• www.openbci.com starting at $449
• No iOS or Android yet
19. Emotiv EPOC
• 16 wet electrodes
• 14 EEG electrodes + 2 reference electrodes
• Gyroscope to read head movements
• 4 mental states, 13 conscious thoughts & facial
expressions – 4 Processing suites
• Wireless (encrypted) for Windows, Linux or Mac
• $399 for Desktop
• $499 for Bluetooth Smart (EPOC+)
• www.emotiv.com
20. Emotiv Insight
• Expected eventually
• Starting at $299 (estimate)
• 5 dry sensors + 2 reference
• Bluetooth 4.0 LE (Smart)
• Battery Life: 4+ hours
• SDK: Android, iOS, Mac, Linux and
Windows Platforms
• www.emotiv.com
• Comparison http://bit.ly/1cfliTn
22. Expressiv Suite
• Detect facial expressions
• Eyelid & eyebrow positions
• Wink, blink, furrow
• Horizontal eye movement
• Smile, laugh, clenching & smirking
• EEG sensors picking up signals to muscles
(not brain waves)
• Very fast (10ms)
Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg
23. Uses for Facial Detection
• Anything requiring fast input
• Facial gesture detection
• Basic eye tracking
• Fast input & responses
Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg
24. Affectiv Suite
• Detect emotions / mental states
• Excitement
• Engagement or Boredom
• Meditation
• Frustration
• Passive detection
• Variable levels of each
• Short & long term tracking
Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg
25. Uses for Emotion Tracking
• Realtime media evaluation
• Stress analysis and management
• Mood monitoring
• For more accuracy combine with
other data sources
• Eye tracking, heart rate, etc.
Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg
26. Cognitiv Suite
• Detect conscious thoughts
• Requires training
• Detect 13 thought patterns with intensity level
(4 at a time)
• Left, right, up, down, forward, pull, CW, CCW,
left, right, sway-back, sway-forward,
Disappear
• Not as fast as Expresiv
Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg
27. Uses for Cognitive Thought Identification
• Whole new input method
• Variable intensity for fine control
• Observe subconscious responses
• More training improves accuracy
• Improves the model
• Trains your brain to focus
Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg
28. Brain-Computer Interface
Internet of Things
Autonomous Vehicles
expectations
Speech-to-Speech Translation
time
Plateau of
Productivity
Slope of EnlightenmentTrough of
Disillusionment
Peak of
Inflated
Expectations
Innovation
Trigger
Plateau will be reached in:
more than 10 years5 to 10 years2 to 5 yearsless than 2 years
obsolete
before plateau
Gartner's 2014
Hype Cycle for
Emerging Technologies
August 11, 2014
Smart Advisors
Data Science
Prescriptive Analytics
Neurobusiness
Biochips
Affective Computing
Smart Robots
3D Bioprinting Systems
Volumetric and Holographic Displays
Software-Defined Anything
Quantum Computing
Human Augmentation
Connected Home
Quantified Self
Smart WorkspaceVirtual Personal Assistants
Digital Security
Bioacoustic Sensing
Natural-Language Question Answering
Consumer 3D Printing
Cryptocurrencies
Complex-Event Processing
Big Data
In-Memory Database Management
Content Analytics
Hybrid Cloud Computing
Gamification
Augmented Reality
Machine-to-Machine
Communication Services
Mobile Health
Monitoring
Cloud Computing
NFC
Virtual Reality
Gesture Control
In-Memory Analytics
Activity Streams
Enterprise 3D Printing
3D Scanners
Consumer Telematics
Speech Recognition
Wearable User Interfaces
30. Command Strings
• Connect via WiFi access point to 192.168.1.1 UDP port 5556
• Command string format: AT*[cmd]=[seq],[arg array]
• Sequence number must not repeat
• Argument array is comma separated typically
• Floating point numbers must be encoded as decimals (more later)
• Send command every 30 ms (>2 seconds = session dropped)
31. Command String Examples
• FlatTrim: AT*FTROM=[seq]
• Movement: AT*PCMD=[seq],1,Roll, Pitch, Vert, Yaw
• Values of 1.0 to -1.0
• Use 0 in all values to hover
• Takeoff: AT*REF=[seq],290718208
• Land: AT*REF=[seq],290717696
• Emergency: AT*REF=[seq],290717952
32. Float as Integer
• Single precision floating point number is 32 bits
• Integer is 32 bits
• Map IEEE-754 bit value from float to integer
• -0.8 = 0xBF4CCCCD =
10000001011001100110011001 = 1085485875
33. Drone Concerns
• Manual override - keyboard override BCI, just in case
• Send emergency command to clear error state
• Start with FlatTrims to level drone
• Use state machine to send command 30 times a second
1. Keyboard override
2. BCI Input
3. Hover
34. ToDo with Drone
• Process NavData (sent back on UDP port 5554)
• Get battery data to know remaining flight time, errors, etc.
• Process Video stream (sent back on UDP port 5555)
• Display on Google Glass for HUD
• Find use for drone “Animations” like flips
• Maybe use LED animations when receiving BCI input
36. SDK Format
• On Windows a Visual C++ DLL provides the interface
• Examples in VC++, C#.NET, Java, Python & Matlab
• All my code is in Object Pascal
• On OS X a dylib provides the interface
• Examples in Qt & Xcode
37. Emotiv EDK Workflow
• Call EE_EngineConnect to connect
• Create event handler EE_EmoEngineEventCreate
• Create state structure EE_EmoStateCreate
• Define actions to listen for and then train those actions (more later)
• Every 10 ms call EE_EngineGetNextEvent (more later)
38. Handle Cognitive Actions
• If state is EDK_OK then call EE_EmoEngineEventGetType
• If EventType is not EE_EmulatorError then call
EE_EmoEngineEventGetUserId
• If EventType is EE_CognitivEvent then it was a training event (next slide)
• If EventType is EE_EmoStateUpdated then get
EE_EmoEngineEventGetEmoState
• Read ES_CognitivGetCurrentAction and
ES_CognitivGetCurrentActionPower
39. Listen & Train
• Listening:
• Call EE_CognitivSetActiveActions with mask of Actions
• Call EE_CognitivSetTrainingAction with the ID of the action to train
• Call EE_CognitivSetTrainingControl with COG_START
• On EE_CognitivEventGetType of EE_CognitivTrainingSucceeded call
EE_CognitivSetTrainingControl with COG_ACCEPT
41. New Emotiv Xavier SDK
• EPOC+ and Insight are Bluetooth Smart
• Connect to mobile devices - iOS & Android
• Allows to additional user definable cognitive actions
• Still only listens for 4 cognitive actions at a time
43. Thoughts to Movement
• Moves while thought active, otherwise hover in position
• Speed of movement = intensity of thought
• How to allow more than 4 directions of movement?
• Use Expressiv facial gesture to switch between sets
• Future consideration: Use strobing grid of controls
44. Typing via BCI
Jonathan Wolpaw
Wadsworth Center
Albany, NY
~15 seconds / key
. . . . . . . .
. A B C D E F .
. G H I J K L .
. M N O P Q R .
. S T U V W X .
. Y Z 1 2 3 4 .
. 5 6 7 8 9 0 .
. . . . . . . .
46. Purpose of Feedback
• Using a visual feedback gives the
wearer something to focus on
• Otherwise they often change their
thought pattern when the drone is
moving
• Also gives them immediate awareness
of what is picked up
47. Moverio BT-200 Product Overview
Integrated Sensors
Accelerometer
Compass
Gyro
Head Set
Camera
Earbuds
with
Microphone
+x
+y
+z
48. Epson Moverio BT-200 Specs
Epson BT-200
Retail Price $699
Display Type Transparent
Resolution 960×540 QHD
Android v4.0.4 API Level 15 - Ice Cream Sandwich
Camera VGA, 640x480
WLAN 802.11b/g/n 2.4GHz
Bluetooth 3.0
micro USB v2.0 (On-The-Go)
GPS P-GPS
CPU
TI OMAP 4460
(ARM Cortex A9 Dual Core 1.2GHz)
RAM 1GB
Flash 8GB
microSD Max 32GB
51. More Information
• My email: jim.mckeeth@embarcadero.com
• My blog: delphi.org
• This Slide Deck
http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone
• Overview Slide Deck
http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input
• The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC