Can brain science + several little motors + a cell phone expand the limits of our biology?
I'm Dr. David Eagleman, a neuroscientist at Baylor College of Medicine in Houston, Texas. My graduate student Scott Novich and I are on a mission to see how far we can push the limits of human perception, and what it means to be human in general. We want to invite you along for the ride!
We're undertaking an audacious task: developing the most advanced sensory substitution system to-date. We're building a wearable vest that communicates sound to the brain using the sense of touch. We expect this will be powerful enough to give deaf individuals a new "sense" of hearing. We call it the Versatile Extra-Sensory Transducer, or VEST.
The idea of sensory substitution is not new: it was pioneered by Paul Bach-y-Rita in 1969 with blind participants. He developed a dental chair with an array of push pins on its back, which was attached to a video-camera feed. Blind participants sat in this chair, and felt what was presented in front of the camera. After practice, the participants began to develop a visual intuition for the sensations they felt. Today, the current incarnation of this device is called the Brainport, and blind individuals have been able to use this in complex visual tasks (like obstacle course navigation).
Our goal is to leverage the promise of sensory substitution in the realm of hearing, using the latest algorithms and hardware. We've developed a working prototype and have great preliminary data:
We hypothesize that our device will work for deaf individuals, and even be good enough to provide a new perception of hearing. This itself has a number of societal benefits: such a device would cost orders of magnitude less than cochlear implants (hundreds-to-thousands as a opposed to tens-of-thousands), be discrete, and give the wearer the freedom to not be attached to it all the time. The cost effectiveness of the device would also make it realistic to distribute it widely in developing countries.
More exciting than this, however, is what this proof of principle might enable: the ability to feed all sorts of new and profound sensory information into our brains. Anything from Twitter, stock markets, the weather, and beyond.
Why we want your help
We need your help in taking this study to completion, and are excited about inviting you to share the ride of science. Funds raised will go towards a wide range of costs associated with this endeavor, including:
- Prototype design and refinement costs (from purchasing components to manufacturing circuit boards)
- Funding graduate student salaries for this research (help smart guys like Scott!)
Visit the website for this project (eaglemanlab.net/sensory-substitution)! We will continually post details and updates here. We are also happy to answer questions on this Kickstarter page about our proposed research plan:
Step I: Prototype Refinement (The hard work!)
We're developing and experimenting to determine the best method of compressing and translating audio so it can be understood using vibrations on the skin. We have engineered several candidate algorithms to be tested and compared against one another. We are also working to optimize the layout of motors on the vest: what is the best mapping between the sensory auditory information and the spatial location and vibration patterns of the vibratory motors?
Step II: Behavioral Testing (The Exciting Part!)
Once we fine-tune the prototype and training procedure, the next step is to quantify how well the device works with deaf participants over a range of ages and types of hearing loss. These results will be published in scientific journals and will be available to you!
Step III: Brain Imaging Studies (Another Exciting Part!)
We'll study the brain mechanisms behind this kind of learning by imaging the brains of participants as they move through training using functional magnetic resonance imaging and diffusion tensor imaging. Again, our results will be published in scientific journals and will be available to you
Current prototype Q&A
- How long does someone need to wear the VEST to be able to understand something? Our preliminary data hints that leaning to interpret audio likely takes on the order of 2-3 weeks. We hope to gather better estimates and answers thanks to your contributions for this project!
- What are the limits? For audio, it's a matter of the quality of our algorithms, which are designed around the constraints of our sense of touch. Our best algorithms at the moment are able to capture enough information that speech and basic environmental sounds are intelligible. Musical enjoyment is not happening at the moment, but this always has the potential to change!
What can it do now? The current prototype is fully usable with
real-time processing capabilities and is portable. The
effectiveness of the device and our algorithms with long-term usage will be measured thanks to your contributions!
- What does it feel like? Does it hurt or chafe? The device is comfortable enough to wear for extended periods. We'll be making it even more comfortable as we move forward.
- How heavy is it? The Vest currently weighs about 10 lbs (4-5 kg). This weight can also be cut down significantly in future iterations.
- Does this device work for all deaf individuals? This is what we hope to find out with your support! Whether someone is young or old, deaf since birth or deaf as an adult, science will answer these questions for us!
We have an assortment of prizes as a way of showing our gratitude for your support of this project. Our graphic-designer extraordinaire friend Ben Mautner has created a stylish VEST logo (the bars are an artistic rendition of the formants of someone saying "vest"), which we're prepared to print up on t-shirts as one of our rewards:
At higher funding levels, we have rewards ranging from signed copies of my books, some one-on-one Google Hangout time, and even getting your very own vest to experiment with.
Highlights/Notes on some of the reward levels:
1. Explorer level ($10): We'll give you an app (Android only) that lets you listen to the audible results of our algorithms in real-time. And we'll send updates so you can track our progress! While the audio we decode may not reflect the actual perceptions one may develop by wearing the VEST, this is one of the diagnostic tools we use in our lab to determine if our algorithms sound good. It is recommended you have a more recent Android device.
2. Hacker level ($25): We'd love to spur others to innovate in this area. We'll help you get started with some of the core functionality that is needed for implementing a real-time sensory substitution device using an Android and Arduino framework. We'll give you well-commented tutorial projects that (i) demonstrate setting up Bluetooth communication between Arduino and Android, (ii) perform real-time audio processing on Android, and (iii) linking both concepts together! This being said, we're not commercial developers, so we make no claims about the optimality or quality of our code. We just know it works!
3. Sense-sationalist ($3000) and Tactile-technician ($3500) levels: We'll wire up and ship you your very own VEST! And we'll send you app-updates as our research progresses. We'll also send you our sound-training apps so you can be a remote participant if you'd like!
Some important notes: (i) You will need to supply your own up-to-date Android tablet or phone (running Ice Cream Sandwich or newer), (ii) The hardware you will receive is in no way a finished product and (ii) we make no warranties on how long it lasts, how well it will work, or how safe it is.
Clear instructions will be provided on how to operate the device. The device will be tested to ensure it is fully operational before being shipped, and troubleshooting applications will be provided. We will do our best to help with fixes and troubleshooting if the Vest breaks at your end.
Risks and challenges
This is primarily a *science* project that involves some cool hardware. Our hypothesis, which has been carefully formulated by previous scientific work and years of research, is that deaf individuals can come to gain a new perception of audition using our device. As with any scientific hypothesis, a negative outcome is always a possibility, yet even a negative outcome still counts as scientific progress. One thing is for certain: we'll definitely discover a lot about the flexibility of the brain along the way. Thank you for helping us on the journey!Learn about accountability on Kickstarter
- (30 days)