An Open-Source Sensory Substitution and Sensory Augmentation Development Kit!
We need your help launching the world's first open source sensory substitution and sensory augmentation development platform! We've taken the technology used in research and medical devices and put it on an easy to use device that plugs right into an Arduino Uno or Arduino Mega. The Cthulhu Shield lets anyone experiment and make devices that can expand your sensory experience!
We've made android apps and example programs that will let you use the Cthulhu Shield and your smartphone to 'see' and 'hear' with your tongue without needing to write a single line of code!
For those of you interested in making your own projects, we've written an easy to use Arduino library and provided example code to get you started on projects including tongue-heat-vision, tongue-based GPS directions, and soon, tongue-ultrasonic hearing. But don't limit yourselves to the examples we've provided, the only limit to what you can make is your imagination!
Finally, we designed the Cthulhu to be used as a tongue based computer interface (because if you already have something in your mouth, why not use it to control your computer)? Write your own code to hotkey video game actions, send text messages, or control a wheelchair or mobility device with your tongue.
How does it work?
The Cthulhu Shield creates short biphasic voltage pulses on the electrodes that are in contact with the surface of the tongue. The electric field created by these pulses and the movement of ions in the electric field cause a change in the potential across the membrane of a somatosensory nerve fiber(s) in the tongue, opening voltage gated ion channels, causing the fiber to propagate an action potential to the brain where it is processed and perceived as a touch sensation. This form of neuromodulation is often referred to as electrotactile stimulation. Most of the sensations the Cthulhu is capable of producing feel a little like pop-rocks or bubbles from carbonated beverage.
Patterns and arrangements of these touch, or tactile, sensations can be used to convey information to a person in the form of sounds, images, or other forms of presentation. Initially, a person learns to associate a given pattern of sensation with some sort of meaning (think of tactile languages like Braille), research shows that over time mechanisms of neuroplasticity can route the information to other parts of the brain where images or sounds are usually processed. The net effect, one can argue, is truly seeing or hearing with the tongue. Of course none of this limited to sight and sound, you can feel radio waves, or stock prices on your tongue just as easily as a painting or concerto.
Why the tongue?
The tongue is incredibly sensitive. There are lots of nerves in your tongue, and it turns out that because of the presence of an electrolyte (saliva) and the type of tissue in the tongue, it's incredibly easy to stimulate with low voltage electrical pulses. Additionally, there's a lot of brainpower dedicated to both controlling the movement of the tongue, and processing touch signals coming from the tongue. This combination of features makes it ideal for sensory substitution, as well as interfacing with computers and machines.
An RC network attached to the Arduino I/O pins is used to generate the biphasic waveform used to depolarize nerve fibers. The network is based on designs developed by researchers who founded modern sensory substitution over 20 years ago but has been adapted to work more easily on Arduino based devices. It actually takes up very little space on the shield. This is the business end of the device that produces the electrotactile signal and stimulates the tongue.
The rest of the hardware is used to display which electrodes are being activated, to help the user more quickly make sense of the sensations they are feeling.
During electrotactile stimulation, we can quickly measure the voltage potential on certain individual electrodes. The potential on the electrode is different if your tongue is in contact with the electrode or not, enabling us to sense the position of your tongue on the array, and determine things like tongue swipes, presses, and gestures. We are excited to see the apps and software users develop to use their smartphones, computers, and other devices with their tongues.
As important as the hardware, the open source software library and examples we developed to let users quickly create exciting and innovative uses for tongue based sensory substitution.
The heart of the library is a simple pulse-train generating function that allows you to generate patterns of nerve-depolarizing pulses that are perceived as touch sensations. This pulse generating function is based directly off of techniques developed by the pioneers of this field, but has been slightly tweaked for some of the early research we performed. The function lets the user input different pulse-train/waveform parameters to create different sensations that have been described by researchers as 'tactile colors'. These 'tactile colors' can be used represent actual colors if you are displaying visual information, or anything else, like sound frequency, temperature, etc.
We have provided example source code for Arduino, processing, and android, that will let you use your Cthulhu with your android device camera and microphone, GPS navigation app, or external ultrasonic sound detector and thermal camera without needing to write a single line of code.
What is the Development Status?
The hardware is pretty locked down and tested. We've ordered parts for and assembled several small batches and have suppliers and quotes for larger scale manufacturing.
The base software library is complete, and we have example code/Arduino sketches for tongue-audio sensory substitution using a smartphone, tongue-motion detection using a smartphone, tongue-heat detection using a small IR camera, tongue-pressure sensing, tongue-cursor and 'wasd' game control, and example code to use the 'Komoot' navigation app with an esp32 for GPS directions to your tongue over bluetooth. Basically everything in our video. We need to add some diagrams, instructions, and make more descriptive README files for the repository. You can access the github repo with the Arduino library and examples here.
We will be adding examples for ultrasonic sound detection, magnetometers, accelerometers (on smartphone and standalone), and others in the coming weeks.
Videos of our team using the current examples can be found below.
Above: Tactile Audio to tongue example using the Funkatronics Tactile Waves library and the demonstration example Arduino sketch from the Cthulhu repository. Hardware used is from the 'Uno kit' reward level.
Above: Tactile Heat Camera to Tongue example using hardware from the 'Mega' reward level, the 'mega_heat_cam_with_shield' example from the Cthulhu github repository examples folder. The Grideye library from Sparkfun is also used.
Above: Tactile Motion Camera to Tongue example using hardware from the Uno reward level, the demonstration example from the example folder, and the Cthulhu Camera Demo app downloadable on our github repo.
Above: Full download and setup of Tongue Cursor Control example. Code and instructions can be found in the Cthulhu Repository.
Above: Using the Cthulhu Shield and Arduino Mega with a small pressure sensor (force sensitive resistor).
Above: Downloading the Arduino sketches and using the Komoot app for tongue-based directions. Code is available in our github repo under the directions example folder.
What can I make?
Using the Cthulhu Shield, you can create sensory substitution and sensory augmentation devices.
Sensory Substitution: Sensory substitution is the name that has been given to tools and techniques that take sensory information you would normally receive on one sensory organ, and presents that information in another form to another sensory organ. Your brain then figures out what to do with this information.
You can use your phone's onboard camera to send color, movement, and pixel intensity information to create low resolution images you can feel on your tongue. You can use your phone's onboard microphone to send audio information to your tongue to use the Cthulhu as a tactile hearing aid, or just to feel your favorite song. You can also attach other external sensors to your Arduino, like a small pressure sensor to use the Cthulhu to feel the force you are exerting on an object if you have nerve damage in your hands or are wearing thick gloves. Many people have used tongue stimulation to improve their sense of balance as well by using pressure sensors or accelerometers.
Sensory Augmentation: Ever wanted thermal vision or extra-human hearing ability? Want to sense the electric fields of living creatures like a shark, or give yourself magnetoreception like a migratory bird? All of these things and more can be presented electrotactiley on the tongue. Your imagination is the only limit to the ways you can expand your sensory experience.
In addition to the senses we witness in nature, there are many exciting opportunities to use the Cthulhu to sense data directly from the internet or virtual world. Upvotes, likes, and stock prices can all be felt on your tongue with the Cthulhu and simple computer programs.
Tongue Controlled Devices: Our favorite applications of the Cthulhu tongue control are for assistive devices. The Cthulhu can be used very effectively to control your computer cursor and type with your tongue, but it can also be used to control wheelchairs, or even cars and other vehicles.
For those without impaired mobility, the Cthulhu can be used as an extra controller for gaming (to move or perform certain actions quicker), or to control your smartphone and send texts or emails silently with your tongue.
There are certainly many uses of the Cthulhu Shield we haven't thought of, and are excited to see what users come up with.
Sapien LLC was founded in 2014 by Dr. John Williams, and Joel Moritz Jr.
The company started as a research project attempting to cure John's tinnitus and develop tactile hearing aids to treat his high frequency hearing loss.
Early on, we found many successful examples of tactile hearing aids worn on the arms, hands, and chest that had been used effectively to improve speech recognition in people who were very hard of hearing or profoundly deaf.
We also found many examples of electrotactile tongue stimulation being used to improve the lives of those with vision, and balance disorders. However, we found very little research on using tongue stimulation to get audio information to the brain. The research we did find showed very inconsistent results. We were determined to find the cause of these inconsistencies and make a better tactile hearing aid.
After forming Sapien LLC, we began working closely with Neuroscientist and tongue expert, Dr. Leslie Stone-Roy, and for the past 5 years have collaborated with her on neuroscience research to learn more about the tongue, and build better human machine interfaces, and tactile hearing aids.
Since 2014, we have bootstrapped Sapien LLC, and funded our development efforts with lots of hard work, late nights, small investments, small state grants, and revenue from working full time selling our services as contract engineers. We've done all of this in order to continue our research with Dr. Stone-Roy, and to continue to develop our neuromodulation and human-machine-interface technology.
We are incredibly excited to launch our first product, share our passion with you, and bring this technology out of the laboratory to grow the space where neuroscience and engineering meet.
From all of us at Sapien LLC, thank you for your support, and we are excited to see what our users create.
Risks and challenges
We've done quite a bit of work to minimize risks to backers. We've made functional release candidate devices, and found multiple suppliers for components, and have quotes from several contract manufactures to handle final assembly and fulfillment. We've also ordered and inspected sample packaging and labeling.
If the campaign is incredibly successful and there are significantly more backers than we anticipate, it's possible that shipping to some backers who ordered kits may be delayed, though we've built extra time into our schedule to accommodate this possibility.Learn about accountability on Kickstarter
- (30 days)