Project image
)}
$7,127
pledged of $5,000pledged of $5,000 goal
67
backers
9days to go

All or nothing. This project will only be funded if it reaches its goal by Fri, March 1 2019 8:00 PM UTC +00:00.

Joel Moritz JrBy Joel Moritz Jr
First created
Joel Moritz JrBy Joel Moritz Jr
First created
$7,127
pledged of $5,000pledged of $5,000 goal
67
backers
9days to go

All or nothing. This project will only be funded if it reaches its goal by Fri, March 1 2019 8:00 PM UTC +00:00.

About

Cthulhu Shield and Arduino
Cthulhu Shield and Arduino

An Open-Source Sensory Substitution and Sensory Augmentation Development Kit!

We need your help launching the world's first open source sensory substitution and sensory augmentation development platform! We've taken the technology used in research and medical devices and put it on an easy to use device that plugs right into an Arduino Uno or Arduino Mega. The Cthulhu Shield lets anyone experiment and make devices that can expand your sensory experience!

We've made android apps and example programs that will let you use the Cthulhu Shield and your smartphone to 'see' and 'hear' with your tongue without needing to write a single line of code!

Cthulhu Shield Uno Kit Reward Level
Cthulhu Shield Uno Kit Reward Level

For those of you interested in making your own projects, we've written an easy to use Arduino library and provided example code to get you started on projects including tongue-heat-vision, tongue-based GPS directions, and soon, tongue-ultrasonic hearing. But don't limit yourselves to the examples we've provided, the only limit to what you can make is your imagination!

Cthulhu Shield Mega Kit reward level (exact brand of breakout board or sensors might change).
Cthulhu Shield Mega Kit reward level (exact brand of breakout board or sensors might change).

Finally, we designed the Cthulhu to be used as a tongue based computer interface (because if you already have something in your mouth, why not use it to control your computer)? Write your own code to hotkey video game actions, send text messages, or control a wheelchair or mobility device with your tongue. 

Writing an Arduino Sketch with Cthulhu Shield and Microsoft Onscreen Keyboard
Writing an Arduino Sketch with Cthulhu Shield and Microsoft Onscreen Keyboard

How does it work?

The Neuroscience:

The Cthulhu Shield creates short biphasic voltage pulses on the electrodes that are in contact with the surface of the tongue. The electric field created by these pulses and the movement of ions in the electric field cause a change in the potential across the membrane of a somatosensory nerve fiber(s) in the tongue, opening voltage gated ion channels, causing the fiber to propagate an action potential to the brain where it is processed and perceived as a touch sensation.  This form of neuromodulation is often referred to as electrotactile stimulation. Most of the sensations the Cthulhu is capable of producing feel a little like pop-rocks or bubbles from carbonated beverage. 

Patterns and arrangements of these touch, or tactile, sensations can be used to convey information to a person in the form of sounds, images, or other forms of presentation. Initially, a person learns to associate a given pattern of sensation with some sort of meaning (think of tactile languages like Braille), research shows that over time mechanisms of neuroplasticity can route the information to other parts of the brain where images or sounds are usually processed. The net effect, one can argue, is truly seeing or hearing with the tongue. Of course none of this limited to sight and sound, you can feel radio waves, or stock prices on your tongue just as easily as a painting or concerto.

Why the tongue?

The tongue is incredibly sensitive. There are lots of nerves in your tongue, and it turns out that because of the presence of an electrolyte (saliva) and the type of tissue in the tongue, it's incredibly easy to stimulate with low voltage electrical pulses. Additionally, there's a lot of brainpower dedicated to both controlling the movement of the tongue, and processing touch signals coming from the tongue. This combination of features makes it ideal for sensory substitution, as well as interfacing with computers and machines. 

The Engineering:

Hardware: 

An RC network attached to the Arduino I/O pins is used to generate the biphasic waveform used to depolarize nerve fibers. The network is based on designs developed by researchers who founded modern sensory substitution over 20 years ago but has been adapted to work more easily on Arduino based devices.  It actually takes up very little space on the shield. This is the business end of the device that produces the electrotactile signal and stimulates the tongue. 

Stimulation Circuitry and Electrode Array
Stimulation Circuitry and Electrode Array

The rest of the hardware is used to display which electrodes are being activated, to help the user more quickly make sense of the sensations they are feeling.

Active Electrode Display
Active Electrode Display

During electrotactile stimulation, we can quickly measure the voltage potential on certain individual electrodes. The potential on the electrode is different if your tongue is in contact with the electrode or not, enabling us to sense the position of your tongue on the array, and determine things like tongue swipes, presses, and gestures. We are excited to see the apps and software users develop to use their smartphones, computers, and other devices with their tongues. 

Software:

As important as the hardware, the open source software library and examples we developed to let users quickly create exciting and innovative uses for tongue based sensory substitution. 

The heart of the library is a simple pulse-train generating function that allows you to generate patterns of nerve-depolarizing pulses that are perceived as touch sensations. This pulse generating function is based directly off of techniques developed by the pioneers of this field, but has been slightly tweaked for some of the early research we performed. The function lets the user input different pulse-train/waveform parameters to create different sensations that have been described by researchers as 'tactile colors'. These 'tactile colors' can be used represent actual colors if you are displaying visual information, or anything else, like sound frequency, temperature, etc.

The four main functions of the Cthulhu Arduino library. Simply tell the 'update' function which electrodes to turn on, and what waveform parameters to use for each electrode, then call the 'stimulate' function.
The four main functions of the Cthulhu Arduino library. Simply tell the 'update' function which electrodes to turn on, and what waveform parameters to use for each electrode, then call the 'stimulate' function.

We have provided example source code for Arduino, processing, and android, that will let you use your Cthulhu with your android device camera and microphone, GPS navigation app, or external ultrasonic sound detector and thermal camera without needing to write a single line of code. 

What is the Development Status?

The hardware is pretty locked down and tested. We've ordered parts for and assembled several small batches and have suppliers and quotes for larger scale manufacturing. 

The base software library is complete, and we have example code/Arduino sketches for tongue-audio sensory substitution using a smartphone, tongue-motion detection using a smartphone, tongue-heat detection using a small IR camera, tongue-pressure sensing, tongue-cursor and 'wasd' game control, and example code to use the 'Komoot' navigation app with an esp32 for GPS directions to your tongue over bluetooth. Basically everything in our video. We need to add some diagrams, instructions, and make more descriptive README files for the repository. You can access the github repo with the Arduino library and examples here.

We will be adding examples for ultrasonic sound detection, magnetometers, accelerometers (on smartphone and standalone), and others in the coming weeks. 

Videos of our team using the current examples can be found below. 

 project video thumbnail
Replay with sound
Play with
sound

Above: Tactile Audio to tongue example using the Funkatronics Tactile Waves library and the  demonstration example Arduino sketch from the Cthulhu repository. Hardware used is from the 'Uno kit' reward level.

 project video thumbnail
Replay with sound
Play with
sound

Above: Tactile Heat Camera to Tongue example using hardware from the 'Mega' reward level, the 'mega_heat_cam_with_shield' example from the Cthulhu github repository examples folder. The Grideye library from Sparkfun is also used.

 project video thumbnail
Replay with sound
Play with
sound

Above: Tactile Motion Camera to Tongue example using hardware from the Uno reward level, the demonstration example from the example folder, and the Cthulhu Camera Demo app downloadable on our github repo.

 project video thumbnail
Replay with sound
Play with
sound

Above: Full download and setup of Tongue Cursor Control example. Code and instructions can be found in the Cthulhu Repository.

 project video thumbnail
Replay with sound
Play with
sound

Above: Using the Cthulhu Shield and Arduino Mega with a small pressure sensor (force sensitive resistor). 

 project video thumbnail
Replay with sound
Play with
sound

Above: Downloading the Arduino sketches and using the Komoot app for tongue-based directions. Code is available in our github repo under the directions example folder.

What can I make?

Using the Cthulhu Shield, you can create sensory substitution and sensory augmentation devices.

Sensory Substitution: Sensory substitution is the name that has been given to tools and techniques that take sensory information you would normally receive on one sensory organ, and  presents that information in another form to another sensory organ. Your brain then figures out what to do with this information.

You can use your phone's onboard camera to send color, movement, and pixel intensity information to create low resolution images you can feel on your tongue. You can use your phone's onboard microphone to send audio information to your tongue to use the Cthulhu as a tactile hearing aid, or just to feel your favorite song. You can also attach other external sensors to your Arduino, like a small pressure sensor to use the Cthulhu to feel the force you are exerting on an object if you have nerve damage in your hands or are wearing thick gloves. Many people have used tongue stimulation to improve their sense of balance as well by using pressure sensors or accelerometers. 

Our example android app capturing the motion of Mac the cat. The coordinates of the screen where motion is detected are sent to Cthulhu, to be electrotactiley displayed to the user.
Our example android app capturing the motion of Mac the cat. The coordinates of the screen where motion is detected are sent to Cthulhu, to be electrotactiley displayed to the user.

Sensory Augmentation: Ever wanted thermal vision or extra-human hearing ability? Want to sense the electric fields of living creatures like a shark, or give yourself magnetoreception like a migratory bird? All of these things and more can be presented electrotactiley on the tongue. Your imagination is the only limit to the ways you can expand your sensory experience. 

Artist Yuping Rae Hsu at an interactive art piece where users experienced changes in her physiological and emotional state via electrotactile stimulation.
Artist Yuping Rae Hsu at an interactive art piece where users experienced changes in her physiological and emotional state via electrotactile stimulation.

 In addition to the senses we witness in nature, there are many exciting opportunities to use the Cthulhu to sense data directly from the internet or virtual world. Upvotes, likes, and stock prices can all be felt on your tongue with the Cthulhu and simple computer programs. 

Tongue Controlled Devices: Our favorite applications of the Cthulhu tongue control are for assistive devices. The Cthulhu can be used very effectively to control your computer cursor and type with your tongue, but it can also be used to control wheelchairs, or even cars and other vehicles. 

For those without impaired mobility, the Cthulhu can be used as an extra controller for gaming (to move or perform certain actions quicker), or to control your smartphone and send texts or emails silently with your tongue. 

There are certainly many uses of the Cthulhu Shield we haven't thought of, and are excited to see what users come up with.

Sapien LLC

The Sapien Engineering Team.
The Sapien Engineering Team.

Sapien LLC was founded in 2014 by Dr. John Williams, and Joel Moritz Jr. 

The company started as a research project attempting to cure John's tinnitus and develop tactile hearing aids to treat his high frequency hearing loss. 

Early on, we found many successful examples of tactile hearing aids worn on the arms, hands, and chest that had been used effectively to improve speech recognition in people who were very hard of hearing or profoundly deaf. 

We also found many examples of electrotactile tongue stimulation being used to improve the lives of those with vision, and balance disorders. However, we found very little research on using tongue stimulation to get audio information to the brain. The research we did find showed very inconsistent results. We were determined to find the cause of these inconsistencies and make a better tactile hearing aid. 

After forming Sapien LLC, we began working closely with Neuroscientist and tongue expert, Dr. Leslie Stone-Roy, and for the past 5 years have collaborated with her on neuroscience research to learn more about the tongue, and build better human machine interfaces, and tactile hearing aids. 

Since 2014, we have bootstrapped Sapien LLC, and funded our development efforts with lots of hard work, late nights, small investments, small state grants, and revenue from working full time selling our services as contract engineers. We've done all of this in order to continue our research with Dr. Stone-Roy, and to continue to develop our neuromodulation and human-machine-interface technology.

We are incredibly excited to launch our first product, share our passion with you, and bring this technology out of the laboratory to grow the space where neuroscience and engineering meet. 

From all of us at Sapien LLC, thank you for your support, and we are excited to see what our users create. 

Risks and challenges

We've done quite a bit of work to minimize risks to backers. We've made functional release candidate devices, and found multiple suppliers for components, and have quotes from several contract manufactures to handle final assembly and fulfillment. We've also ordered and inspected sample packaging and labeling.

If the campaign is incredibly successful and there are significantly more backers than we anticipate, it's possible that shipping to some backers who ordered kits may be delayed, though we've built extra time into our schedule to accommodate this possibility.

Learn about accountability on Kickstarter

Questions about this project? Check out the FAQ

Support

  1. Make a pledge without a reward

    $
    Kickstarter is not a store.

    It's a way to bring creative projects to life.

    Learn more about accountability.
  2. Select this reward

    Pledge US$ 25 or more About US$ 25

    Support Research and Development

    Support our development of assistive technology and human machine interfaces. We will put your name in a list on our github repo as a supporter of this project.

    Less
    Estimated delivery
    8 backers
    $
    Kickstarter is not a store.

    It's a way to bring creative projects to life.

    Learn more about accountability.
  3. Select this reward

    Pledge $75 or more About $75

    Cthulhu Shield

    One Cthulhu Shield, and one flexible electrode array. For those who already have an Arduino Uno or Arduino Mega and have some DIY electronics, software, or app development experience. Hook up your own sensors and electronics (not included), write a computer program, or write an app and use an OTG cable or bluetooth module (also not included) to control or send data to your Cthulhu.

    Less
    Estimated delivery
    Ships to Anywhere in the world
    17 backers
    $
    Kickstarter is not a store.

    It's a way to bring creative projects to life.

    Learn more about accountability.
  4. Select this reward

    Pledge $125 or more About $125

    Uno Package

    Everything needed to power and control your device with a computer or smartphone. Includes one Cthulhu Shield, and one flexible electrode array, one Arduino Uno, one USB A to B cable, one USB type to micro B OTG cable, and one micro B to type C adapter.

    Less
    Estimated delivery
    Ships to Anywhere in the world
    14 backers
    $
    Kickstarter is not a store.

    It's a way to bring creative projects to life.

    Learn more about accountability.
  5. Select this reward

    Pledge $250 or more About $250

    Mega Package

    A package for those interested in Hardware development and DIY electronics. Contains Cthulhu Shield, flexible electrode array, an Arduino Mega, and lots of sensors. Includes force and flex sensor, temperature sensor, light sensor, magnetometer, accelerometer, esp32 devkit C (can be used for Bluetooth and wifi connectivity), Logic level Converter, and a low resolution infrared camera.

    Less
    Estimated delivery
    Ships to Anywhere in the world
    8 backers
    $
    Kickstarter is not a store.

    It's a way to bring creative projects to life.

    Learn more about accountability.
  6. Select this reward

    Pledge $2,000 or more About $2,000

    Educator Package

    Contains 3 Mega packs, and 10 Uno packs. For teachers, researchers, maker/hackerspaces, etc.

    Less
    Estimated delivery
    Ships to Anywhere in the world
    0 backers
    $
    Kickstarter is not a store.

    It's a way to bring creative projects to life.

    Learn more about accountability.