First true Natural Interface-(NI)-Controller for your home! Control any TV, lights, thermostat, etc. with only body gestures and voice. Read more
Funding for this project was canceled by the project creator on September 19, 2013.
About this project
If you only have 2 minutes, please watch our quick feature video below
Do you like what you see here ? Please share this page with your facebook and twitter friends, and tell the world about the future of User Interfaces!
Even if you are unable to back us, you would really be helping out by spreading the word!
Your hand is now the remote
How many of you have at least one remote at home? We bet most of you have 3 or more. Have you ever lost one of them, buried deep in the recesses of your sofa? Or maybe you brought one to the kitchen when you went on that break and forgot it, right there on the kitchen counter. Isn't it such a pain to have to go retrieve it AFTER you've settled back on the couch to watch your show?
What about always having to get up just to turn off a light switch, adjust your thermostat or dim the lamp lights? Wouldn't it be cool to not be faced with those issues any more? How awesome would it be to be able to control all those things and more with just your body? Yes, just your body!
Well, we are pleased to present to you our solution – The Flowton Controller.
It's here! The Flowton Controller allows you to use your body as a universal controller for all your electronics. The software inside works by capturing image and sound from it's environment, and converts this raw data into commands that you may use to control anything from your TV, DVD, to thermostat, lights and more.
Use your hands, your body or your voice to execute commands, replacing switches, buttons, and other artificial devices.
What is a Natural Interface (NI) you ask? Well it pretty much means allowing your natural body to execute commands. There is nothing to touch. You simply push a "virtual button" in empty space in front of you, and the 9-point controller springs up around your hand.
From here you can control your television volume, switch channels, or select other connected devices, such as window blinds, and many others.
Just as well, you may call out a voice command - like "Kitchen lights ON", and the computer responds, swiftly performing the action you want it to - illuminating your way before you ever left your comfortable seat.
Think of it as your digital butler. You can either tell it, or show it, and it does.
Amazing right? We certainly think so!
Well, now that you've decided that you want to back us, we can guess exactly what you are thinking...
What reward option should I choose?
In short, if you just want the Controller in your home to use with TV, DVD, wireless thermostat, wireless blinds, etc., then you should go for the 219$ early standard version (limited #).
If however you have many electronics such as table lights, lamps, decorations, alarm clocks, or other objects that you wish to control with Natural Interaction, then go for either the 249$ early extended bundle (limited, comes with 1 smartplug) or the 299$ complete bundle (comes with 2 smartplugs).
The reward for developers is pretty obvious so if you're a developer, get one of those today.
And if you just want to help out and see this product made, you can still back us in the lower rewards. We count everything and acknowledge all contributions!
Exclusive to Kickstarter!
You will not get this anywhere else. It simply doesn't yet exist.
There is nothing on the market that will allow you true Natural Interaction control over every day objects. We are not just talking PC and iPhone here. It's the things you interact with most daily - TV, lights, etc.
Kickstarter has become a one-of-a-kind portal for innovation and groundbreaking products, and we are happy to be a part of it.
If you pre-order now, you will be one of the first people to get their hands on the finished product in all it's glory! You will be the first to bring NI into your home, and puzzle all those around you with random gestures in the air... that somehow now have an effect on the surrounding devices. This is as close to magic you will get for another few years. Believe me, we checked.
And the best part is - we are setting up an exclusive portal just for our backers, so you can get in on the development and evolution of this incredible technology! You will get unprecedented look behind the scenes, and witness this technology become the future.
It is our goal to work closely with our backers and bring the most utility out of this capable device, so we will keep our ears firmly to the ground. However, there is only so much advice and feature requests we can take and implement. Thus, being selective with who we bring in to throw around ideas with, we must be sure that it's people who are passionate about NI and dedicated to making this product all it can be.
So we invite you to come and invent with us!
Is the future here yet?
Flowton unit is a device built for advanced Human Computer Interaction (HCI). Certain forward technologies have lagged behind in the research and development stages. They remain the stuff of science fiction, and our imagination.
We were all promised something extraordinary when the likes of Microsoft® Kinect™ came onto the market. It was now more affordable than ever to own devices that can understand our body. To see it. To hear it. Kinect™ being the prime example consequently became the "fastest selling consumer electronic device" in the world! Tens of millions of people bought into the vision of the next stage of computers. Well, we are happy to inform you that your patience paid off. And here is the result...
The next user interface revolution...
The internet of things. Ubiquitous computing. Things that think. You have heard it all before. And the time has come.
We are surrounded every minute of the day, by computers, everywhere. Ones that only a decade or so ago would have been called supercomputers. It’s time we start putting these concepts to work!
Flowton Controller (command)
So how does it actually work, you ask?
The Flowton Controller receives commands through a built-in color RGB (Red/Green/Blue) camera and 2 microphones. Both need an initiator command for the unit to "start paying attention".
Once the Controller starts capturing image (after performing a gestural initiator command) or sound (after speaking a voice keyword), you may issue follow-on commands addressing a particular device. Such as in example where you initiate the TV controller with your wrist, and do a rightward swipe to change the channel. Or when you say "Bedroom Thermostat Up 5 degrees", you first address the device, and then issue a follow up command.
The Flowton Controller also is able to work with existing depth sensors such as the Microsoft® Kinect™, and others (listed below). By utilizing a different method to capture images, proper depth cameras such as remove any limitations that are dim lighting lighting, color noise, etc. This is why we strongly advocate it's purchase, however it is not mandatory.
Some of the current choices available for purchase:
- Microsoft Kinect – 109$
- ASUS Xtion – 119$
- Primesense Carmine – 199$
- SoftKinetic DepthSense311 – 299$
The speech recognition is based on an offshoot of Julius (Japanese open-source large vocabulary CSR engine) and is capable of recognizing key words necessary for TV, lights and other devices. As with other examples, you "wake" the system up with a trigger command, and follow with an executable command.
The voice input is received via Flowton's 2-microphone array, or in the case of Kinect, utilizing that 4-microphone array. Using an array helps with ambient background reduction and localizing of the sound source, thus better sound capture. There is also an active sound cancelling in place to isolate the sound coming from the TV or any other pre-identified source output. This feature also ensures reliability even with background sound.
Flowton Controller (execute)
Now to actually execute commands, the Controller needs to interact with your electronics and appliances. And it does so through:
Infrared (IR) – this is your TV, DVD, VCR, Stereo systems, and other.
- (1 peripheral comes standard)
Bluetooth – this could link to your Thermostat, Cell phone, Game Consoles, Kitchen appliances, and more.
Wi-Fi – issue commands remotely to your PC, Laptop, Tablet, etc.
ZigBee – aside from using this protocol to interact with other Flowton peripherals, it is also used in various home automation systems, alarm systems, home security, access control (garage doors), and such.
Smart plugs (with dimmer) – this pretty much constitutes everything else, from lamp, to fan, iron, coffee machine, alarm clock, radio, Christmas decorations, and of course the curling iron.
- (1 peripheral with the EXTENDED bundle)
Remotes be gone...
This is it - the final blow to the good ol' remote control. You have served us well...
But let's be honest. The remote controller is still good for SOME things. You just probably will not use it anymore to turn on your TV (it is much simpler now to wave your hand or speak a command). Or switch channels. Or change volume. Etc. The freedom of knowing you can issue any command with only your body is liberating. And once the remote is gone, so are the fights over who has it. Anyone can simply "pull up" the controller at any given time.
So yea, the remote controller is still welcome inside the house. It just won't get much usage in the future...
How easy is the Controller to use?
It takes a bit of practice, yea. But just as with the first mouse, the first touch screen – once you get it, you can never imagine it any other way.
This is a brand new user interface we are talking about here! One without a necessary screen, invisible yet present. Imaginary, yet functional. Without buttons. In essence, we had to fundamentally rethink the way people interact with their devices once at home, or would wish to, had the technology allowed it.
Gestures (Motion) -
Where the sensor can see you, is where you can use your body to execute commands. The Flowton camera unit has ~ 70 degree field of view (horizontal) and ~ 50 degree field (vertical). The practical range is between 0.5 - 3.5 meters away from the sensor.
This is our Flowton Field – a space within your home that is not so empty anymore.
When it comes to an attached sensor, you may refer to the sensor maker for exact details, but typically they all stay in the same operational range, and are close to the numbers stated above.
Everything starts with an initiator command – a certain action that lets the camera start watching you for other commands. This comes preset as the main command – a hand push towards the camera, and also programmable commands, such as you getting up from your couch, and many others.
Just put up your hand, and “push” a virtual button. Done? Ok – you now have a virtual TV controller around your hand… raise your hand a few inches, and the volume starts going up! Flick your hand left, and the channel changes down.
It’s that simple!
Next, push a “button” in the direction of your light stand. This way you directly bring up a “controller” for the light switch/dimmer. Again, move your wrist up to turn the light on, and bring the hand back to center. Now maybe hold your hand left for a few seconds to dim it, and get that movie theater feel.
But wait, no movie is complete without some chips! And as you get up, the Controller recognizes it, and springs into action – pausing the movie until you return.
You get the idea.
Voice (Sound) -
When the sensor can’t see you, is when you may tell the Controller how to behave. Just as with motion, this also requires an “initiator command”.
You can address the device you wish to control, as in the example of: “TV, ON”, or “LIVING ROOM LIGHTS, OFF”.
Intuitive commands such as “ALL APPLIANCES OFF” issued prior to leaving the house will ensure yet again, that you are preserving every ounce of energy.
Want to give your “butler” a name? In the future, you can! I mean, who wouldn’t want their own Jarvis?
Programming the Controller
One of the key important issues is how to program this Controller and connect it with all your current electronic devices. We have ensured that this experience will be swift and effortless.
After receiving the device, you will be required to connect the Controller to your PC, where a quick tutorial and set-up process will take place, which will be available through our website. Here you will create a personal network of all your home electronics and apply it to the Flowton Controller.
From there, you will be directed towards a section where you can assign devices and commands to your gestures. There will be plenty of template controllers available, for the television set, the lights, etc. Equally, you will be able to customize the Controller and personally tailor your brand new natural user interface to the liking.
This is a concept stage portrayal of how the graphic user interface will allow you set up your own controller.
What stage are we in?
We have developed our own motion tracking software with no dependencies. Basically, we had to build everything from the ground up. So there were hurdles along the way, but the result is that we now have a system that can be tweaked and played with, without any restrictions.
Our focus is depth tracking. It is simply unrivaled in object detection and tracking. Color camera tracking on the other hand has an increased rate of false positives in imperfect conditions, does not allow for complex gestures such as directional vectors (when you point to the device you wish to control), and the natural interface is generally less precise. That being said, we are committed to bringing a superb experience with every device, regardless of whether you choose to use the Flowton Controller as is, or supplement it with a depth camera.
Since our vision is to bring this product to the masses in form of a consumer electronic device, we had to make everything run ultra fast so we can have it on a tiny (relative to it's power) ARM board.
Basically, we have developed a proof of concept prototype that functions as intended in its current form. You can see the Flowton Controller V1 in the picture below (as well as in the video during the demo)
We didn't bother fitting a color camera into the V1 prototype because the processor + motion tracking + speech recognition is the crucial combination that we had to spend time on. Also we used a different motherboard and generally had to go through a few different iterations of the circuitry and other hardware inside.
We have finished the hardware designs for the Flowton Controller V2 and worked everything out to allow it to be mass produced. And now we need your help!
We have scoured far and wide seeking the best and brightest. Despite being small in size (under 10 people), we have assembled a well diversified team comprising of software and hardware engineers, designers, business folks and creative thinkers.
The current lineup goes as follows:
Software - Daniel Mohammadi (BASc)
Computer Vision - Ray Lo (PhD)
Hardware - Martin Labrecque (Postdoc)
Computer Science - Amir Hejazi (PhD)
Designer - Faraz Farizi
Product - Alec Gordon
Finance - Kevin Panchalingam
Adviser - Danielle Zhou
Individually, all of our members are over achievers - from running their own hardware companies, to spearheading tech development in fortune 500 firms, to publishing research papers in top journals, and winning scientific innovation awards.
Collectively, the combined expertise and aptitude for building great products is much greater than the sum. Everyone gets to wear many hats and influence the product from every angle. We have every facet of development covered, and are looking forward to building the platform of tomorrow.
After nailing down the core concepts of the interface, we subsequently filed for a patent for this novel method of control. Also we have received a research grant to study the human side of the equation - to find out how well people perform while navigating through this new natural user interface, what can be improved, and so on.
True remote control
Anything you can connect to you can control. The future trend is that of everything going wireless. Thus there is a good chance that with ensured software updates our Controller will age exceptionally well. We would be perfectly happy if all currently owned TV sets had Bluetooth. But they don't, IR is still very much present, and so we have to build around it.
Soon, even your toaster (?) is going to be wireless enabled, it will become smart, and light switches are all going up into the "cloud". With enhanced wireless functionality more and more electronics and appliances can be connected to a personal network.
And we will work on applying that personal network of devices onto an ever present interface that any person can pull up using only what nature gave them. Our bodies are sophisticated enough, we shouldn't need to actually touch anything.
We are working hard on creating an environment where Natural User Interface enthusiasts and developers can all come together and be able to build incredible stuff. There are plenty of ideas already floating around, but we can only build so much, and will need your help!
With depth cameras, think about being able to ID and track any XYZ point in space. You can drag it around. You can issue a trigger command to another. You can place a light switch onto an empty wall. Or if the light stand is within the field of view, you can put the "switch" right onto the appliance, and so you can now touch it and it till turn on/off. This is something we want people to grasp, and then run away with. The possibilities are truly limitless. And we will work very hard on opening up the development to all those who want to build with us.
Follow the updates for when our API will be released. You can expect some time later this year, well before the products ship, so you will have ample time to develop something groundbreaking.
Flowton Controller -
- 1.7 GHz processor
- 2GB of RAM
- 640x480 VGA RGB camera @ 60 fps
- 2 microphones
- 8GB of flash storage
- WiFi 802.11
- Ubuntu OS
Risks and challenges
Just as with any hardware product, there are some inherent challenges that come with working on bringing the best and most functional device to backers.
It is not enough to put together some parts, write some code, and then multiply it in a factory. We had to rethink how to assemble all the powerful hardware into the smallest form possible. There are many aspects that we needed to consider in order to come up with a more efficient and cheaper solutions. A huge amount of time and effort was put into designing, testing and manufacturing the circuitry needed. And in the process we had to design few key parts ourselves.
CHALLENGE 1: Manufacturing
We have developed some key partnerships that will allow us to swiftly transition from building prototypes in the eponymous garage, to mass producing quality consumer electronics devices.
Equally, we took the necessary steps to design a number of our own hardware components along with the enclosures as they will appear in final form when shipping to backers. We have researched the necessary production material costs and validated critical components against required specifications. That said, we had to resort to some strategic component sourcing in order to make the ship date.
However it would be foolish to assume that everything will remain the same once the production process actually begins. Despite the fact, we stand prepared to face whatever challenges come our way, and are committed to delivering an exceptional product experience in the allotted time.
THE NEXT 7 MONTHS
The time after Kickstarter will arguably be the most critical time yet. We have been actively working on the Controller for over a year and a half now, and we utilize research from our team that extends yet further back. We are on track to finish the hardware development in the proceeding months, in order to deliver this product to you early next year.
So our revised timeline will look as follows:
September 2013 - Finalize all hardware designs and ready the parts for manufacture
October 2013 - Obtain the necessary safety/electronics certifications
November 2013 - Pass on the product designs and ramp up production. Backer portal
December 2013 - Complete the SDK and establish smart home integration
January 2014 - Testing and revision. Ship out the Dev Kits
February 2014 - Ready the GUI and finalize the NUI of the Controller
March 2014 - Package everything neatly together and start shipping
It is a tall order to fulfill the obligations in a timely manner, however most of the design choices have already been polished out, and our own brand of user interface has long been in the making. The task remains to deal with whatever setbacks as they come (manufacturing is notoriously tricky), and make sure we do not deviate from the plan at hand.
CHALLENGE 2: Natural User Interface
Transforming code into a usable user interface that people will actually get utility out of requires plenty of abstract thinking, and is the other difficult half to take care of. There have been a few examples of natural user interfaces to control the television set, however since they generally replicate the pointer on screen, and thus rely on the presence of the screen, they cannot be ported well onto other everyday devices without serious restructuring. Not only that, it is important to create something that will be easy and intuitive, and devoid of minor imperfections that can take away from the seamless experience.
We have invested considerable resources to try and design a functional interface that people will have a fun time with. We want to infuse new life into mundane interactions that people often avoid if they could. That is why we are keeping a close eye on our beta testers and are extracting valuable data of user interactions.
For more information and insight please visit our website.Learn about accountability on Kickstarter
Yes, we tried to organize the motion control interface to be easy to understand and intuitive. We will provide plenty of visual support in forms of tutorials, as well as show people how to do an easy setup so that they can get to the fun part sooner.
If used with a Microsoft Kinect depth sensor, our controller will essentially act as a splitter, receiving and transmitting the same raw data from the Kinect and to the Xbox.
Our unit will automatically turn off when Xbox is on (or rather go into a reduced functionality state), not to interfere with the gaming experience. Due to fact that our interface works differently than the Microsoft one, means you will still be able to control volume when gaming. Speech recognition is also turned off.
The range between 2 say the Controller and the IR blaster is around 10 meters. If however you have a vast living space, and the smart plug is quite a distance away, then other peripherals can act as a node and relay the signal down the line. So if it's A->B->C setup, signal from C will jump to B and then to A. Thus your operational range will be increased the more units you have connected to your home Flowton network.
There is one "ON" button, which allows you to either have the controller on all the time in a perpetual low power state, that gets awoken when initiated, or turn the controller off all together. There is also a "pair a device" button, only used in the initial setup, which will also double as a "reset" button should things require you to do so.
We advise everyone to connect the Controller to your home Wi-Fi network during the initial setup, this way we can automatically push new updates as they become available. You unit will notify you when it's being updated with a distinct blink.
In it's current state, only English language is currently supported, however expanding language capabilities is one of the key features we are looking to introduce.
Any device that receives commands through an Infrared Remote Controller, will be able to be controlled through the IR blaster. The setup process will consist of identifying your device model on our website, as well as the Flowton ID, and you will have the commands loaded onto your controller.
We are looking to support all popular and semi popular TV's in the last ~10 years. If your TV is not currently supported at the time of setup, you can either request the database to be updated with your model (to which we will get to in no time), or you can write IR commands to the IR blaster (since it has an IR receiver built in), and program those buttons you use most, while we update with your TV model in the meantime.
We understand the inherent dangers of putting light imaging devices (a.k.a. cameras) in every room, we will ensure that the Controller doesn't pick up any more visual data than it needs to operate. While in low power "disengaged" mode, it will only listed to the speech keyword, or only look for a gesture initiator. No data is stored.
Should you wish to temporarily turn the Controller off, you may do so. You can also turn off one functionality (voice) but leave the other (gestures).
Security is of uttermost concern to our team, and will ensure to have a very strong firewall that will not allow any unauthorized connections to the device. Should there be a breach, the Controller will instantly reset it's network settings, while maintaining normal functionality.
When operating the Flowton Controller, there will be a memory buffer accounting for 30 seconds (by default) of interaction. This will serve to better understand the stream of commands and keep track of previous ones, as well as allow people to initiate "sessions", when you can put your hand down and later continue the interaction without having to push the "virtual button" again.
Since we understand that hovering your hand continually in the air during gestural control can be tiring, we built in the capability to keep track of your hand even when it's outside the 9-point interaction range.
Say if you browsed through a few TV channels, and then decide to rest your hand on your lap, the Controller still tracks your hand, however while it's outside the interaction box (refer to pictures above) no commands can be performed. If you put up your hand within the 30 second time period, you will not need to "reinitiate" the 9-point, otherwise you do.
Should you wish to have a longer buffer time, you can adjust it for up to 1 minute.
Yes, you can move the Controller, and upon it's new place it will re-calibrate itself and do the normal background scan.
You can technically have an unlimited # of Controllers active, and they will learn of each other and include them in the common network (through pairing). If however all your Controllers are hooked up to depth sensor cameras, we advise not to place the depth cameras too close together and pointed in the same direction, to to interference with IR light scatter.
We will support European standard along with Chinese standard. The adapter will also have interchangeable blades.
We built in the LED feedback mechanism for exactly that! The device lights up when the controller is initiated and engaged. Then with subsequent commands it will flash to indicate that its transmitting and receiving. We are working to make a the feedback system minimal and noninvasive, yet functional, and more importantly noticeable!
Also we have an optional sound feedback. It gives off this distinct beep, but so far people use it mainly to become more used to the device, and then they would usually turn it off, and just rely on the LED feedback. In time, they learn how the device works, and they can already know which command will register and how.
Sound feedback is ON by default.
Pledge $5 or more
PART OF THE TEAM: Your contribution will help make the product the best it can be! As our thanks, your name will be included on our website for everyone to see, and you can say - you were there from the beginning! You future order will also be given priority.Estimated delivery:Ships anywhere in the world
Pledge $20 or more
BACKER PORTAL: This is it! You become part of a close-knit community of innovators, futurists and technology enthusiasts. The Flowton Team is forever at reach - this is the first place we look to come up with new features for our natural user interface! Your website name bears a special badge.Estimated delivery:Ships anywhere in the world
Pledge $45 or more
BACKER EXCLUSIVE: A custom designed shirt to show off to the world that you believe in Natural Interaction! Also access to the portal and to help you generate some ideas, we will send out a cool limited edition Flowton Poster with graphic descriptions of all commands and controls that you may now perform in (formerly) empty space! Think of it as a fun game you play with electronics. (10$ shipping for Canada)Estimated delivery:Ships anywhere in the world
Pledge $179 or more
4 backers Limited (716 left of 720)
DEVELOPERS ONLY! For those who want to experiment and build. You will get access to the API, as well as 1 Flowton Dev Kit for your PC (connection required) that communicates with all the peripherals. Also comes with 1 IR blaster & 1 smartplug. Depth sensor not included. (10$ shipping for Canada)Estimated delivery:Ships anywhere in the world
Pledge $219 or more
5 backers Limited (195 left of 200)
EARLY FLOWTON CONTROLLER (STANDARD): Limited unit run, and it's numbered, so you remember the moment! The Flowton Controller comes with 1 IR blaster. (10$ shipping for Canada)Estimated delivery:Ships anywhere in the world
Pledge $229 or more
FLOWTON CONTROLLER (STANDARD): One main unit along with 1 Infrared blaster. Immediate control over all your wireless devices. (10$ shipping for Canada)Estimated delivery:Ships anywhere in the world
Pledge $249 or more
19 backers Limited (181 left of 200)
EARLY FLOWTON CONTROLLER + 1 SMARTPLUG (EXTENDED): One main unit (numbered) along with 1 IR blaster & 1 smartplug. A full range of devices is now interconnected. (10$ shipping for Canada)Estimated delivery:Ships anywhere in the world
Pledge $265 or more
FLOWTON CONTROLLER + 1 SMARTPLUG (EXTENDED): One main unit along with 1 IR blaster & 1 smartplug. Every device in your home is now under your control. (10$ shipping for Canada)Estimated delivery:Ships anywhere in the world
Pledge $299 or more
FLOWTON CONTROLLER + 2 SMARTPLUGS (COMPLETE): The main unit comes with 1 IR blaster and 2 smartplugs. Every device in your home is now under your complete control. (10$ shipping for Canada)Estimated delivery:Ships anywhere in the world
- (35 days)