KenColour will allow visually impaired people see colours - always, always correctly, and always by themselves, free of charge.
KenColour will allow visually impaired people see colours - always, always correctly, and always by themselves, free of charge. Read more
About this project
"I have a strong opinion about colours when I dress or buy something for my flat. I am often impatient with people's descriptions because I concluded that people see colours often very subjectively, depending on their mood, culture, background or even abilities to recognise colours." Daniela, Hamburg, Germany.
KenColour aims to help Daniela and 300 million more visually impaired people to bring the colour to their lives.
We want KenColour to be FREE for all users. Your contribution will help a good cause - make the life of visually impaired people significantly better.
What is KenColour?
Our goal is to create a colour recognition iPhone application which will significantly improve existing methods out of the box. We believe that by applying heuristic learning algorithms, and in some limited cases by using a calibration card, we will be able to reach close to 100% colour and pattern recognition in diverse lighting conditions.
We want to create an app which correctly and quickly recognises the colour of the objects viewed through the iPhone built-in camera. The app must be easy to use even for people with severe vision impairment or totally blind. For this reason, we will analyse the complete image recorded by the iPhone camera in order to determine the dominant colours and patterns. In this way users won’t need to select a part of the viewable area through a viewfinder. To provide feedback and instructions the app will communicate with the user through iPhone's VoiceOver technology. A simple on-screen text will provide also visual feedbacks to an accompanying sighted person.
Technically correct colour and pattern recognition is a very challenging task to solve. The main computational obstacles include:
- Collecting enough photons in poor lighting conditions to produce an initial good quality image.
- Recognising complex patterns.
- Implementing correct linguistic and cultural description of colours, hues, and patterns.
It's relatively easy to detect the colour of a red t-shirt in bright sunlight. But our goal is to be able to describe a patterned garment in the closet.
Currently existing solutions have limited usability in poor lighting conditions, or require the use of a viewfinder and are therefore unaccessible for blind people. None of the existing solutions provides acceptable pattern recognition, neither do they take into account specific cultural and cognitive needs of the different groups of visually impaired people (born blind, blinded later in life, severely limited vision, etc). Our goal is to overcome all these limitations.
Why is it important?
According to World Health Organisation over 285 million people in the world are visually impaired, of whom 39 million are blind and 246 million have moderate to severe visual impairment. Colour blindness (colour vision deficiency) affects approximately 1 in 12 men (8%) and 1 in 200 women in the world.
The notion of colours is deeply interwoven in our modern language, emotions, culture, and traditions. KenColour will help people who are denied the ability to recognise colours to be an integral part of our society by contributing inclusiveness in all aspects of community life.
How to use KenColour?
KenColour is super simple to use:
- Aim the iPhone towards the direction you would like to scan.
- When the device is roughly aimed towards the object, the VoiceOver will announce the name of the prime colour.
- Hold the iPhone still for a little while, and the VoiceOver will announce first the precise colour name, followed by the pattern description.
What is the technology?
The feedback from organisations and communities of the visually impaired is that current solutions do not work, or work only in very limited cases. Specialised (non-mobile phone based) hardware solutions are excruciatingly expensive and also work with serious limitations.
Our knowledge and experience in developing multiple large astronomical software packages, including astronomical image processing and pipeline, gives us the confidence that we can crack this problem significantly better than any currently existing solution.
- Full image sampling. Contrary to existing solutions KenColour will analyse the entire image. After all, visually impaired people are unable to aim the small area of the focus field towards the object of interest. Analysing the entire image requires extremely fast full image sampling in order to give immediate feedback.
- Motion and focus detection. KenColour will use iPhone's motion detection capabilities in order to detect when the iPhone remains relatively still. When that happens the app will assume that the camera is pointing towards the object of interest and will perform rapid image acquisitions and analysis to resolve colours. Fine lateral movements of the device won’t affect this process. On the contrary, the iPhone can still be moved towards or away from the object (zoom in/out) without causing the data acquisition to stop.
- Determining lighting conditions. Depending on the iPhone model, various tests will be performed to determine as close as possible the exact lighting conditions (e.g. exposures with different ISO settings, exposure times, use of device specific low-level API's, etc.). The exact set of calculations will be determined case by case - the poorer the lighting conditions, the more complex the set.
- Image calibration. In rare cases of very poor non-natural lighting conditions (e.g. an almost dark room with a weak incandescent spotlight) extra calibration is required. The app will ask for the use of a calibration card (which will be produced and purchased separately, or downloaded as a PDF file).
- Producing high quality images. KenColour needs high quality images in order to determine the correct colour composition and colour patterns. To do that we will apply the same well proven methods astronomers use to produce deep-sky photographic images, such as stacking and averaging of multiple images, and subtracting from the result the combined - device and lighting specific - calibration images.
- Identifying colour clusters. In most cases in a single image there will be lighting gradients and shadows. By calculating gradients and comparing histograms produced for different samples, KenColour will determine the set of visible colours regardless of those lighting effects. Only dominant colour(s) will be identified.
- Speed vs. precision. When KenColour is used in familiar environments (expected object arrangement, e.g. "a plate with zucchini in different colours", "a row of clothes hanging in the wardrobe", etc.) fast scanning is essential. By briefly holding the iPhone pointed at the object, only the simple colour (e.g. "red", "orange", "brown", etc.) will be announced. If you hold the iPhone still for a little longer, the detailed colour name and - optionally - the pattern, will be described (e.g. "sulphur yellow", "blue and white stripes", etc.). This mode is important for unfamiliar environments.
- Describing colours. KenColour will be used by people with different language and cultural backgrounds, and different level of visual impairment. The app will support different mapping dictionaries, and will be able to enhance the dictionary with recorded by the app colour names and pattern descriptions.
- Pattern detection (optional, see Stretch goals below). There are patterns that are relatively easy to determine, such as stripes, plaid or dots. But others can be extremely difficult, for example a flower pattern. Depending on the level of your support, we will implement different pattern recognition algorithms. Most of them will be based on a combination of edge detection, cluster analysis, and repetition detection. As a minimum, KenColour will list the dominant colour names.
- Additional learning heuristics (optional, see Stretch goals below). In order to minimise the need of using a calibration card, a set of heuristic learning algorithms will allow KenColour with the time to remember specific lighting environments and object compositions (e.g. "home in the evening", "my kitchen desk", "my wardrobe", "the office shelf").
Why we need your help?
- We want KenColour to be a free of charge application!
- We want KenColour to be precise, fast, and reliable.
- We want to support and enhance KenColour longterm.
Unfortunately the majority of the visually impaired belong to the low-income part of society - any help is important! The ability to recognise colours - always and reliably - makes these people less dependent on others' help. The application needs to be maintained for any future hardware and software releases.
Although we have experience with most of the image processing, clustering, fast histogram analysis, etc. algorithms we plan to use, a significant amount of time and resources must be spent on fine tuning and proofing with a large image database and many field tests.
Our entire research contribution will be made publicly available and free of patents.
What are the stages and the timeline?
- Capture and classify a large amount of test images.
- Bundle the algorithms in a playground environment which will allow us to easily chain different methods and play with their parameters.
- Design, develop, and test for usability different UI control elements.
- Finalise our app UI.
- Transfer the chosen algorithms and the most optimal parameter sets to the iOS app.
- Initial performance analysis.
- Test on various iPhone models.
- Deliver KenColour app to early-bird testers, collect feedback.
- Rapidly implement suggestions and improvements.
- Beta testing.
- Submission to AppStore.
Phase 5: (optional, if we raise more than the pledged amount)
- Translating to more languages.
- Colour blindness specific functionality.
- Complex patterns recognition.
What are the stretch goals?
The most important goal to meet is to correctly recognise and describe in English colours in various lighting conditions. This will be covered by the pledge amount. Any further fundings will help us further enhance the ability of KenColour as follows:
- Add support for additional languages.
- Add colour-blindness specific enhancements.
- Add heuristic learning based on location, time of the day, and past experience.
- Add complex textures recognition.
- Add support for image analysis for people with normal sight.
Who are we?
We are spread all over Europe and come from different professional backgrounds - astronomy, physics, molecular biology, even computer science. We have been close friends with iOS and all its siblings and predecessors, back to the years of NeXTstep. So far some of our projects include:
- Specialised financial and business solutions.
- Raw data processing and analysis for mobile operators.
- Software for research projects, e.g. a complete software package for control of a large and heterogeneous network of remotely controlled robotic astronomical observatories.
If you want to learn more about us...
Risks and challenges
We are a very experienced software development and deployment team with over 10 years of solid project track record, therefore most potential risks are already taken into account.
Currently we see the following challenges that could lead to project delays:
1. Some of the algorithms might need significant effort to be made performant enough to work on older (but supported by the current version of iOS) iPhone models or to be adapted for iOS, in a way, that preserves battery life better.
2. As any AppStore app, there are risks related to Apple's app approval process, that could delay the launch day.Learn about accountability on Kickstarter
Support this project
- (40 days)