AR-Pinboard: An Augmented Reality (AR) Virtual Pinboard
AR-Pinboard: An Augmented Reality (AR) Virtual Pinboard
Pin Augmented Reality (AR) Virtual Pinboards anywhere you want in the real world.Interact with them using smartphones and smartglasses.
Pin Augmented Reality (AR) Virtual Pinboards anywhere you want in the real world.Interact with them using smartphones and smartglasses. Read more
AR-Pinboard is an Augmented Reality (AR) Virtual Pinboard that can be pinned anywhere you want in the real world, even in mid-air! Users can see, and interact with, AR-Pinboards using smartphones, tablets and smartglasses (such as Google Glass and others). The Virtual Pinboards can also be shared with your social media contacts for collaborative use, making it a unique and new form of social media platform.
Once you pin a virtual pinboard somewhere (with just an aim, tap, and go actions), the pinboard remains fixed to that place (saving its position and orientation) and you can start inserting and organizing items within the pinboard, such as photos, videos, text notes, contents from the web, etc. The virtual pinboard doubles as a virtual web browser too enabling users to just browse the web or easily copy contents from the web into the virtual pinboards.
There are three components in AR-Pinboard:
- The AR-Pinboard App.
- The AR Web Browser feature.
- The AR-Pinboard Web Platform.
The AR-Pinboard App is built upon a unique Augmented Reality (AR) engine named Owl Engine. The Owl Engine is the result of over two years of research and development in the fields of Computer Vision and Image Processing. The Owl Engine uses the camera (and soon the motion sensors too) of a smartphone, tablet or smartglasses to track and map the scene around a user to create something like a 3D panorama of the scene. This tracking and mapping happens automatically as the user moves the smartphone or smartglass in space to interact with the AR-Pinboards. The Owl Engine makes it possible to save the exact location and orientation of the virtual pinboards within the user's scene. The AR-Pinboards can be any desired size, even sizes larger than the camera's field of view. The AR-Pinboards will remain fixed to a position and orientation on the scene selected by the user by a simple aim, tap, and go! (see videos).
Below you can see a video demo of a working AR-Pinboard App prototype running on Android and showing some use cases (Please focus on what happens rather than how it looks, it's just a prototype):
The AR-Pinboard App only requires a standard colour camera and motion sensors that are widely available in smartphones, tablets and smartglasses.
Please, read the FAQ at the bottom of the page for details on how to operate AR-Pinboards.
AR-Pinboard users will often want to add to the pinboards items that come from the web. This will be easily achieved with the AR Web Browser feature.
You just have to place an AR Web Browser near an AR-Pinboard, and drag&drop whatever you want, from the browser to the pinboard (by tapping on the item on the AR Web Browser, moving the smartphone to the AR-Pinboard and releasing the tap). At the same time, items in the pinboard can be hyperlinked to webpages in the browser (launching a new AR Web Browser if none is already launched).
Below you can see a video demo of a working AR Web Browser prototype running on Android (Please focus on what happens rather than how it looks, it's just a prototype):
Obviously, you can just use the AR Web Browser on its own if you only need to browse the web. And as an extra advantage, you will be able to browse websites using only one hand! no need for two hand navigation gestures anymore!
New AR Web Browsers can be launched from within the same AR-Pinboard App, and the browsers can also be permanently anchored to any desired places.
This platform will allow AR-Pinboard users to manage their AR-Pinboards and edit their attributes, such as: Who can see and modify the pinboards? Can the pinboards be moved? Copied? Or cloned (synchronized copies)?
AR-Pinboards are extremely flexible tools that can be used in lots of ways. As well as managing your AR-Pinboards, the main goal of the AR-Pinboard Web Platform will be to foster the growth of a community where users can share ideas on how to use their AR-Pinboards. The platform will feature a blog showing news and articles about creative uses of AR-Pinboards and a forum for users to discuss and exchange ideas about how to use AR-Pinboards. The AR-Pinboard Web Platform will incorporate other common social media tools, such as comments on pinboards, likes, and messaging.
The AR-Pinboard Web Platform can be accessed from any web browser, including the AR Web Browser! Below you can see a mock-up of what a user profile may look like in the AR-Pinboard Web Platform:
The AR-Pinboard product is patent pending.
AR-Pinboard is capable of combining some of the advantages of physical pinboards (it can be placed at specific locations) and all the advantages of digital pinboards (easy to update and flexible use). In particular:
- You can easily place any number of AR-Pinboards anywhere. For example, one at work, one at home, one in each room, outdoors, etc. It would quickly get tedious and expensive to have so many physical pinboards.
- You can easily share the AR-Pinboards with a selected group of people. A physical pinboard is always public.
- You can update the contents of an AR-Pinboard from a distance.
- You can easily enable/disable groups of pinboards at will, or schedule the enabling/disabling of groups of pinboards to happen at certain times of the day.
You can easily drag&drop
contents from the web into an AR-Pinboard.
- You can have multimedia and interactive widgets inside an AR-Pinboard.
- Contents on an AR-Pinboard may have an expiry date, after which time they disappear.
- Keep the AR-Pinboard contents and locations saved in the cloud.
- Keep a history of the contents on the AR-Pinboards.
In addition to the applications described in the main video, some more applications of AR-Pinboards can be:
- Easily attach information to items in a store, museum, gallery, crime scene, etc.
- Treasure hunts for children or adults.
- Show location information for tourist tours.
- Family scrapbook that children can easily update remotely while they are on holidays.
- Leave messages to strangers.
- Enable collaboration of geographically separated teams.
- Show instructions about how to operate a machine on the machine itself.
- Educational aid in schools, for example showing information about a subject in an AR-Pinboard that a student can take home and check at a later time.
- AR-Pinboards can contain widgets to control the internet connected devices at home.
- Easily announce offers or events yourself by pinning an AR-Pinboard outside your shop.
- Test how paintings or other items would look like on a wall or shelf.
The following video shows some example uses of AR-Pinboards in retail applications, for more details check update #6:
The AR-Pinboard App and the AR-Pinboard Web Platform will be free to use for regular users. The AR Web Browser will be a paid feature.
You can insert items from the web into the AR-Pinboard even if you don't have the AR Web Browser feature, but you will first have to download them yourself to your smartphone or smartglasses using any other web browser. This means you'll have to switch to another App to download the items. This is very laborious compared to using an AR Web Browser, so the AR Web Browser is highly recommended!
The work plan extends for 10 months starting in November 2014:
I will be delivering progress updates to all backers during these 10 months. A detailed description of the work packages (for the interested techies) is available in the FAQ.
Ten months will be required to finish and launch AR-Pinboard for Android (Goal 1). If Goal 2 is reached, in a further three months the iOS version will be released.
Note: I have removed the Layers Extension stretch goal to save space.
The final goal of AR-Pinboard is to be used with smartglasses. The same AR engine (Owl Engine) used for smartphones and tablets could be used for smartglasses (such as Google Glass, Meta Glasses or Moverio Glasses) with little or no modification. The AR engine just requires a colour camera and motion sensors. At the moment there are a few dozen smartglasses being developed by various companies all over the world, but there is no clear winner. I think it's a little early to focus on any particular smartglasses, but in a year's time this may be more clear. At the moment, it makes more sense to begin with smartphones and tablets because there are a lot more than smartglasses and because the most used platforms are well established. Aiming first at smartglasses would also be more expensive and there would be a lot less users. For these reasons, I prefer to focus on smartphones and tablets first, and when the AR-Pinboard platform is stable then it'll be the time to include smartglasses. It'll be cheaper and easier to do it this way.
The interaction with AR-Pinboards may vary depending on the specific smartglasses. Possible methods of interaction include hand or finger tracking (using the smartglasses camera), use of hardware pointers or control pads (if included with the smartglass), use of your smartphone touchscreen (wirelessly connected with the smartglasses) to control a pointer in the field of view, or just use voice commands. Building and testing this interaction will take some time. Various interaction methods will be included to accommodate for different smartglasses hardware and different user preferences.
A patent pending method of interaction with smartglasses that I plan to include in AR-Pinboard is to use the smartglasses for visualisation in combination with a smartphone for interaction. In this method the smartphone will work as a 3D pointer, with haptic feedback included!
This method of interaction can be achieved by running the Owl Engine both in the smartphone and in the smartglasses, and sharing the same map of the scene between both of them. Then both smartphone and smartglasses can operate within the same coordinate system.
The smartglasses version of AR-Pinboard will come soon after the smartphone and tablet version. And smartphones, tablets and smartglasses will be able to access the same AR-Pinboard platform and share the same AR-Pinboards between them.
From as little as £1 any pledges are very welcome and I'm very grateful to you for helping make the AR-Pinboard project a reality. All backers will have the option to appear in the AR-Pinboard of Fame. This is our own version of the “Wall of Fame”. Whenever an AR-Pinboard user (backer or not) aims their smartphone, tablet or smartglasses to the AR-Pinboard logo (which still needs to be designed) the logo will be recognised and an AR-Pinboard of Fame will appear floating on top of the AR-Pinboard logo. See the figure below to get an idea:
The AR-Pinboard of Fame will list all the Kickstarter backers in order of their backer's score. The backer's score will be calculated using the backer's pledged amount followed by their pledge order of arrival. Higher pledges will receive a higher score and earlier pledges will receive a higher score within the pledged amount section. You will be able to easily see who were the first backers within each pledge amount section as well as the general classification. All backers will receive an individual code that, when entered on their AR-Pinboard profile will enable extra reward functionalities, including causing the AR-Pinboard of Fame to appear centred on their name. The AR-Pinboard of Fame will show different information depending on the size of the pledges. As a minimum, it will show the backer's position, score and the full name of the backer.
The final form of the AR-Pinboard of Fame still has to be decided. It could be of any shape and size, even sizes much larger than the camera's field of view (the mapping and tracking capabilities of the AR engine can allow for this). I will put effort on making an AR-Pinboard of Fame that will make backers proud.
In gratitude to backers, the AR-Pinboard of Fame will be a feature that will stay with AR-Pinboard for as long as AR-Pinboard exists.
All backers will receive an exclusive code that when entered in their profile it will give them access to various rewards depending on the size of their pledge.
Thank you very much for helping make AR-Pinboard a reality!
How will the £60,000 (~$98,000 ~€75,000) be used?
Discounting the Kickstarter+Amazon fees and various other expenses, the largest expense is salaries.
Where am I at?
Can I deliver?
Detailed Work Packages
T1. Finish AR Engine (1st Nov 2014 - 31st Mar 2015)
This will involve:
- Integrating motion sensors into the current tracking algorithms.
- Speeding up the image processing pipeline.
I have a good idea about how to do all this, I just need the time to do it.
T2. AR-Pinboard App development (5th Jan 2015 - 31st Jul 2015)
This will involve developing the Android App, and the iOS App if the stretch goal is reached. The AR engine is written in C++ and will be reused in both platforms. The App will have a C++ core, with some OpenGL, and this core will be wrapped with Java (for Android) and Objective C or Swift (for iOS).
T3. AR-Pinboard Web Platform and API development (19th Jan 2015 - 31st Aug 2015)
This will involve developing the web platform and the API that the Apps will use to communicate with the platform. This will be developed in Django/Python (I'm a Pythonista myself, I have prototyped all the Computer Vision part of AR-Pinboard in Python first and I want to be able to play around with the Web platform and API too even if it's not my main task)
T4. Alpha/Beta testing (1st May 2015 - 31st Aug 2015)
Everybody in the team will be involved in the Alpha/Beta testing, and you can too if you wish to collaborate!
will I operate AR-Pinboard?
The steps to create and place a new AR-Pinboard will be:
- Select create new pinboard
- Aim the smartphone to the place where you want the AR-Pinboard to be (anchor position).
- Tap the screen.
- That's it!! The AR-Pinboard will remain fixed on its anchor position, and now you can start inserting items to your new AR-Pinboard!!
The AR-Pinboards will be created with default attributes. Then, you will be able to interactively change the AR-Pinboard attributes:
- Name of the AR-Pinboard
- Select who can see and modify the pinboard (from your list of contacts). It could be a private pinboard too just for you to keep your own notes attached to places!
- Can it be moved? Copied? or Cloned (synchronized copies)?
- Customise other pinboard features, such as size, shape, material, border, etc.
To interact with an AR-Pinboard:
- Just aim the smartphone towards the place where the AR-Pinboard is anchored.
- Assuming that that is an AR-Pinboard for which you have access to, the AR-Pinboard will then appear on the smartphone screen as seen from the smartphone's point of view.
- The user is then free to see or interact with any items inside the pinboard.
The types of items that will be possible to insert on an AR-Pinboard include:
- Text notes
- Photos (from the camera, storage card, or from the internet!)
- Freehand drawing.
Photos and Text notes will be available from the first release of AR-Pinboard. Videos and freehand drawing will come with a later version.
Text will be entered using a unique AR Keyboard that will appear floating, at a predetermined (adjustable) distance between the user and the AR-Pinboard, whenever the user taps on a text note. This AR Keyboard will be fantastic for “fat fingers” issues on smartphones.
Items on the AR-Pinboard can be rearranged within the pinboard by just tapping on each individual item and dragging it somewhere else within the pinboard (2D rearrangement). AR-Pinboards will also have horizontal and vertical scrolling if this attribute is selected.
Users will be able to set the attributes of an AR-Pinboard so that it can be moveable or fixed. If moveable, an AR-Pinboard can be dragged&dropped to a new location, or be selected from a list of pinboards and be re-anchored to wherever the user happens to be. If the AR-Pinboard is moveable, then it will also be possible to completely operate the pinboard by just using the smartphone touch-screen, this can be useful to operate the AR-Pinboards in the dark.
If desired, users will be able to set-up notifications for updates on shared AR-Pinboards. Also a nearby AR-Pinboard radar (using geolocation) will be available to notify of nearby AR-Pinboards for which the user has access (this feature may not be available in the first release).
How will I know where the AR-Pinboards are?
Colleen McCants, Illustrator - Author of the following illustrations in the main video animation: “Hand holding smartphone”, “Front door”, “Various food items inside the AR-Pinboard on the Front door”, “Home office with desk and bookcase”, “Kitchen with fridge”.
Risks and challenges
Provided we reach the funding goal, then a possible risk will be delays in the development and not reaching the planned release date at end of August 2015. The first work package (completing the AR engine) may have a slightly higher technical risk of lasting longer than planned compared to the other two development work packages. If the desired performance/robustness for the AR engine is not reached by the end of the first 5 months, the completion of this work package may be delayed. However, this is mitigated by the fact that this work package is at the beginning of the time line, which means that this work can safely overlap with the development of the App and Web platform. In general, I think that the 10 month schedule is very reasonable and achievable. I will be doing everything in my power to meet the schedule, and I will be sending progress updates.
Another possible risk is finding the right mobile developer and web developer that are experts for the technologies I want to use. I have the first 3 months to find these two people so that they can start in January 2015. I hope that the awareness created by this Kickstarter project will also attract interested developers with the right skills and that are located, or willing to relocate, near Manchester, UK. As I mentioned above, I also have a grant from the Technology Strategy Board in the UK that, if I'm able to use it, will add £48K more to the budget so that may help to attract the best talent.Learn about accountability on Kickstarter
- (30 days)