WARP Mechanics brings true parallel computing home
What is the MicroPod HPC project all about?
The MicroPod HPC is a parallel computer that you can afford to use at home. You can "stand up" a parallel computer using inexpensive commodity hardware, or even use the images as VMs to run a completely virtual development environment.
Once the project is funded and completed, the software will be hosted here:
Aside from personal use for learning and fun, it can also help companies, universities, and government organizations that are developing code for full scale parallel computers by giving them an easy and affordable R&D platform.
Why is this important?
Completing this project will dramatically expand the number of people who have access to basic parallel computing systems, which in turn will expand the number of people who know how to program and operate these systems. That, in turn, will allow more super computers to be built.
This is important. The rate of scientific progress world wide is largely limited by the number and speed of super computers that scientists can access. All of the "big science" problems these days have to be modeled and analyzed by such machines, and there just aren't enough to go around.
What should I do?
If you think that accelerating scientific progress is an important issue, or just think this is a cool nerdy project, the obvious answer is "pledge your support". Even if it's just buying a T-Shirt, every bit helps.
Other cool T-shirt options include:
- I had a life once… Now I have a super computer
- I don’t byte… I MicroPod
- I have a WARPed sense of (super) comphumor
- MicroPod HPC: simple enough for a n00b; power enough for a nerd!
- I'm not a geek - I'm super-computer literate.
- Most people fold jeans on their washroom counter. I fold genes on my WARProom super computer.
But just as importantly as contributing yourself, please send a link to this project to any cool nerdy friends you may have. Particularly university students, who might benefit from the project deliverables. If we can get a large group of people engaged in this, we will have an immediate community of people to start using MicroPods once the project is funded.
Plus, university computer science students can always use more T-Shirts. :-)
Why is WARP Mechanics doing this?
WARP Mechanics works on full scale HPC systems, so why would we execute a project focused on a commodity market?
We keep running into situations where our customers want HPC systems, and have budget to buy them, but can't find enough qualified staff to run them. The lack of staff is constraining the growth of the full scale HPC market.
Of course, WARP would like the market to expand so we can sell more full scale super computing systems. :-) But we also, as citizens of the world, want those computers to exist whether we get paid for them or not. Super computers are driving most of the world's scientific and engineering progress these days, and not having parallel computers to run these jobs is constraining how quickly scientists can improve our world.
How will the project money be spent?
The world doesn't need more commodity hardware to solve this problem. WARP will be spending some of the money to buy hardware, but only enough to deliver the rewards, and run our software development effort. We will not be developing any new hardware of our own.
The goal of the MicroPod HPC project is to leverage the low cost hardware that already exists, and provide easy to install and easy to use software sitting on top of it. It is the software layer that makes parallel computing complicated, so this is the area where value really needs to be added.
In fact, customers who have access to virtualized environments will be able to use the project deliverables without any hardware at all.
What exactly will WARP Mechanics deliver? First, you have to understand a little bit about what a parallel computer is.
Conceptually, in the WARP Mechanics architecture, a parallel computer consists of three major components: compute nodes that run users' jobs, one or more nodes that control the cluster, and nodes that store the cluster's data. (There are some diagrams below.)
Each of these components performs a very different job, and thus has very different software: different applications, utilities, and even kernels. And the software is complex. In a full-scale HPC environment, there are usually different staff specialists required to operate the software in each of these components.
New to parallel computing? This project is a great resource to learn about how these systems work: http://www.oscer.ou.edu/education.php
WARP Mechanics will create turnkey code for all three components which can be deployed in minutes on existing commodity hardware or as VMs. That turnkey HPC software is, in short, is the value of the project.
This is going to involve considerable effort on our part. It is expected that custom software will be developed to facilitate installation and management of other packages. Research will have to be performed into compatibility, and code changes may be required to existing GPL packages to accommodate that. Any other GPL packages used will be documented and placed into pre-configured images which can be flashed onto bootable media, to make roll-out a snap. Everything specific to the MicroPod HPC - software and documentation alike - will be released under GPL.
But at the end of the day, the deliverable is simple:
You, the Kickstarter community, get three fully-integrated, turnkey software images: one for each of the three parallel computer components. Each will already have a complete set of HPC software pre-installed and pre-configured. You will also get a quick start guide showing how to get it up and running, and a manual showing how to operate and customize it. Finally, you will get documents showing how each image was created, in case you want to customize things to a larger extent.
What software will be included in these images?
In short, everything needed to make a MicroPod work as a parallel computer.
- Custom kernel based on CentOS 6.3 or similar
- Parallel filesystem based on Lustre
- Job submission and control software
- Cluster administration software
- Example programs
Each of these bullets and more have too many sub-components to list, and even more associated tasks that super computer architects and administrators normally have to perform to make the system work... which is really why this project is needed.
For instance, all three delivered images will have Lustre kernels and tools installed. Lustre is the premiere parallel file system for HPC: a vast percentage of the world's fastest computers use this storage method. It is pointless to try to learn to program for a parallel computer if it doesn't have a parallel filesystem, as your software will work completely differently once it moves to the real world.
Lustre is fast and scalable, but it is very difficult to get it working. It can take months just to start using Lustre if you aren't already an expert. WARP Mechanics already has this expertise, so we will port our existing Lustre solution to the MicroPod image.
These images can be copied directly onto hard drives or USB sticks, connected to supported commodity hardware, wired up according to a provided diagram... and you will immediately have a small parallel computer. Or you can just implement that diagram with virtual machines, or even cloud services!
So rather than exhaustively list every package that is included in a parallel computer, let's phrase it this way: WARP Mechanics will deliver sufficient bundled software to produce a working parallel computer, with a collection of tools comparable to existing parallel computers... just at a much more affordable scale.
If you pledge at a moderate level, you will get this software delivered on bootable media, or even pre-installed and tested for you on supported hardware. If you pledge at a lower level, you will still get access to the software and documentation.
What will not be included?
Parallel computers can be customized in an almost unlimited number of ways. Different job submission software, different load balancing algorithms, different management tools... Obviously WARP Mechanics cannot provide every possible combination of software. So:
The system will be immediately usable as a parallel computer. It will also be customizable by the user. If you want to download a different piece of HPC software to put onto a MicroPod, you're free to do so. We will have installed "a" job control package; we will not have installed "every" job control package. The operating systems will all be based on CentOS 6.3 (or a similar RHEL derived OS depending on release timing), and virtually all open source HPC software packages are available for that OS series.
In short, you do get a rapid path to having a productive parallel computer test bed or learning platform, which you can then modify or add to as you see fit. You do not get a parallel computer which has already been modified in every conceivable way.
How long will this take?
As shown in the delivery timeline for the rewards, the project will be completed by June 2013. It isn't possible to narrow it down to the exact day, but the general timeframe is solid based on WARP Mechanics extensive experience packaging code for other parallel computer hardware architectures.
This is a big project. Why does WARP Mechanics think it can do this?
WARP Mechanics Ltd. is well-qualified to deliver this because the firm has been working on full-scale super computers for years.
For example, the latest edition of WARP Mechanics PetaPod system (first announced in November 2012) contains over 2000 CPU cores, two Petabytes of high-speed storage, and over 50 Terabytes of RAM, all running over 40 gigabit Ethernet network infrastructure.
This is analogous to the work that WARP Mechanics did on Stanford University's new BioX-3 system: That system uses WARP storage products and commodity servers. (BioX-3 is the latest Stanford super computer, and has superseded the BioX-2 system as of January 2013.)
The MicroPod HPC project will leverage WARP Mechanics' expertise in this area, and place a true super computing architecture within reach based on much more affordable hardware.
We also have an extensive track record of contributing to open source software. Because WARP Mechanics has OEM relationships with major technology corporations, our contributions are generally uploaded by them. For instance, our fixes and enhancements to ZFS are delivered via Nexenta, and our Lustre innovations are delivered via our OEM relationship with Intel (formerly WhamCloud).
What is a super computer?
There is no specific size or speed that makes a system "super".
One way of looking at it is that a super computer embodies the pinnacle of modern computer science techniques. These techniques can be applied in a variety of different scales to suit different budgets and computational power requirements.
Today, super computing is better known as "High Performance Computing", or HPC. But the principal is the same: an HPC system is architected using whatever design principals currently produce the best results to meet the scale and budget in question.
It may be helpful to explain something that isn't a super computer. A single chip with many cores does have a parallel processing element to it, but its fundamental scalability is still limited to one chip. A modern super computer is a networked cluster of full-scale systems, each of which contains many chips, which is why it can dramatically exceed the performance of even the fastest chip. This, today, is the technique which differentiates a true parallel super computer from a mere "somewhat faster chip".
The fastest super computers at any point in time should be the fastest computational devices in the world. While most super computers are not in that top bracket, they are still vastly faster than systems which most users - or even most corporations - can access. This is because they use the architecture of the fastest computers. It is this parallel clustered/networked architecture that the MicroPod HPC brings to the masses.
This diagram is core to understanding the project. WARP Mechanics is promising to deliver all software needed to make this diagram work. At the higher levels of contribution, WARP will deliver that software on commodity hardware matching this diagram.
Why would somebody want that at home?
There are many useful things you can do with a personal super computer, which aren't practical any other way.
A modern parallel computer allows a programmer to spread out complex tasks across many individual systems. No matter how powerful a single CPU or chip gets, a parallel computer will always be many times more powerful since it can leverage 10s, 100s, 1000s... potentially even millions of the fastest chips to aggregate their power.
So what could somebody do with a personal super computer? Ultimately, that is up to your creativity. But here are just a few ideas to get started:
Learn to program for parallel systems
Simply writing software for one chip, no matter how advanced it is or how many cores it contains, does not qualify somebody to write software for a real parallel super computer. There have been some unfortunate claims along those lines lately. But programmers need to take into account system level locality, network delay, storage performance, and many other issues which cannot be learned using a single system with a proprietary chip, or even multiple such systems with proprietary interconnects. Using a system that follows a real super computer design plan, regardless of its computational speed, will allow programmers to understand how the full system truly works when developing their code.
Learn to deploy and administer parallel systems
As super computers become more prevalent in businesses of all sizes, the world will need systems administrators and engineers to work on them. Right now, the talent pool is limited, yet the demand is growing exponentially. By proliferating MicroPod HPC systems to computer science students, WARP Mechanics will be helping the IT industry as a whole to create relevant skill sets to keep up with that demand.
Run parallel software
A large and growing ecosystem of software exists for parallel computers, which isn't practical to run any other way. Even software which can be run on a single computer may be able to run better on a parallel computer.
By way of example, let's say you want to run Stanford University's Folding@Home. If you run it on one system, you will be contributing to the project to the extent of one CPU. But if you are running an 8-node MicroPod HPC and execute the same software, your contribution will increase by that same factor. This is an example of how you could put your MicroPod to productive use even when you aren't using it yourself!
Develop parallel applications for commercial use
The market for parallel applications is skyrocketing. But there is a stiff barrier to entry for companies which want to develop software for these systems. Just writing commercial code on a single multi-core chip will not suffice, for the same reason that it doesn't suffice as a learning tool. But buying a super computer is far beyond the reach of most software development firms.
Using a MicroPod HPC, you can write code and test it in a much more realistic environment before trying it out on a full-scale system, where time is absolutely equal to money. This architecture is low enough cost that firms could provide a dedicated HPC test system to every software engineer!
More ideas of software you could write?
- Ray tracing or rendering farm
- Do your own weather modeling
- Host your own multiplayer online game
- Bitcoin harvesting
- Video repository or torrent cluster with enough power to serve a city block
- Create your own analytics system to examine twitter feeds
- Powerful home automation systems
Who gets to play with super computers today?
Super computing has always been the domain of major governments, top-tier universities, and Fortune 100 companies.
Historically, super computers were built by a handful of major corporations such as IBM and Cray. They would use exclusively proprietary hardware, and equally vendor-locked software.
Today, much has changed in the HPC space. Most of the required software is now open source, and will run on open hardware platforms. Yet it is still impossible for typical users to get access to this class of computing technology. Even with the core intellectual property being freeware or commodity hardware, it still takes millions, tens of millions, or even hundreds of millions to get an HPC system running.
WARP aims to change that with the MicroPod HPC Project. This project will allow a typical small business, or even a home user, to construct their own parallel processing super computer... albeit on a much smaller scale.
Why is this a good idea?
Simulating aircraft designs to reduce plane crashes, creating more accurate climate change forecasts, running software to predict the likely results of various economic models before testing them in the real economy... There are a massive number of ways that faster computers can help the world as a whole.
Yet it isn't possible to scale single chip designs to solve these modern computational challenges. At some point, you have to create scale by aggregating systems of chips, rather than just packing in a few more cores per chip.
The good news is that HPC does this. Parallel computing is a "solved problem". You can buy a full scale parallel solution from WARP Mechanics today, and get started on solving those looming engineering, scientific, social, and economic issues.
But the parallel processing world is short on talent. There aren't enough programmers, engineers, administrators, or architects. Even though HPC systems are getting more common every day, it is getting harder and harder to find personnel to work on them.
"For want of a nail, the shoe was lost," goes the old story. Being nerds, and therefore science fiction fans, the WARP Mechanics team looks at the "talent gap" in super computing like this:
In 2015, the world fails to stop a pandemic, and millions die. Why? Because the world lacks accurate simulations of viral oubreaks and thus the capability to simulate the results of vaccination strategies... because it has a shortage of HPC systems... because it lacks systems administrators for HPC systems... because they had no way of learning their trade without already having access to multi-million-dollar super computers.
This example may seem dire, but this - or something like it - is a likely result of society lacking HPC experts. WARP Mechanics believes that proliferating HPC knowledge to all levels of computer enthusiast is the best way to ensure that the world's forthcoming hyper-scale simulation/modeling systems can be built and managed competently.
Besides. Running your own super computer is FUN!!! :-)
Why is WARP the right firm to do this?
WARP's mission statement is to bring super computing technologies into broader IT markets. And the MicroPod couldn't be a more extreme example of that.
WARP Mechanics has been building HPC systems for years. In 2012, WARP released its "PetaPod HPC" system in partnership with Hyve Solutions. (Hyve is the firm which makes the entire server layer for Facebook, for example: their credibility in "Hyper Scale" compute architectures cannot be beaten.)
The MicroPod is a scaling down of that already-established architecture.
And - let's be honest about how important this is - WARP Mechanics personnel like working on HPC systems. We don't do this because we have to. We do it because we are true nerds at heart - Just as, we hope, some of you are too. If so, please consider making at least a small donation to this super cool super computing project.
Risks and challenges Learn about accountability on Kickstarter
Because WARP Mechanics builds full-scale HPC systems today, which already run on Intel hardware, this is a less risky project than you might think. We will still incur substantial costs to port our software to consumer grade hardware, but we have done lots of down-scaling projects like this before.
Risks? It may prove difficult to find sufficiently inexpensive broadly-available parts which can nevertheless run our code. If this happens, WARP may need to slip the delivery date in order to acquire hardware from overseas sources, or find a bulk supplier domestically. WARP does have these relationships, so the major risk is time rather than e.g. a complete failure to execute.
It is also possible that a component vendor might go out of business or discontinue their products while the project is ongoing. If this happens after software has already been ported to that hardware, WARP will find a substitute vendor and re-test the code, but that too might cause schedule slip.
Conceivably, some element of the WARP Mechanics software stack might be found to infringe a patent. This seems unlikely as variations on the same software stack is already used in most of the world's fastest parallel computers. But if that happens, WARP will, again, need to substitute different code. Again, the most likely result is delay rather than project failure.
Of course, there is always the chance of major disasters. If the west coast of the United States experiences such a disaster, it could preclude completion of the project. The unexpected death of key WARP Mechanics personnel in such a scenario - or due to any other cause, for that matter - would clearly create risk of project failure.
To mitigate this risk, WARP Mechanics will attempt to distribute the project tasks to team members in different geographical areas, to provide the best chance for completion even if a worst case scenario occurs. But this level of risk cannot be managed completely, and should therefore be acknowledged as having the potential for causing project collapse.
Have a question? If the info above doesn't help, you can ask the project creator directly.
seconds to go
Pledge $15 or moreYou selected
SUPPORTER REWARD: Be in the loop! You will receive updates whenever we hit a project milestone. More importantly, you will help us bring super computing technology to the consumer market. Get access to the WARP MicroPod HPC software by FTP download.Estimated delivery:
Pledge $29 or moreYou selected
FUN REWARD: Everything in the SUPPORTER REWARD, plus your choice of "Team WARP" T-shirts. These all have nice graphic designs, and a clever tag line. Select from: "I had a life once… Now I have a super computer" - "I don’t byte… I MicroPod" - "I have a WARPed sense of (super) comphumor" - "MicroPod HPC: simple enough for a n00b; power enough for a nerd!" - "I'm not a geek - I'm super-computer literate." - "Judge not a nerd by his car, but by his super computer" - "Most people fold Jeans on their washroom counter. I fold Genes on my WARProom super computer."Estimated delivery:Ships within the US only
Pledge $49 or moreYou selected
2 backers Limited (998 left of 1000)
FAN REWARD: Everything from the FUN REWARD, but the limited edition T-shirt will be hand numbered and signed by the WARP Mechanics Founders and Board of Directors. A special, personalized message will be written in by WARP Mechanics Chief Technology Officer, Josh Judd. (Author of best-selling nerd books on storage topics; pioneer in the storage networking community.) This reward may be combined with any of the other rewards by making it an additional pledge.Estimated delivery:Add $10 USD to ship outside the US
Pledge $99 or moreYou selected
DIY REWARD: "Do it Yourself" MicroPod fans will love this! It includes the FUN REWARD package, plus a DVD ROM as well as the FTP login to download the complete MicroPod HPC software stack, and (very important!) a white paper explaining in detail how to construct a MicroPod using widely available commodity equipment. You provide the equipment, and install our code on it as a fun project. Build your own super computer! This, essentially, is the main deliverable of the project.Estimated delivery:Add $10 USD to ship outside the US
Pledge $199 or moreYou selected
4 backers Limited (1996 left of 2000)
DIY PLUS REWARD: This adds to the DIY package by including bootable USB sticks for the cluster control node, HPC client nodes, and storage server. You can acquire commodity hardware off of the "supported" list from whatever vendor you prefer, put these sticks into their USB ports, and you'll be up and running in minutes.Estimated delivery:Add $20 USD to ship outside the US
Pledge $699 or moreYou selected
1 backer Limited (49 left of 50)
QUICK START REWARD: Some people will want a head start on the project. This includes everything from the DIY PLUS reward, and includes an Atom-based system with the super computer's cluster control logic pre-installed on an internal HDD. The cluster control node is the really hard part of making a super computer work, so having this done and factory tested by WARP Mechanics' qualified HPC technicians reduces your effort and project risk. You will also have advance access to all of the documentation and software via FTP download, so you can get started on the DIY aspect even sooner than the DIY level backers.Estimated delivery:Add $30 USD to ship outside the US
Pledge $2,999 or moreYou selected
0 backers Limited (4 left of 4)
CONSULTANT REWARD: Receive everything in the QUICK START REWARD, but delivered in person! If you live near San Francisco, this could be the right answer for you. A member of WARP Mechanics senior staff will deliver the reward package to you on location as long as it is within 100 miles of San Francisco. Or, if you are planning to travel here, it can be conducted at a WARP facility. (Your travel cost is not included.) The staffer will conduct a one on one training seminar to explain how the system works. They will then help you get the project started by getting the cluster control node running. If you have already provided the required additional hardware, they will show you how to connect and boot the entire cluster, time permitting. (The overall session is limited to half a day.) Consider that classroom training on this kind of topic normally costs as much even when there are a dozen people in the room, and the value of a 1:1 session with a senior staffer comes into focus!Estimated delivery:
Pledge $3,999 or moreYou selected
0 backers Limited (20 left of 20)
TURN-KEY REWARD: Perhaps you actually want to get started using the thing right away, instead of building it. This package includes the QUICK START items, plus two Atom-based systems to run HPC jobs, an 8TB raw RAID storage array, and the cluster network interconnect switch. The pledge amount is in line with the MSRP of the provided equipment, plus the pledge value of the installed software, so it would cost you about the same amount to build it yourself.Estimated delivery:Add $50 USD to ship outside the US
Pledge $4,999 or moreYou selected
0 backers Limited (4 left of 4)
SUPER FAN REWARD: If you live in the San Francisco area, or are planning to travel here, this might be the reward for you! This includes everything from the TURN-KEY package, plus you will get email support for your project. Then, you get to have a thank you lunch with the CTO of WARP Mechanics and select members of the WARP senior staff and special guests! This is a once in a lifetime chance to talk shop with luminaries in the technology industry. (Lunch is on us; travel costs are not included.)Estimated delivery:
Pledge $9,999 or moreYou selected
0 backers Limited (10 left of 10)
INTEGRATED SYSTEM REWARD: Maybe you want a full-scale system, with the ability to run more complex parallel jobs. Kind of the opposite of a DIY use case. This reward includes everything from TURN-KEY, but adds another storage array and six more Atom-based HPC compute nodes. The total system has a cluster control node, 16TB of disk, and 16 cores of parallel processing power!Estimated delivery:
Pledge $10,000 or moreYou selected
0 backers Limited (4 left of 4)
PRO REWARD: This is not for the faint of heart. Only select this reward if you are planning or currently working on a full-scale HPC project. In addition to the SUPER FAN REWARD items, this includes a half day of custom consulting by WARP Mechanics' CTO and senior staff to discuss how the MicroPod HPC code could be usefully applied to your real-world, full-size super computer. Go nuts on a white board with the folks who invent new super computing technologies! The dollar value of the consulting time alone is worth the price! This can be delivered to within 100 miles of San Francisco.Estimated delivery:
- (21 days)