Update: Lighting talk and slides from Bay Area Lisp Meetup
(Correction: a byte port advances 8 bits, not 8 bytes.)
What is the 3L project?
The 3L project is building an operating system from the ground up to be secure and extensible.
With 3L you can download an untrusted program from the internet and run it safely.
Anything in 3L can be modified while in use. This means software updates will be easy and seamless. And if you are a programmer, it means development and experimentation will be much easier.
3L also brings many other innovations and improvements as well. One example is that 3L can save the entire system as you are using it. This means that if the power gets cut or the battery dies you can (instantly) reboot the system and it will be like you never lost power. All your apps are running and with the same content they had before.
It might seem like our current operating systems could be modified to support these features but they practically can't.
The operating systems we use today were initially developed when hardware was much more limiting and security was at most an afterthought. There is potential for operating systems to be much more than they are today but they are held back by outdated architectures. While an OS that we have today could potentially be retro-fitted with a better architecture it would break all the programs that run on it and everything would have to be rewritten. Since that is the case we might as well start clean and build something designed for the present and the future.
We desperately need a more secure operating system. As more and more systems are digitized and more money is involved in computer systems, security will become more and more paramount. The only way to have a truly secure operating system is to create one that is secure from the ground up and today's operating systems are anything but.
3L is built on a simple security principle: instead of programs choosing what they have access to, the user chooses. When you run a program you get to choose if it has access to the network or your printer. It doesn't matter what the program tries to access it will only succeed if you allow it.
3L is also extensible from the ground up. Anything can be modified, added, or removed while the system is running. The 3L system is built on a very dynamic programming language (Lisp) that makes this easy and natural.
This is a very large and long term project. This first round of funding is to just take the current working proof-of-concept and turn it in to an open source project and something that one can play around with and start developing applications and drivers for.
3L is a general purpose operating system; it is for everyone. It can be tailored to run on many different devices and for varied scenarios from mobile devices to hardened servers.
3L is also free. It does not cost anything but it will also be run as a fully open source project for anyone to contribute to and the 3L project will be a non-profit with the purpose of serving it's users, not making money off of them.
3L is currently a proof-of-concept more than anything else. It will run by itself on "bare metal". It will also run simple programs and supports the development of drivers although it does not yet contain any major hardware support. (A driver is a piece of software that allows the system to interface with things like printers and web cameras.) This round of funding is to develop a more polished "alpha" version that can be more widely used. A more developed system with things like networking and graphical display support is still a long ways off.
This kickstarter is to get the ball rolling on creating the OS of the future. The money raised will support 12 months of further development on the OS by me, Thomas Hintz. It is hoped that by the end of that period more money can be raised to get more people involved in development.
The goal by the end of the 12 months is to have an alpha version available that people can run. It would at least: implement the basic security model, include a full Scheme runtime, allow for interactive development and introspection, and include a basic ACPI driver to load the kernel and save and load data from disk. If things go quickly it could also include more drivers, an editor, and more polish in general.
3L is really just a single Lisp program. There is only one address space and one runtime that everything else runs inside of. We can do this securely because the language runtime provides and enforces "first class environments". An environment is an object containing references to resources available in the system. It's like running a program but the program doesn't choose what libraries or hardware resources it has access to. Instead programs are run inside their own environment that provides bindings to things it is allowed to use. If you don't want a program to access the network then run it inside of an environment that has no network bindings. If the program calls a function that interacts with the network that function won't be defined in its environment and an exception will be triggered. This mechanism provides for a very robust, fine-grained, comprehensive, and simple security system.
At its core 3L is a R7RS Scheme runtime. It provides all relevant portions of the R7RS Scheme standard as well facilities to manage first class environments and hardware access. The runtime also provides facilities for hardware and memory access, debugging, introspection, first class environments, and first class source code and documentation manipulation.
- Interactive, iterative development of the entire OS
You can boot into a REPL (Read Evaluate Print Loop) and can inspect or rewrite any part of the OS as you are using it.
- Dead simple and strong security
Whenever you run a program you tell it what it has access to, it doesn't get to choose. And every operation is type checked and bounds checked removing many common security vulnerabilities.
- Command driven
Just like with emacs, even the GUI, desktop environment, or window manager can be controlled with commands and functions.
Find a critical bug with an OS function while writing your app? Just put your cursor over the buggy OS function or enter it into the console and your editor will jump you to the source code for it. Fix the function and viola, every program that uses that function instantly gets the fixed version, no need to even recompile or reload anything.
Insert a breakpoint or trace any part of the OS while using it. If an error occurs you can catch it, fix it, and restart execution to just before the error occurred.
Documentation is provided for the entire system in a first class fashion. (Meaning you can read and change the documentation in your programs.) Along with a comprehensive document explorer.
- Fully Source Mapped
At runtime the user can view and modify the source of any component from the REPL, an editor, or in a program itself.
- Symmetric Multi-Processing
3L is designed from the ground up to utilize multiple processors.
3L only supports 64-bit architectures, starting with x86_64. 3L is for the future, not the past.
Why (in detail)?
Why rewrite the OS?
The structure and interface of OSes in use today are based mostly on the state of computer science 45 years ago and much more research has been done since then that improves dramatically upon our knowledge at that time. Hardware is also not as limiting of a factor now and is likely to continue improving. The concepts behind this OS is the result of decades of research and development. While implementations of OSes has improved considerably since the first Unix systems were created the fundamental design has changed little. This is not limited to Unix based systems either. All major OSes are built with the same underlying model of an OS kernel being separate from a language runtime and user programs. And they generally use the same tacked-on security measures like user privileges and access-control-lists.
The OSes of today aren't going away for a long, long time but that doesn't mean they are the end-all-and-be-all of the OS. Instead we need to look to the future and build a system based on our new knowledge and understanding.
Why not just improve the OSes we have now? That would be ideal but they are so fundamentally far away from where we need to go that it would have the same net result as starting from scratch, except take longer. That being said, not all is lost. It is still possible to port programs over but how easy that is will depend greatly on how high level they are and how well the language they are written in can deal with the module system and security model. Emulation layers can also be created but should be avoided since they will require an almost unrestricted access to all hardware and system libraries which would completely ruin the improved security infrastructure.
Dropping the filesystem
The filesystem is not the best way to store every type of data. 3L instead just provides a simple key-value storage mapping. A key is a label that maps to a set of blocks on a storage device. Traditional filesystems can be built on top of this mechanism as simple libraries. Security is provided via libraries and the first class environments. If you want to only allow a program access to a specific set of blocks you can create a function that will only read and write to those blocks and pass that function in to the program you want to utilize it. (A more flexible abstraction is provided in practice.) This means you have 100% control of the storage available. Dropping OS level filesystem control also makes it much easier for things like databases to optimize their read and write operations.
The safety and programmer facilities come a cost but mechanisms will be provided to over come much of these limitations. You can choose to block compile parts or all of the OS and programs and functions can be integrated and inlined along with many other optimizations enabled by block compilation. One major performance issue that current OSes deal with is interrupt handling which requires a lot of memory manipulation and processing power; 3L does not have to deal with that issue. With those performance enhancements 3L could likely achieve 80-90% the speed of current OSes. (Also, see note on 'unsafe operations' below for another performance improvement possibility.)
3L does not architecturally prevent an extension that would allow programs to directly access the memory or run "native" code. It could even be extended to allow programs to run in their own address space. This extension won't be in the default installation but could be provided as a separate version of 3L. This is an option if you want the maximum performance.
Languages Besides Lisp
Scheme itself needs runtime support, like a garbage collector, to manage things like anonymous functions. This means that before the full Scheme can be used something else has to set it up. 3L uses "PreScheme" which is a restricted subset of Scheme that provides low-level memory access. Aside from a few things that must be written in assembly the entire Scheme runtime and OS is written in at least a subset of Scheme. The advantage of this, aside from being easier to develop and more concise, is that a full Scheme system can be used for development, including a debugger and profiler. Of course some things, like drivers, must be emulated but much of the code, like standalone functions and libraries, can still be developed on a full Scheme runtime. This greatly speeds up development.
Right now 3L is still very early in development. It currently supports much of the Scheme language but significant parts are still missing. Currently the syntax-rules facility is not finished although getting close and the runtime does already support a lower level explicit renaming macro facility. Much of the rest of R7RS Scheme can be done from the reference implementation in the R7RS standard once syntax-rules is available. A memory access facility is provided but still needs a lot of work.
Next stage of development:
- R7RS features: syntax-rules, dynamic-wind, ports
- More low-level error handling
- Continue work on R7RS port interface for low-level memory access
- Garbage Collector
- Rewrite PreScheme terminal/REPL in 3L's Scheme
- First Class Environments
- Rewrite the assembly ACPI driver BMFS in PreScheme (currently provided by BareMetal OS), which is used to load the kernel from disk
- Basic editor
- key/value storage mechanism
- Disk Drivers
It is planned for 3L to be dual licensed under the 3-clause BSD license and GPL version 2. The goal is for all code to be owned by the organization developing 3L so that other licenses can be added in the future if desired. Currently some code 3L uses is licensed under BSD. Eventually the goal is to rewrite that code under 3L's management organization and license it accordingly or get a copywrite change on that code. PreScheme will also need to be likely rewritten to meets these goals, although it likely needs to be regardless, at some point.
If funding is successful a non-profit organization will be created to facilitate development of 3L. Initially development will be mostly limited to a few core developers, although the source will still be available. When it makes sense the development will be opened to everyone to contribute and will be run in an open manner. It will remain a mostly closed development process to start with for practical reasons. The overhead of maintaining an open project early on will greatly slow down the process and would likely kill any chance of long term success. As soon as possible it will be opened up though. Of course anyone is free to provide feedback or fork the project if is not developing in the manner desired.
Risks and challenges
An operating system is complex and presents many challenges. It is difficult to predict how long one specific component might take to develop. It helps to have a detailed plan but even then one can't know for sure. In a project of this size it is inevitable that some features won't make it in the first release but some features that weren't planned for the first version will certainly make it in. Every effort will be made though to stick with the original as best as possible.
Most of the features planned for the first version are already partially implemented. The biggest challenge will be polishing them off so they are not rife with pitfalls and bugs. To help meet this challenge some new features may be cut to devote more time to polishing off more critical components. This will help keep quality high and provide a base for a healthy project long term.Learn about accountability on Kickstarter
- (15 days)