Apologies that it's been a while since I've written a Kickstarter update. As many of you know, I'm working full-time at System76 now. With less time for Novacut, I decided to forgo the release announcements for the sake of eking out just a bit more development each month.
We've been quietly making big improvements in three critical areas: Data Safety, Security, and Performance.
Dmedia makes the bold promise of "making file management go way" for creative professionals. That's pretty much like promising a unicorn. And if you promise a unicorn... well, you better deliver a unicorn, and your unicorn better work.
As they say, the proof is in the pudding (er, unicorn), and our pudding works:
For every file in your library, Dmedia will try to maintain three known-good copies, each on a different physical drive. This is the equilibrium point toward which Dmedia will always strive.
Think of Dmedia like a marble in bowl. If you pull the marble up the side up the bowl and release it, gravity will pull the marble back down toward the center, and friction will slow the back-and-forth motion till the marble comes to rest.
Dmedia has only three behaviors, or compulsions if you will, that drive its quest for equilibrium:
- Create new copies
- Delete redundant copies
- Verify existing copies
Think of the first behavior (create new copies) as the way Dmedia moves the marble toward the center. Think of the second behavior (delete redundant copies) as what Dmedia does when it overshoots the center.
And for the third behavior (verify existing copies), imagine someone is holding the bowl and tilting it side to side, introducing unpredictable external influences. Dmedia does constant reality checks, and when reality doesn't match its metadata, Dmedia updates its metadata to match reality.
(The "marble in a bowl" metaphor isn't perfect, but hopefully it helps you visualize what Dmedia does.)
Although Dmedia has had all these behaviors turned-on for some time, Dmedia now handles extreme abuse with ease. It can gracefully recover when there are unexpected crashes in any of its background tasks. It has multiple, independent mechanisms by which most data-safety tasks can be accomplished. In general, Dmedia has greatly matured as is almost ready for the main stage.
However, with great data comes great responsibility :D So I'm still not comfortable giving Dmedia the "production ready" seal of approval. We still need some additional UI for communicating certain data-safety-sensitive things with the user. And we also need a way to run automated, multi-node simulations for validating each monthly Dmedia release. Once I declare Dmedia as "production ready", we aim to keep that promise release, after release... after release.
For details on our current manual testing procedures, and our plans for long-running simulation testing, please see Testing Dmedia.
Still, I think it's a great time to give Dmedia a spin and see what all the fuss is about. Just please do so with files that are safely backed up elsewhere.
I strongly feel that passwords aren't a viable authentication mechanism going forward, especially for the so called "Internet of Things".
For example, I think it's a super bad idea to use password authentication to locate and unlock your car from anywhere on the Internet. (In their defense, I think Tesla is an extraordinary company, but they need to up their game here, to match their excellence elsewhere.)
If you want to authorize a new phone to be able to unlock your car, it's reasonable to require you to be physically in your car with said phone. And with this constraint, Tesla could use something like the Firefox Sync device peering, or the Dmedia device peering (which was heavily inspired by FireFox Sync, but Dmedia uses direct P2P communication on the local network instead of mediation through a 3rd party server).
I'd love to see more well-vetted, standard solutions emerge in this space, and that's a big part of why I've split out the internal Dmedia HTTP server into a new library called Degu. Degu includes an HTTP server and a matching HTTP client, and is squarely aimed at implementing REST APIs for device-to-device communication on the local network.
I plan to implement the next generation of the Dmedia peering protocol and identity framework in Degu with the aim of it being generally useful for many types of applications. I hope Degu can be a productive research platform for security and usability in the coming Internet of Things. If you're interested in being involved with this work, or building something atop Degu, please shoot us an email or stop by the #novacut IRC channel.
We've also modernized our SSL configuration (partly thanks to Python 3.4). We're now using TLSv1.2 with ECDH-based perfect-forward-secrecy. And we're now using 4096-bit RSA keys with sha384 signatures for our machine and user certificates (up from 2048-bit RSA keys with sha1 signatures).
Performance is closely related to data safety as the time it takes Dmedia to converge at its equilibrium point largely depends on how well Dmedia uses the available IO capacity. Files must be downloaded from one peer to another, copied from one drive to another, etc. And remember, all this can be happening simultaneously among several devices, so it needs to be done in a way such that different peers don't needlessly create new copies of the same fragile file at the same time.
Yet Dmedia is fundamentally a lock-free and completely distributed system. A surprisingly effective way to keep peers from stepping on each other toes has simply been to randomize the order in which fragile files are dealt with, and we use this technique in a number of places.
We've also made big performance improvements when it comes to our metadata layer. Much of this improvement is thanks to the Degu HTTP client, which is a fair bit faster than the http.client module in the Python3 standard library.
Degu 0.7 also brings the first step in replacing the common HTTP parser and IO abstractions shared between the server and client with a high-performance C extension.
For example, this was edited in Novacut, finished in Blender:
Dmedia (and Novacut) are far more compelling if the underlying Dmedia platform can be used across a broad ecosystem of applications.
Install Novacut 14.07
To install Novacut on Ubuntu 14.04, just open a terminal and run these three commands:sudo apt-add-repository ppa:novacut/stable sudo apt-get update sudo apt-get install novacut
If you want to help develop Novacut, it's best to install from ppa:novacut/daily.
Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. For more details on the PPAs, read about our Monthly Release Process.
You can download the source code from each component's Launchpad project page: