Share this project

Done

Share this project

Done
Heartbroken?  Fill the void with a pro editor built for HDSLR storytellers.
Heartbroken? Fill the void with a pro editor built for HDSLR storytellers.
802 backers pledged $28,276 to help bring this project to life.

Recent updates

Photos

E93858506ea06c342f1f9c42eb4e7fa8 original
54331006071fa1ab6daffb126b94f188 original
1cef799e151ea6032263aa015931a028 original

Still alive and kickin'

3 likes

Apologies that it's been a while since I've written a Kickstarter update. As many of you know, I'm working full-time at System76 now. With less time for Novacut, I decided to forgo the release announcements for the sake of eking out just a bit more development each month.

But we've still been chugging along, and have continued to deliver our monthly releases like clockwork.

We've been quietly making big improvements in three critical areas: Data Safety, Security, and Performance.

Data Safety

Dmedia makes the bold promise of "making file management go way" for creative professionals. That's pretty much like promising a unicorn. And if you promise a unicorn... well, you better deliver a unicorn, and your unicorn better work.

As they say, the proof is in the pudding (er, unicorn), and our pudding works:

For every file in your library, Dmedia will try to maintain three known-good copies, each on a different physical drive. This is the equilibrium point toward which Dmedia will always strive.

Think of Dmedia like a marble in bowl. If you pull the marble up the side up the bowl and release it, gravity will pull the marble back down toward the center, and friction will slow the back-and-forth motion till the marble comes to rest.

Dmedia has only three behaviors, or compulsions if you will, that drive its quest for equilibrium:

  1. Create new copies
  2. Delete redundant copies
  3. Verify existing copies

Think of the first behavior (create new copies) as the way Dmedia moves the marble toward the center. Think of the second behavior (delete redundant copies) as what Dmedia does when it overshoots the center.

And for the third behavior (verify existing copies), imagine someone is holding the bowl and tilting it side to side, introducing unpredictable external influences. Dmedia does constant reality checks, and when reality doesn't match its metadata, Dmedia updates its metadata to match reality.

(The "marble in a bowl" metaphor isn't perfect, but hopefully it helps you visualize what Dmedia does.)

Although Dmedia has had all these behaviors turned-on for some time, Dmedia now handles extreme abuse with ease. It can gracefully recover when there are unexpected crashes in any of its background tasks. It has multiple, independent mechanisms by which most data-safety tasks can be accomplished. In general, Dmedia has greatly matured as is almost ready for the main stage.

However, with great data comes great responsibility :D So I'm still not comfortable giving Dmedia the "production ready" seal of approval. We still need some additional UI for communicating certain data-safety-sensitive things with the user. And we also need a way to run automated, multi-node simulations for validating each monthly Dmedia release. Once I declare Dmedia as "production ready", we aim to keep that promise release, after release... after release.

For details on our current manual testing procedures, and our plans for long-running simulation testing, please see Testing Dmedia.

Still, I think it's a great time to give Dmedia a spin and see what all the fuss is about. Just please do so with files that are safely backed up elsewhere.

Security

I strongly feel that passwords aren't a viable authentication mechanism going forward, especially for the so called "Internet of Things".

For example, I think it's a super bad idea to use password authentication to locate and unlock your car from anywhere on the Internet. (In their defense, I think Tesla is an extraordinary company, but they need to up their game here, to match their excellence elsewhere.)

If you want to authorize a new phone to be able to unlock your car, it's reasonable to require you to be physically in your car with said phone. And with this constraint, Tesla could use something like the Firefox Sync device peering, or the Dmedia device peering (which was heavily inspired by FireFox Sync, but Dmedia uses direct P2P communication on the local network instead of mediation through a 3rd party server).

I'd love to see more well-vetted, standard solutions emerge in this space, and that's a big part of why I've split out the internal Dmedia HTTP server into a new library called Degu. Degu includes an HTTP server and a matching HTTP client, and is squarely aimed at implementing REST APIs for device-to-device communication on the local network.

I plan to implement the next generation of the Dmedia peering protocol and identity framework in Degu with the aim of it being generally useful for many types of applications. I hope Degu can be a productive research platform for security and usability in the coming Internet of Things. If you're interested in being involved with this work, or building something atop Degu, please shoot us an email or stop by the #novacut IRC channel.

We've also modernized our SSL configuration (partly thanks to Python 3.4). We're now using TLSv1.2 with ECDH-based perfect-forward-secrecy. And we're now using 4096-bit RSA keys with sha384 signatures for our machine and user certificates (up from 2048-bit RSA keys with sha1 signatures).

Performance

Performance is closely related to data safety as the time it takes Dmedia to converge at its equilibrium point largely depends on how well Dmedia uses the available IO capacity. Files must be downloaded from one peer to another, copied from one drive to another, etc. And remember, all this can be happening simultaneously among several devices, so it needs to be done in a way such that different peers don't needlessly create new copies of the same fragile file at the same time.

Yet Dmedia is fundamentally a lock-free and completely distributed system. A surprisingly effective way to keep peers from stepping on each other toes has simply been to randomize the order in which fragile files are dealt with, and we use this technique in a number of places.

We've also made big performance improvements when it comes to our metadata layer. Much of this improvement is thanks to the Degu HTTP client, which is a fair bit faster than the http.client module in the Python3 standard library.

Degu 0.7 also brings the first step in replacing the common HTTP parser and IO abstractions shared between the server and client with a high-performance C extension.

Other news

David Jordan has been making great progress on importing Novacut edits into Blender:

For example, this was edited in Novacut, finished in Blender:

Dmedia (and Novacut) are far more compelling if the underlying Dmedia platform can be used across a broad ecosystem of applications.

Install Novacut 14.07

Packages are available in ppa:novacut/stable for Ubuntu 14.04 LTS.

To install Novacut on Ubuntu 14.04, just open a terminal and run these three commands:

sudo apt-add-repository ppa:novacut/stable sudo apt-get update sudo apt-get install novacut

If you want to help develop Novacut, it's best to install from ppa:novacut/daily.

Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. For more details on the PPAs, read about our Monthly Release Process.

Source code

You can download the source code from each component's Launchpad project page:

Novacut 13.05, all in

This is a very exciting release as all the Dmedia automation behaviors have now been turned on. Although note that we still don't consider Dmedia production ready, so please only test Dmedia with files you have safely backed up elsewhere.

This screencast walks you through installing Novacut on Ubuntu 13.04, and gives you a tour of all the new features:

This is also an exciting release because the Version One Hashing Protocol has been finalized and is now used by default. The only downside is that because V1 uses our Dbase32 encoding, it's not possible to support Version Zero alongside Version One.

This screencast explains how to upgrade your V0 Novacut library to V1:

In a nutshell, if you've used Novacut prior to this month's release, after you install Novacut 13.05 you'll need to open a terminal and run this command to upgrade your library:

novacut-v0-v1-upgrade

As it seems very few have existing Dmedia libraries they care about, we didn't put much effort into this user experience (obviously), but we did put a lot of effort into making sure the upgrade works correctly. However, if anyone out there has problems upgrading or needs help, please email team@novacut.com, and we'll make sure you get your data migrated.

Special Thanks

Thanks to Marco Buono for making this great Novacut intro sequence:

Thanks to saras fox, David Jordan, and James Raymond for helping me test the V0 to V1 upgrade this month.

And thanks to the many people who have patiently reviewed the many protocol iterations leading up to Version One, especially Hagen Fürstenau, Robert von Burg, and David Jordan.

Install Novacut 13.05

Packages are available for Ubuntu 13.04 in ppa:novacut/stable.

If you want to help develop Novacut, it's best to install from ppa:novacut/daily.

Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. So same effect as if you only added the daily PPA. For more details on the PPAs, read about our Monthly Release Process.

Source code

You can download the source code from each component's Launchpad project page:

Novacut 13.02, with caution

This month's release is a bit different. We're in the process of turning on the remaining Dmedia automation behaviors, and switching from the interim Version Zero hashing protocol and schema to Version One.

Please note that we don't yet consider Dmedia "production ready", so please don't yet trust Dmedia as the only place where you store your files. But turning on these remaining automation behaviors means I feel we're almost there.

We want to be very cautious during this transition, and so we recommend that most users skip the Novacut 13.02 and 13.03 releases. To make this more than just a suggestion, 13.02 has only been released for Ubuntu Raring (the Ubuntu development version, which will be released as Ubuntu 13.04 on April 25th).

Most of our PPA users are on Ubuntu 12.04 LTS (Precise) and Ubuntu 12.10 (Quantal), while we have only a handful on Raring. This is a good thing while we validate these changes. We likewise don't recommend our users upgrade to Ubuntu Raring yet... wait till after the stable release on April 25th.

In the mean time, Precise and Quantal users can continue to use the Novacut 13.01 release. You'll be able to migrate your Dmedia library and Novacut edits from V0 to V1 whenever you upgrade to Ubuntu 13.04.

For details on how we'll do the V0 to V1 migration, please see this post.

What's New

After more feedback and testing, I've decided that we are indeed switching to the Dbase32 encoding I proposed last month. The V1 protocol now uses Dbase32 instead of standard RFC-3548 Base32 encoding, although V0 is still the active protocol.

Most of the work this month was in preparing FileStore and Dmedia for the migration. Dropping Precise and Quantal support also meant we could drop support for Python 3.2, and start taking advantage of some of the richer POSIX features available in Python 3.3. There was also a critical fix so that Dmedia downgrades a drive when metadata about the drive is missing.

For details, see the release notes for FileStore 13.02 and Dmedia 13.02.

Special Thanks

Thanks to Simon Wells, saras fox, and James Raymond for diligently reviewing my many merge proposals.

Thanks to Robert von Burg for continuing to provide feedback on the Dbase32 encoding and the V1 Protocol Specification.

And thanks to David Jordan for his work on porting Graffik to Ubuntu, which is Dynamic Perception's open source control app for nanoMoCo, an open hardware stepper motor controller:

Plus David had a chance to play with the Ubuntu Touch SDK:

As always, if you want to get involved with Novacut design or development, please stop by the #novacut IRC channel on freenode and introduce yourself.

Install Novacut 13.02

Packages are available for Ubuntu Raring in ppa:novacut/stable. But remember, we recommend that most users skip the Novacut 13.02 and 13.03 releases.

If you want to help develop Novacut, it's best to install from ppa:novacut/daily.

Note if you've added both the daily and the stable PPAs, the versions in the daily PPA will supersede the stable versions. So same effect as if you only added the daily PPA. For more details on the PPAs, read about our Monthly Release Process.

Source code

You can download the source code from each component's Launchpad project page:

13.01, first release of the new year

What's New

Thanks to Simon Wells, the Dmedia browser got a nice makeover and now displays useful metadata:

Other than that, this was a slow month as far as changes that users will notice. However, huge progress was made in finalizing the version one Dmedia Hashing Protocol, including drafting a formal protocol specification. (See the FileStore 13.01 release notes for more details.)

Before finalizing the protocol, we'd like to have at least one independent implementation. Robert von Burg has started work on a Java implementation, and has already provided very valuable feedback on things that are unclear in the specification. Thanks, Robert!

Now all that sounds very geeky, but there is a good reason why we've put so much work into this part. I felt it was deeply important that Novacut work without the cloud, that it be easy to move assets between different cloud providers, and that ferrying hard drives across the sneaker-net be a first-class way to share the assets needed for collaboration.

Cloud services can be unprecedented traps for customer lock-in, and I don't want artists to check-out of one roach motel only to become permanent residents at another. Unless you can get your data out of the cloud, and you have access to the software needed to use your data without the could, you're boned.

Although it's not being used yet, there is a new software component this month: D-Base32. It's an experimental base-32 encoding designed for document-oriented databases. Unlike standard Base32 encoding, it preserves the sort order (so sorting by the binary IDs gives you the same order as sorting by the D-Base32 encoded IDs). We haven't decided yet whether we're going to switch, and we'd love feedback on this. Please see the D-Base32 Design Rationale if you're interested.

As always, if you want to get involved with Novacut design or development, please stop by the #novacut IRC channel on freenode and introduce yourself.

Special Thanks

Special thanks to Greg McQueen for taking the time to interview me for Digital Gleu:

Install Novacut 13.01

Please follow these instructions to install Novacut on Ubuntu 12.04 LTS (Precise), Ubuntu 12.10 (Quantal), or Ubuntu Raring (the development version). Note that if you've already installed a previous version of Novacut, you'll automatically get Novacut 13.01 the next time the Ubuntu Update Manager runs.

If you're trying Novacut for the first time, you probably want to start by watching this so you understand a bit about Dmedia, and then watch this to get a good tour of Novacut.

Source code

You can download the source code from each component's Launchpad project page:

Announcing 12.12

What's New

This month was a big milestone for Dmedia because we turned on the active file monitoring.

(Please note that we do not yet consider Dmedia production ready. Please don't yet trust Dmedia as the only place you store your files.)

The active monitoring verifies that the files in your library still exist, have the expected modification time, and have the exact expected file content. This monitoring allows Dmedia to detect hardware failure and respond accordingly. Think of it like really smart RAID that works seamlessly between multiple computers and even the cloud.

The first time Dmedia 12.12 runs, you might notice a lot of CPU usage and disk activity as Dmedia verifies all the files is your library. We don't yet have a smart way of scheduling this background verification, so if it is too annoying you can open a terminal and kill Dmedia like this:

dmedia-cli Kill

For details, please see the Dmedia 12.12 release notes.

Special Thanks

Special thanks to all the new people who continue to file crash and bug reports, including HeCSa, Donalbain, Thibaut Brandscheid.

Special thanks to Simon Wells for diving into development this month. Simon has already made merge proposals for Novacut, Dmedia, and UserWebKit.

And special thanks to Kevin Cloinger for trying a tutorial hang-out-on-air with me. I walked Kevin through adding a new CouchDB view in Dmedia. The only problem is often the text is difficult to read, so be sure to watch it fullsceen at 480p.

As always, if you want to get involved with Novacut development, please stop by the #novacut IRC channel on freenode and introduce yourself.

Install Novacut 12.12

Please follow these instructions to install Novacut on Ubuntu 12.04 LTS (Precise), Ubuntu 12.12 (Quantal), or Ubuntu Raring (the development version). Note that if you've already installed a previous version of Novacut, you'll automatically get Novacut 12.12 the next time the Ubuntu Update Manager runs.

If you're trying Novacut for the first time, you probably want to start by watching this so you understand a bit about Dmedia, and then watch this to get a good tour of Novacut.

Source code

You can download the source code from each component's Launchpad project page: