A new, flexible and modern migrations system built into Django.
Funded after just an hour and five minutes, all stretch goals met in less than four hours! Thanks so much to all who've contributed, and those who are still planning to.
Any further funding over the £7,000 stretch goal will go to both improving migrations in the long run (and looking into things like Percona online schema migration for MySQL and fully-supported Oracle and MSSQL modules) and future related Django features.
Schema migration with Django has had a long and complex history, but for the last few years South has become the go-to choice. Now, with South's four-year-old design hitting serious limits, it's time to add migration support into Django itself.
The goal is to add two things to Django core, replacing South entirely:
- A database-independent API for altering database schemas on a per-field or per-model basis
- An all-new migrations framework, generator and runner
The first of these is pretty much complete, and is sitting in a branch awaiting a merge after a bit more clean-up - you can see the changes in this pull request.
The bigger task is the second one - writing a new migration runner system that keeps the basic simplicity of South while improving on its autodetection, tackling its need for long migration histories, and its excessively complex migration files. Those improvements have already been planned and figured out over the past months, and you can read more about them below.
The new features
The new code will have quite a few improvements over South, though being based around the same core concepts - migrations per-app, autodetection of changes, and data migrations alongside schema migrations.
- Improved migration format. As well as making the migration format much more compact, it will also be readable without having to executing it, meaning migrations can be examined and optimised if needs be.
- Rebasing. It will be possible to create new initial migrations as a project evolves, meaning there's no need to keep or execute the whole history every time.
- Improved autodetection. An improved field API and built-in support inside Django means that new and custom fields will be detected and usable more easily.
- Better merge detection. The new migration format means that merging between different VCS branches will no longer need any work or extra migrations, provided the changes are mergeable.
I've discussed this upcoming design at two previous talks - the most recent one at DjangoCon US 2012. Slides and a video are available on that page.
I'm a Django core committer, am the original author and principal maintainer of South, and I've been writing Django code for about six years now.
I have a lot of knowledge of databases, schemas and how annoying they can be (possibly too much, in fact), and I've been planning and discussing this project with people - including other core committers - for just over a year. There are other people I'd trust with this project, but not very many.
I also have one full day a week to guarantee to this project, in addition to my spare time in the evenings and on weekends.
A good question - a lot of open source code gets done for free. However, my free time is limited. I currently have a day a week available for work, and I'd love to spend it improving Django rather than doing consulting or contracting.
The idea here is twofold - to guarantee the project a solid period of work and at least 80 or so hours of coding time, as well as to try and show the world that open source software really can pay for developers' time.
This Kickstarter is entirely my own thing - it's not an official Django one. However, both the Django Software Foundation and the Django core developers have given their approval.
The Django name and logo are trademarks of the Django Software Foundation and are used here with permission.
When will it be released?
As the intention is to get something into Django core itself, the idea is to land this for either the 1.6 or 1.7 release of Django. The work itself should be mainly completed by July 2013, and ready for alpha testing and feedback at that point for those who want to try a development version of Django.
There's no intention to release this as a standalone app, nor will it work with Django 1.5 or below (as it will be part of core). If enough extra funds get raised, there's the possibility of some of the changes getting backported into South for those on older versions of Django.
Obviously, Django's release timings factor into when the project finally ships - currently, the release date for 1.6 is not yet decided, but the project will definitely land within a few months, and thus in 1.7 at the latest.
Where's my money going?
The amount I'm trying to raise - £2,500 - will go towards at least 80 hours of work on the project.
The database backends are essentially complete, but a lot of work remains to be done on the migration runner itself. Some things, like the dependency solver, can be used almost unchanged from South, but others (like the new model representations) will need a lot of work and some fixes elsewhere in Django, too.
However, the time is more than sufficient to cover implementation, documentation and testing of all parts of the project, both extensive unit tests as well as playing around with testbed projects to see how it works in situ.
Any additional funding on top of the £2,500 minimum will go towards further work, including:
- £3500: I'll make sure the release that this ships in is out as fast as possible, by having time to fix any blocker bugs.
- £4500: More native operations, such as being able to move columns across foreign keys while retaining data, and CREATE INDEX CONCURRENTLY support
- £5500: Migration plan analyser that tells you which migrations will block which tables, and for roughly how long (PostgreSQL and MySQL only)
- £7000: A backport of the key features to a new major version of South to support those on Django 1.4 and 1.5
Risks and challenges Learn about accountability on Kickstarter
Software development always carries a few risks - in particular, unexpected edge cases and design flaws - but in this case my previous experience writing South and the year or so of planning and discussion means that most of the kinks have been worked out by now.
There's also the risk of the project not making it into a release of Django in a timely fashion, but as long as it proceeds relatively on-time and gets enough code review it should make it. If the funding goes above the set limit, I'll use some of it to help compensate any code reviewers for their time.
Have a question? If the info above doesn't help, you can ask the project creator directly.