Django is a database-agnostic web framework. This means that the database fields provided by Django are designed to work across SQLite, Oracle, MySQL and PostgreSQL. In fact they also work on several third-party database backends. However, the majority of applications built using Django are not, and do not need to be, database agnostic.
PostgreSQL is arguably the most popular database within the Django community and it certainly has the widest feature set of the core supported databases. PostgreSQL's advanced features have been the subject of many talks at Django conferences and events, and there are a handful of libraries in the community for using them. Historically Django has made it very hard to implement well-designed custom fields. This was in part due to a limited field API, which provided no way to do custom lookups. Whilst this worked fine for a UUID field in PostgreSQL, it meant that the implementations for arrays, hstore, JSON etc. either lacked some nice features or had unpleasant implementation details. For a developer, it represents a sometimes bewildering selection of libraries with their own distinct quirks and flaws.
I want to make things better. There should be one obvious way to use these data types and they should be a core supported feature of Django.
I am aiming to write a new contrib module for Django: django.contrib.postgres. The key features will be:
Support for many new PostgreSQL-specific data types, including:
- interval (maps to a timedelta)
- int4range, tsrange and other range fields
In addition, there will be support for using tsvector-based full text searching on appropriate fields. These new features will of course be fully tested and documented.
Any additional funding above the initial £2500 will go towards more features.
- £5000 - Improved date-based lookups for all databases. This means it will be possible to do lookups like field__year__lte. It will also include other time filters for PostgreSQL, such as __decade and __century.
- £7500 - PostgreSQL-specific custom lookups for existing fields, including string and mathematical operations. The new PostgreSQL-specific data types will also now benefit from a wider range of custom lookups.
- £10000 - Custom indexes, including different index types (e.g. gin indexes) and functional indexes (e.g. indexing on year of a date field).
- £14000 - Support for views (and materialized views) in PostgreSQL and other database backends.
Additional funding over the £14000 goal will go towards improving the support for custom transforms throughout Django, refactoring contrib.gis to remove the need for custom managers, and helping with release blockers for future releases. Some funding may also be given to help compensate those who do a lot of review work on the project.
What about $MYDATABASE?
The plan is to implement contrib.postgres without hackery. It will not be required to change the database backend or use a custom manager. This will mean some refactoring of ORM code, including some refinement of the new Lookup and Transform classes for providing custom lookups. It will make implementing a custom database field much easier on any database, as well as providing a lot of reference examples for the PostgreSQL types. The choice of PostgreSQL comes down to a combination of personal preference and the existence of a wider range of missing features than any other database Django supports.
I'm a Django core committer, a full-time parent and part-time freelance developer. I have time around my parenting commitments to work on Django, and the enthusiasm to code in the evenings and at weekends, which comes from not working at a computer all day.
I love PostgreSQL and have used it as my main database for three years, often wishing that it were easier to use its more advanced features. Whilst there are libraries which have met my use cases, the code has always felt ugly. We can do better.
The idea of contrib.postgres has been in my head for about a year. I've been researching how the ORM works and helping with the implementation of custom lookups. This means I've got a fairly clear idea how to implement the features I'm proposing.
I'm a full time parent, but I have to earn some money. This is a project of significant size, and I would not be able to dedicate the necessary time to it without funding, as I would need to take on more paid freelance work instead. Kickstarter was very successful for the new migrations framework, and I feel that being able to guarantee a significant amount of time to work properly on a project like this is the best way to ensure the end result is of the highest quality.
The kickstarter is my own - not officially organised by the Django core team or the Django Software Foundation. However, both the DSF and the core team have given their approval for the project.
The Django name and logo are trademarks of the Django Software Foundation and are used here with permission.
When will it be released?
The project is very modular. You could say that it is in fact 20 or more smaller projects, with each data type being a feature on its own. This means that each part can be reviewed and merged individually. Work will start towards the 1.8 release of Django which should see most of the simpler fields. Full text search and some more advanced cases like the enum data type will hopefully land for 1.9. Stretch goals may take longer.
Where is my money going?
The amount raised will guarantee at least 80 hours of work on the project. This will include the implementation of model fields and form fields, full unit testing and a lot of documentation with examples.
Risks and challenges
Software development always carries some element of risk. Unexpected edge cases can appear and good design can turn in to complicated implementation. However I'm confident these issues can be overcome.
There is also the risk that the patches would not get merged due to lack of review. Quite a number of the core team have expressed that they are very keen to see these features land and have use cases in their own projects, so there is significant interest in seeing the patches land. One of the developers of django-hstore has offered to help with review, and you can help too!Learn about accountability on Kickstarter
Support this project
- (31 days)