The Allwinner ARM processors are very popular in a large number of hobbyists development boards, and more and more widely used in embedded products. Support for those processors was initiated in the official Linux kernel by Bootlin (formerly Free Electrons) engineer Maxime Ripard in 2012, and has since then been improved up to the point where actual Allwinner-based products are being shipped with the official Linux kernel.
However, one key feature that is missing from the support in the official Linux kernel is support for the VPU (Video Processing Unit). This unit allows to offload to the hardware the decoding and encoding of video streams, avoiding the need to involve the CPU for such intensive operations. Having VPU support would provide a smoother video decoding/encoding experience, at a much lower CPU consumption.
A lot of existing work has already been done around supporting Allwinner’s VPU (including by Bootlin, see below), and the goal of this Kickstarter is to build on top of this work to complete VPU support and have it supported in the official Linux kernel.
It is worth mentioning that this goal is listed as one of the High Priority projects of the Free Software Foundation.
A Wiki page on the Wiki of the Linux Sunxi community summarizes very well the existing work done around Video Engine, the code name for Allwinner's VPU.
In the Linux BSP provided by Allwinner, the VPU is supported through a closed-source binary blob. Therefore, a reverse engineering effort was started, which has produced a nice register documentation. This great effort was done by members of Sunxi community, and has allowed several open-source implementations to appear:
- libvdpau-sunxi, documented at http://linux-sunxi.org/Cedrus/libvdpau-sunxi. This solution implements a libvdpau backend, but is meant to work with the Allwinner vendor kernel, not a recent official Linux kernel.
- sunxi-cedrus, documented at http://linux-sunxi.org/Sunxi-cedrus, which has been started in 2017 by Bootlin (formerly Free Electrons). It consists in a Linux kernel driver working on recent upstream Linux kernel, and libva backend. It currently supports MPEG2 decoding and has partial support for MPEG4 decoding, and has been tested on Allwinner A13 and A33. See also https://bootlin.com/blog/support-for-the-allwinner-vpu-in-the-mainline-linux-kernel/ for more details and a demonstration video.
We believe that our approach of sunxi-cedrus is more future-proof, as it is based on a standard V4L driver, which will use a standardized and upstream acceptable user-space API. This will allow the V4L kernel driver to be accepted in the official Linux kernel, ensuring its long-term availability and maintenance.
The main goal of this Kickstarter campaign is to continue the development of sunxi-cedrus and bring it to a fully usable state for MPEG2 and H264 decoding, supported in the official Linux kernel. This involves the following steps:
- Making sure that the codec works on the older Allwinner SoCs that are still widely used: A10 (Cubieboard), A13 (A13-Olinuxino), A20 (Cubieboard 2, A20-Olinuxino), A33 (A33-Olinuxino, BananaPi M2-Magic), R8 (CHIP) and R16 (NES and Super NES classic). Support for the newer SoCs (H3, H5 and A64) requires more work, and is part of our first stretch goal below.
- Polishing the existing MPEG2 decoding support to make it fully production ready.
- Implementing H264 video decoding, since H264 is by far one of the most popular video codec.
- Modifying the Allwinner display driver in order to be able to directly display the decoded frames instead of converting and copying those frames, which is very inefficient from a CPU consumption point of view
- Providing a user-space library easy to integrate in the popular open-source video players
- Upstreaming those changes to the official Linux kernel
The target of 17600 EUR of this Kickstarter campaign allows to achieve those goals.
In addition to the initial Sunxi-cedrus support described as our main goal above, we have additional stretch goals to further improve Sunxi-cedrus if we receive more funding than our target:
- Supporting the newer Allwinner SoCs, such as the H3 (Most of the Orange Pis, Nano Pi M1, ..), H5 (Orange Pi PC2, NanoPi NEO2, ...) and A64 (Pine64, BananaPi M64, ...).
This goal requires a funding target of 22000 EUR (i.e 4400 EUR in addition to the main goal of 17600 EUR)
H265 video decoding support
This goal requires a funding target of 30800 EUR (i.e 8800 EUR in addition to the main goal of 17600 EUR and the first stretch goal of 4400 EUR)
H264 video encoding support
This goal requires a funding target of 44000 EUR (i.e 13200 EUR in addition to the main goal of 17600 EUR and the previous stretch goals of 4400 EUR and 8800 EUR)
Engineers for this project
Two persons will be involved in this project:
- Maxime Ripard, working for Bootlin (formerly Free Electrons) as an embedded Linux engineer since 2011. Maxime has pioneered the support for Allwinner processors in the official Linux kernel, and is currently the co-maintainer for Allwinner processor support in both Linux and U-Boot. Maxime has written numerous drivers to support Allwinner platforms, including the DRM/KMS driver for display support. See Maxime's Linux Kernel contributions and U-Boot contributions.
- Paul Kocialkowski, who will be working as an intern at Bootlin (formerly Free Electrons) starting March 2018. Paul has contributed to numerous open-source projects: lead developer of the Replicant project (a fully-free Android-based OS for smartphones), and contributor to projects such as U-Boot, Linux and Coreboot. See Paul's U-Boot contributions, Coreboot contributions and Linux contributions.
We expect to deliver our main goals by end of June 2018, and our stretch goals by the end of December 2018.
During the entire period of development, we will be sharing the progress of the project every week in a blog post on https://www.bootlin.com.
Risks and challenges
Bootlin has already developed and released a kernel driver and associated user-space library providing MPEG2 support and partial MPEG4 support for the Allwinner VPU. We have therefore shown our ability to work on such topics. Nevertheless, there are obviously some challenges:
- There is no publicly available official datasheet for the Allwinner VPU, and all the information available has been found through reverse engineering. There is always a risk of this information to be incomplete or inaccurate, requiring tedious additional reverse engineering effort.
- The Linux kernel V4L API to support VPUs such as Allwinner's is not available in the official Linux kernel yet. Adding such an API has been in discussion in the Linux V4L for several years, but hasn't reached mainline at this point. Recent progress and discussions have been encouraging, but it might remain a challenge for this project.
- (45 days)