Lib-Ray Template System

The Lib-Ray authoring wizard will include generation of menus from theme templates. I've been working the last couple of days on the internal side of this. The following is of interest to people considering writing their own themes. I plan to include at least two themes with the release (at minimum there will be themes based on the layout for the Blender and Sita releases).

So, if this is too technical, do not fear -- you won't need to know this just to use the Wizard to create a Lib-Ray volume, this is internal stuff.

Happily, I found that my old friend "SimpleTAL" is still readily available. This, combined with the "zipfile" and "ConfigParser" modules from the Python standard library covers everything we need for creating templates.

So template files will be ZIP files with SimpleTAL page templates and some control data, like this:

(Not shown: the template will typically contain additional static resources like image files or fonts. I also have not gotten into the full-blown internationalization system that I described earlier. For now, we just have a "setup" page as on a DVD. This will get expanded).

The template files will be expanded into a Lib-Ray structure based on user input and the configuration file included with the template. For example, the program will negotiate the number of chapters that appear on each chapter page and how they are laid out. The logic available in TAL templates is sufficient to handle this kind of layout problem, so the template can set upper and lower limits on this parameter and allow the user to choose.

As a result, the Lib-Ray volume image generated from the template will look something like this:

This also shows the internal object names for the page templates inside the Wizard script.

This layout includes all of the Lib-Ray specific HTML requirements, with stubs provided for the special feature hooks (volume special features, title special features, and "extras" -- where the "extras" are eqiuvalent to "DVD-ROM" content and do not have to conform to the security limitations on Lib-Ray menus, such as no external links or scripting). Once you get into authoring special features pages, you're designing HTML, and it will be more sensible to use existing HTML design tools for that (including a plain text editor).

One special case that's worth considering is the "title_in_index" option. For volumes with a single video title, it's kind of inconvenient to make the user hop through two menus, so we provide an option to condense the "title page" onto the volume "index page", like so:

(I haven't quite decided whether to only provide this option for template designers, or to require templates to allow for either case).

Lib-Ray Title Generation (Testing)

The code to generate Lib-Ray title video conversion scripts appears to work fine. I need to do some more to have it work from PNG streams correctly, and I need to do some testing on the audio and subtitle stream generators. The script actually writes a conversion script and then runs that, which allows for a lot of flexibility. I also need to actually have the script launch this new script into a background process, as it may potentially need to run for hours (vpxenc, in particular, tends to need quite awhile to work).

I'm not quite following the proper "test driven development" process, which dissatisfies me a little. I really should be creating unit tests and coding to meet them.

As it is, though, I have some doctests which work and some which don't, because they are interactive. I'll either have to drop the interactive ones out of the testing system or come up with some clever way to run the tests on them. I'm leaning towards just leaving them out and testing the interactive stuff manually, at least until I think of something clever.

"And that's the news from Lake Wobegon..." -- I'll be back with more later. :-)

Just a quick update on a lot of little points...

First of all, I can announce a small enhancement: the Blender Open Movie Collection release definitely will include Tears of Steel since all of the necessary files for mastering it have been released and there will be plenty of room for it.

After installing a new hard drive to make room for them all, I can also say that I have all of the video mastering data for the Blender Open Movies downloaded and ready to convert (I've had Sita Sings the Blues on disk and already converted for a long time now, as I'm sure I've mentioned).

I am a little self-conscious about blow-by-blow accounts of my programming frustrations -- I hate feeling stupid and I know that at least a few of you reading this are probably better programmers than I and might not have made the same mistakes. I will say that am not having any problems that require some sort of intervention, even though there is quite a bit of churn under the waterline. This part is taking longer than I hoped, though I don't think it changes the overall picture.

I'll just summarize a few of the high points there:

• Player code -- I quickly got a skeleton with the different elements (Webkit menu viewer, VLC-based playback module, and original volume data model which handles language and region settings and so on) working fairly well separately. Integrating them led to some issues with access to event signaling and other interaction problems that need to be sorted out so that these libraries will play politely with each other. That got a little frustrating, and I may have blown some time going down dead-ends. I have spent some time researching this, and am also looking at how Novacut dealt with this in their "userwebkit" package, which will hopefully give me some better insight on how to manage it.
• I spent some time researching the state of "MCE compatible" infrared remote control interfaces to learn how they work. It actually looks like there may be more than one way. I have two different models for testing, one of them appears to emulate a keyboard while the other may require using the IR library directly. I still need to do some more direct testing of that.
• These things made it very clear to me that my next step was to build a good test suite. I started on this, although I did run into an issue with the fact that a useful test suite is sort of ridiculously large compared to the code itself (something like 100MB versus well under 1MB). I tried making some smaller test videos, but if I'm going to test the high-fidelity transcoding parts, I'll need somewhat larger files to work with, and if I'm already committed to that, I might as well use them for the file-structure and other elements. So I've been thinking about putting the test fixtures in a separate archive ("tarball"). And yeah, that means I'm also a little behind on updating the SVN as well, while I sort out what belongs in it and what doesn't.
• I'm using the recently-downloaded Blender Open Movies now to test the "Lib-Ray Wizard" code that I'm developing, and these and a few additional test articles (including the "Lunatics" animatic Lib-Ray volume I created for some of the backers on that project, as well as a few that are purely for testing).
• Which brings me to the progress on the wizard software: I've started with a stream-based (command-line) interactive version. This just asks the relevant questions and constructs a chain of scripts which use avconv, vpxenc, qpsnr, and other utilities to convert source video, audio, and text into the combined Lib-Ray video file. Soon this will also be able to create a simple menu system based on templates for the main page, chapter pages, and so on, and we'll have at least a couple of templates available. After a GTK GUI is added, this will become the Wizard promised in the Kickstart.
• And of course that wraps back around to testing the player again, which I hope to update in the next few days so that it's actually starting to be useful. That will also include the test code and the wizard, along with (probably) a separate download for the test fixtures.

Of course, there is a lot more to this project than the player and wizard code...

I've also been working on the HTPC test player, which is currently working okay with Debian "Wheezy" and a basic XFCE environment, except that I'm still having some trouble getting audio playback through the HDMI monitor I'm using to test it. I'm not sure where the problem lies there -- ALSA does recognize the HDMI hardware and seems to think it's playing, but no sound is getting through, so that's some more troubleshooting to work on.

Then there's the matter of sourcing the SDHC cards, duplicating data onto them, and packaging them. After some experimentation, it looks like the 3D-printed SD card holder is never going to be as good as commercially-manufactured cases, and I've found some that are suitable for this kind of application, which I will be ordering in the next few days. Likewise, the tin cases are no problem and I should have those soon.

I was watching the prices on the SDHC card market, but the supplier I was originally planning on using just stopped carrying them, leaving me looking at more expensive suppliers. Which was really frustrating (if only I'd just bought them a little earlier!), but not show-stopping -- the rates are still low enough to manage on the existing budget. The only thing that really worries me about SD cards now is that "Micro SD" cards are more common now, and I hope this won't affect their availability (I really don't want to use Micro SD, because they are just so tiny -- there's already some concern that SD cards will be too easy to lose as it is, so I don't want to go any smaller).

Another little setback was the hike in US Postal Service rates for international shipping of packages, which just about doubled in January.

On the other hand, one nice thing about SDHC media, versus optical disks, is that they are small, thin, and flexible -- it's very likely that at least the compact versions can be shipped as "letters" under USPS rules, which could be useful for Lib-Ray not just now, but as an advantage for the format in general. The compact version will probably be a folded paper booklet with an attached pouch for the SD card, as this seems to be the simplest way to do it.

I created a draft of the "informal specification" which is now available for download. I was hoping to update the section on infrared remote controls, but that's taking awhile. Most of the rest of it is finished. I emphasize that this is only an informal description of the standard -- of course, there is a more definitive version still to come.

So, overall, there's several frustrating setbacks, but also some progress. I do feel quite overdue on my immediate milestones -- especially the player software -- but I still don't feel this changes the big picture or final delivery date.

I hope my current frustrations turn out just to be minor stumbling blocks, and I'll have good news next week, but one way or the other I'll post again then to let you know how it's going.

Last week I went through a lot of experimental work on the menu system. I'm currently coding a draft of the player code based on that.

"lrdetect.py"

Basically, there will be two scripts here: The first is just a short script to be run from udev (or other automounting software) which will detect that a Lib-Ray volume has been attached. It will mount it, verify that it is Lib-Ray and  then (optionally) spawn the Lib-Ray player in a new process. This will probably need to be modified for different environments -- in an embedded player, for example, you'll to want it to launch immediately whereas in KDE, you'll probably want the notifier to pop up and ask the user what they want to do with the volume.

Other than figuring out how this would work, I haven't done too much with it yet -- the first code will just take the already-mounted Lib-Ray partition as an argument.

"lrplayer.py"

The second script (or perhaps I should say "program") is the actual player. This will launch a GTK "gui" thread that launches a browser. Then it will launch a separate thread with a SimpleHTTPServer (from the Python library) which is modified to sniff for settings needed by the video player and also for links to the actual video files.

State variables are passed by HTTP GET. So I will have settings URLs that look like "?audio=en" (which would set the audio to the first English-language track and return to the same menu) or "feature.mkv?chapter=3" which will set the playback to start from the 3rd chapter defined in the MKV file and then launch VLC to play it.

I started with an excellent tutorial by David Baird on creating Python GUIs with HTML.

After some experimentation with the Javascript-injection approach that he used in his tutorial and a quick look over the SimpleHTTPServer in the Python standard library, I decided that it would be simpler all around to just launch an HTTP thread at the same time (otherwise, I would have been looking at re-implementing anchor-link semantics in Javascript which seemed kind of pointless).

Since I'm adopting a REST-like interface for the controls, this eliminates a lot of the need for injected Javascript (It'll probably come back when I want to do things like control state settings or highlight buttons -- but I'll probably skip that in this first implementation).

A few months ago, I demonstrated launching a VLC video player and playing the Lib-Ray MKV/VP8 video with control over all of the features needed (I had some serious dropped-frames with playback in Gstreamer -- a problem to be revisited later).

One way or another, the code is simply going to launch the VLC player in a new logical window when the player transitions to video (in fullscreen mode, it will take over the screen from the menu system). This eliminates potential sources for playback problems.

You might ask why I'm keeping the GTK code in the mix, since I don't seem to be using it much. The truth is, I've considered removing it, but I may want to use it to listen to keyboard events and it may be desirable to pop-up GTK menus to control aspects of the player outside of the HTML menu system (much as you can gain direct control over audio tracks and subtitles with many embedded video players).

Designing Menus for Lib-Ray

I also spent some time working on menu design. My earliest versions of Lib-Ray made the rigid assumption of a 1920x1080 video display, but it's obvious to me now that this is too restrictive. People are going to want to play Lib-Ray on smaller and larger displays, so it needs to scale properly.

HTML and CSS -- even HTML5 and CSS3 -- are not really designed for true scaling on displays, and getting a layout that looks consistent on different size windows or monitors is apparently impossible in the current version of Webkit that I'm using in development with only HTML and CSS and no Javascript.

CSS3 did introduce some new scaling units for text -- "vw", "vh", and "vm" -- but they are not widely supported, although I understand that the very latest development version of Webkit does have them now.

Still, it may be that this design issue is not too critical. I was able to very easily come up with layouts that look "good enough" on displays ranging from 1280x720 to 1920x1080. And although the text looks smaller on the high-resolution screen, it is of course readable. So this may not be something to obsess over.

Another solution would be to use images or SVG drawings -- both of which are supported and do scale properly. I believe I've already mentioned the possibility of implementing menus in SVG -- this might be one reason to do it.

Adding looped music to menus is trivial with the "audio" element. It's probably possible to do an animation loop with video elements or perhaps with SMIL/SVG animation. This isn't a feature I'm particularly excited about, so I haven't really tried it, but I suspect it can be done.

I also haven't given much thought to "transitions" as used with DVDs (i.e. where a brief video is played between pushing a button and doing the requested action), but as we can pass variables along with the video link in the current model, it should be possible to specify a target for the video player to return to after playing the video it is given. So that should include the same kind of functionality.

I'll be coding for the next couple of days -- I'm hoping to make an initial check-in to the Google Code site on Friday.

Hello all!

I've gotten more than a few concerned queries about how Lib-Ray is going, because there isn't a lot to show for it right now. I completely understand the concern. However, I'm very much planning to deliver on or before the July 2013 "Estimated Delivery" date for the project. I have a very high confidence in that.

"Significant Pauses in Production"

What has been taking so long has nothing to do with problems in Lib-Ray, but with my other commitment, "Lunatics", which has been taking up a great deal more time than I estimated (which, yes, is due to bad estimation on my part -- fortunately my estimates for Lib-Ray itself are based on much more experience).

I am truly sorry about the delays, but this was  an expected possibility documented in the proposal under "Other Commitments". "Lunatics" is a prior commitment, so it had to take precedence. This is what I meant by "significant pauses in production" although I have to admit that "significant" sort of turned into "absurdly long" in this case.

Back to Work on Lib-Ray!

On the other hand, this problem is past now. All that's left on the "Lunatics" deliveries I was working so hard to complete is burning and shipping disks, which will be taken care of by Saturday. So the point is now moot.

I will be working 100% on Lib-Ray until I have achieved the following milestones:

• All basic playback features supported in the software player
• Prototype releases mastered (i.e. "Sita Sings the Blues" and "Blender Open Movies")
• Informal description of file system, file structure, and file formats
• Documented "from scratch" production of releases

I will refrain from making predictions on just how long this will take, but there will certainly be some code checked in on the software by the end of January.

I'm getting pretty excited about being able to work on this without interruptions, as I'm sure you can appreciate how hard it is to code anything when you're having to multi-task a lot. I think we can expect pretty rapid progress once I get into it.

Things Already Done on Lib-Ray

Even during the delays from Lunatics, I was not completely idle with Lib-Ray. I've done a great deal of experimentation with the design and research into the standards on which Lib-Ray is based. This will translate to rapid progress now that I have an adequate block of time to put it to use.

In previous updates, I've already described design changes that have greatly reduced the scope of the programming part of the project. The software player will now be implemented entirely in high-level Python, with no need to hack the browser code itself or the Javascript engine.

At this point, there's only one small design issue to resolve, which is whether to actually run a server for the menus and POST the button pushes to it, or to mimic this behavior by directly controlling the browser library. The latter appears to be a good approach, and might be a little more responsive. The only downside, is that it does involve injecting code into the menu HTML on-the-fly, which might be a little messy (the Javascript is fairly trivial and is part of the player in this design). However, I have a good source to follow on this, so I'm pretty confident it will work nicely.

One thing that I recently started experimenting with is using SVG embedded into HTML, which is permitted in HTML5 and supported by the browser engines (Webkit and Mozilla/Firefox) that I'll be working with.

That's pretty neat, because it raises the possibility of designing menus with Inkscape, which would allow me to go beyond the simple "wizard" approach that I first proposed, and allow for some interactive design.

No doubt, however, there are some of you recoiling at the idea of designing menus with SVG. Have no fear: the design may allow SVG, but it won't require it. I will probably design one of the prototypes using HTML+CSS and the other with SVG, just to illustrate the possibilities. So far, I'm not aware of any reason why this won't work.

Experimentation with Gstreamer was ultimately disappointing -- playback freezes at certain points, which I think might be bitrate extremes in the video. I therefore decided to test libvlc instead, and that proved much better. So VLC it will be! I'll try to add a Gstreamer backend afterward for anyone who wants to try to get it to work.

It may seem much more trivial, but I also spent some time on the package-design for the releases, costing out various options for how to deliver both the "Economy/Compact" versions and the "Deluxe Editions". The vinyl pocket seems to be by far the simplest and most reliable approach for the compact version, and the program book provides a place to mount it. The deluxe edition will be pretty much as planned, although I'm debating over whether my 3D-printed SD card holder is actually better than just using a commercially-available clamshell case for the card. I'll probably want to go through at least another iteration on that design.

Is This Going to Happen Again?

Possibly. But not nearly as badly! Now that I've delivered on the first Kickstart for Lunatics, Lib-Ray is the prior-commitment, and Lunatics work will have to wait. The last delivery on Lib-Ray will undoubtedly be the book, "The Complete Guide to Lib-Ray" and that might not come out until this Summer, close to the deadline.

The hardware (HTPC) part of this project may have a few delays as I need to do some serious troubleshooting on the test machine. I built that awhile back, but the sound driver is proving recalcitrant -- the motherboard has a new chipset from Realtek. That's probably not a serious obstacle, it's just a time sink, while I troubleshoot the problem.

The binary packaged version of the software and the Live-CD edition will also be fairly late, since they require all of the other pieces to be completed first. However, everything should be done by July as predicted.

I'll post again at the end of next week -- hopefully by then, I'll have some cool stuff to show you!

For the last week or so I've been putting in some serious work on Lib-Ray in hopes of having a nice milestone to report. A lot of that was trying to get the ALSA driver for the Realtex sound system on the HTPC test machine to work, which is kind of boring to talk about, but I did also do some more research and testing of the components for the software player.

The biggest deal, is that after doing some more testing with Gstreamer, I've discovered it has some serious playback problems with the high-bitrate VP8 video I'm using. VLC doesn't seem to have these problems, so, even though the Python API is not quite as well documented (there is a full API document, but not so many nice tutorials as there are with Gstreamer), it looks like using VLC may be a better option.

So, I've begun testing libvlc/python (which is a ctypes wrapper around the libvlc C library).

It's not a bad API, and the performance is very good -- I haven't seen a single glitch so far.

Right now, I have it streaming "Sita Sings the Blues" fullscreen with the Advanced Substation Alpha subtitltes -- all driven from the Python interactive interpreter. That's a really promising test. Adapting this into a playback module in the Lib-Ray player should be relatively simple now. It also fully supports MKV's named-chapters support.

Screenshot below!