Everyone loves a good read...
As the time for releasing the code approaches, I thought it would be worthwhile updating you on a few things:
Firstly, the development environment - I'll got through the details of what you need to install so you're ready to go as soon as we open up the repo.
Secondly, I've previously talked about the details and challenges of emulating tags, so I'm going to look at the other side of the coin - reading them.
I'll also cover some of the design/coding philosophies, as I feel it's important to understand not just how stuff works, but why it's been implemented in the way it has...
The Development Environment
As previously mentioned, we've settled on the PIC32 from Microchip as the main processor, and so it makes sense to use the Microchip development environment for the code. This can be found here:
We are using Mplab-X IDE V1.41 under Linux, but it has also been tested under Windows and Mac OS-X. There are subtle differences in the initial installation process for each OS, but once installed and running, they all work the same.
For the compiler, we use the free version of Microchip's XC32 V1.21:
and USB stack and peripheral support is provided by Microchip Application Library v2013-06-15:
*** Install to '/microchip_solutions_v2013-06-15'
*** Install to installer default location
*** Install to default location, which will normally be your home directory.
We'll provide detailed instructions of how these link into the project when we release, but if you get the above tools up and running and building sample projects you'll be in good shape to get started!
One of the joys for me about this project is the complete freedom to code as I see fit... Often with micro-controller projects you are heavily constrained in terms of code or memory space, and so everything is written as tersely as possible, and can be quite obtuse. However, in this case, we are using a honking great chip with stacks of memory and oodles of code space, so no such restrictions apply. Accordingly, my code is written as longhand as possible, with ease of comprehension being the prime focus, not efficiency or elegance. That's not to say that the code isn't efficient or (hopefully) elegant where it needs to be - on the contrary - I've gone to great lengths to optimise ISRs (interrupt service routines) etc., where timing is critical, but where there are no real-time issues, then everything is done idiot style, so even I can understand it when I read it back a week later... :)
Bear in mind, also, that this code is derived from a base of projects that we've amassed over a period of years, so not everything has been written entirely from scratch (although the bulk of it has, as a lot of the higher level routines have been ported from python), so if you come across a section where the style doesn't quite match, or I've done the same thing in two different ways, please feel free to let me know and/or submit a patch that tidies it up!
OK, so let's get onto the nitty gritty...
Like emulation, when reading tags we are basically still dealing with the same issue: everything boils down to ASK. Whatever the higher level modulation scheme, what's happening down at the coil level is an ASK data transfer. Before we go any further, let's just remind ourselves how this is all supposed to work, and why we need an analogue (ick!) circuit in the first place... In theory, we could use the power of the processor to do nearly all the analoguey bits for us, but in practice we don't want to do that because we'll be using up way more of the processor than we need to (and, yes, we've got a really beefy processor so it shouldn't matter, but we'd also like to be able to use smaller/cheaper processors if required). Although an LF tag is only operating at 125/134 kHz, and most processors are running in the MHz range, we still don't want to have to try and process 125,000 events per second when actually we're only interested in a much smaller data set - the actual data bits being carried on the 125 kHz carrier, which will be more like 2,000 to 4,000 events per second. Accordingly, our circuit will do the initial filtering to capture the bit-level events and ignore the carrier, leaving more of our processor cycles available to deal with the higher level modulation.
However, in the real world, this is not as simple as it sounds. One would think, if we are really just dealing with ASK, that a single simple ASK circuit would suffice, and we can do the rest in software. Sadly, this is not the case, Code Monkey assures me... When you start adding secondary modulation schemes, strange things happen to your analogue circuit, and a circuit that is good for say, FSK, will be utterly useless for PSK. Spikes that you need to read PSK simply don't appear on an FSK circuit, and vice-versa. Weird.
Happily though, Code Monkey is a bit of a genius when it comes to this kind of thing, and he came up with a simple and elegant solution: why build different circuits for each modulation type, when you can build one software controlled circuit that can modify itself according to the task at hand? Awesome. And so he did. And the secret sauce is a potentiometer. Actually, two. The circuit has two software controlled (or jumper selectable screwdriver controlled) pots that filter the spikes according to which ones you want to see. Specifically, we can filter out the additional 'low' spikes caused by PSK tags when we don't want to see them, and filter them back in when we do. I'll leave Zac to explain this in more detail, but luckily, as Code Monkey, that's all I need to know! It just works! :P
So, about those tags...
For all tag types we have a data line that goes LOW (we are 'active low') when it 'sees' some data (i.e. a pulse from the tag), so all we need to do is work out what the pulse means for that particular modulation scheme...
The code is arranged in layers to make implementation of different tag types as easy and intuitive as possible. At the lowest level we have a single ISR that does the actual reading. It fills a buffer and stops when it's read the required number of bits or it gets a timeout. It's told what modulation scheme is being used and acts accordingly. The buffer itself consists of one byte per bit set to 0x00 or 0x01. This is to avoid having to do bit-level manipulation which can introduce timing issues at byte boundaries and since we have plenty of memory we can afford to keep it simple.
At the layer above we have one setup routine per modulation scheme - one each for ASK, PSK and FSK. These are responsible for setting up the interrupt timers and globals for the lower level to pick up.
Above this we have the first layers that deal with binary data. Again, one for each modulation scheme. These will be called by higher layer protocol modules with details of the data they are expected to retrieve - number of bits, lead-in sequence, timings, etc., and will return the number of bits they managed to read and arrange the buffer so the first expected bit is where it's expected to be.
This means that implementing a new tag type is fairly straightforward - your top level routine just needs to know how many bits, what modulation scheme, and what the expected bit pattern is, and all it has to do is interpret the actual bit values that get returned by the lower layers.
In practice, this is actually the case. Implementing new tag types is very straightforward if you know the appropriate details, and some of them have taken as little as tens of minutes to add once the low level routines were solid and reliable.
Hopefully this all makes sense so far, so let's get down to the lowest level and some specific examples:
Well this is about as simple as it gets. We set up an appropriately timed interrupt that will read the data line at every bit period, and if it's LOW it's a 1, and if it's HIGH it's a 0. That's all there is to it!
There are a couple of minor issues to consider at startup time: we don't have to worry too much about low-level timing, as we are providing the clock to the tag by energising our reader coil, so we shouldn't get any drift or synchronisation errors, but we do need to align ourselves correctly to the bit period in order to get an accurate read. This becomes more important with other modulation schemes as you will see later on, but for ASK it's simple - we just wait for the data line to switch from HIGH to LOW or vice-versa and we're done. We're at a bit boundary. Wait a little bit longer to deal with any lag in the tag's processor (they have tiny brains), say a quarter of a bit period, and we should be safely sampling the correct bit value.
There are two ways of accessing this data (actually three, but I'm ignoring the API for now - that will be the subject of a later update) - you can dive into the code and add your own modules that do whatever you want, or you can use the CLI provided on the Serial/USB interface and gather data that way.
Here is an example of reading an ASK tag:
So as you can see, it's pretty easy to grab some data from a tag and start working with it offline - for example figuring out how the HEX relates to the ID printed on the tag itself, or working the data through a higher level protocol specification to see if decodes correctly.
Manchester is a higher level protocol, but it often goes hand-in-hand with ASK and is very often referred to as if it's a low level modulation scheme in it's own right. In fact, it can be applied to data modulated in any scheme, but when it's applied at the lowest level that usually means Manchester encoded ASK. Reading it is pretty much the same as reading plain ASK, but we have to deal with an extra layer of encoding. Instead of the data line giving us the bit value directly, it will flip from HIGH to LOW or LOW to HIGH during the bit period, and those sequences can be interpreted either as a 0 or a 1. The simplest way to deal with this is just to read at double the data rate so we get two 'bits' per actual bit, then re-encode them as appropriate - HIGH/LOW is a 1 and LOW/HIGH is a 0 (or vice-versa). This has the additional advantage of giving us an opportunity to adjust the clock timing if we are slightly out of phase: if we start reading a half-bit out of time, we will end up getting wrong data values, but this is easily detected because we will soon come across a pair of 'bits' that make no sense - the only correct values are HIGH/LOW or LOW/HIGH, but if we are half a bit out we will come across pairs of HIGH/HIGH or LOW/LOW. The fix is simple - restart our clock half a bit later and we are back in sync. If we get another such error then that is a genuine data error and we can abort the read operation. The lowest level ISR takes care of all of this for you if set the Manchester config option.
Incidentally, it's pretty easy to spot when you've got Manchester encoded data: try reading at double the data rate you started with, and if your hex fills with sequences of 'AA' and/or '55', then your data is most likely Manchester encoded: AA is binary '10101010' and 55 is '01010101'. Your original read will have been only 'reading' the first half of each bit, so you will still get sensible looking data, but as you don't have the half-bit clock correction operative, it's possible you're actually reading the inverse of the true values. Switching on Manchester encoding/decoding within RFIDler will confirm:
BiPhase is kind of like a poor-man's Manchester. I'm not sure what the rational is behind it, as it adds the complexity of having to decode a secondary layer, but lacks the advantage of allowing for auto clock correction. This is because it only applies the secondary level of encoding to one of the bit values - a '1' will be a HIGH/LOW combination, but a '0' simply alternates between HIGH and LOW for the full bit period, depending on the value of the trailing edge of the previous bit. Actually, thinking about it, I'm lying: I do understand what it's for as I've come across it before... When decoding magstripes, you find this in the form of 'Aiken BiPhase' or 'F2F' (Frequency Twice Frequency), and it's very useful as a self-clocking scheme for systems that don't have their own clock (you just measure the time period between transitions, and if it's roughly half, double, or the same as the previous one, you know exactly where you are). You then adjust your base period value to the period you just read and you've accounted for the minor acceleration or deceleration you get when, for example, manually swiping a credit card. Neat!
As with Manchester, you'll also get '55' or 'AA' values when reading at twice the data rate, but unlike Manchester, this won't (or shouldn't) apply to every bit, so this can be misleading:
This is almost as easy to read as ASK - it's just a question of checking the data line at each bit period, and if we get a blip then we toggle the bit value. This is because a phase-change in the reader circuit will create a momentary pulse on the reader data line. In practice, we can be a bit more selective about how we read this pulse - we can choose to look for it for as long as we like within the bit period, thus allowing for timing differences between different tags, and we can also measure the length of the pulse, which will tend to get longer as the tag is better coupled with the reader, thus giving us a measure of quality. Accordingly, we allow the setting of the minimal acceptable pulse length when configuring for PSK reading.
This is the trickiest protocol to read, and is still giving me problems. This is because every tag I've looked at so far behaves slightly differently, so whatever scheme I come up with for demodulating works perfectly for one tag type, but not another. I'm sure there is an overall solution, and I'm dying to get the code released so you guys can help me find it!
There are several ways you could demodulate the signal, and the first I tried was to simply count the number of pulses within a bit period. This will be one value for a '0' and a different value for a '1'. The only issue is making sure you line up on a bit period, and the simple way to do that is to measure the width of the individual pulses or gaps, and when it changes you've just crossed a bit boundary as you've gone from a sequence of '0' sized pulses to a sequence of '1' sized pulses, or vice-versa. Great, that sounds simple, and indeed it was! Worked (almost) first time, and very reliably. I have a huge stack of HID 26 bit cards that read and decoded perfectly, no problem. However, I have another stack of older HID 26 tags that don't work at all. Ever. I was very puzzled by this until I looked at the input to the reader coil on a scope: instead of being a different number of different sized pulses in each bit period, there were the same number of pulses, but their size was massively different. Very strange, but I guess in the analogue world this all somehow balances out to create the two frequencies required to impart the desired data signal, so it's back to the drawing board for me... I guess I'll measure size of pulses rather than number, and hopefully that will work for both tag types. We shall see!
Higher level decoding
So now we've got the fundamental bit-level reading working, what about the higher level stuff? Well, in most cases implementing higher level tag decoding is simply a matter of reading some specs and doing a little bit manipulation - strip off a header, remove some parity bits, re-order what's left and convert it to HEX and you're done. We are adding all the tag types we know about, but there are always the ones that need reverse engineering, so we've tried to make RFIDler as useful as possible in that regard by providing command-line tools for taking things a step at a time:
That's it for today, I hope you found it useful and I'm really looking forward to the release so we can start working on this stuff together!