Day 691: Crawling toward PVT
TL;DR: Initial FCC/CE pretesting results; Component sourcing craziness; We're guessing that we're about two weeks out from starting PVT
Hello from Oakland,
When we last wrote, we were pretty sure that April would see Jesse in Shenzhen overseeing manufacture of the first 100 Model 01 keyboards.
That didn't happen and we're pretty bummed out about it.
As of this writing, we believe Jesse will be getting on a plane to Shenzhen in two or three weeks. Below, we're going to lay out what's happened in the past month and what still needs to happen before we start manufacturing.
When we wrote last month, we told you that the factory had enough RJ45 connectors for the first 1800 keyboards in their warehouse. While that is now true, it wasn't then. We only found this out about two and a half weeks ago, when the factory told us they hadn't yet sent keyboards to the test lab for FCC/CE pretesting.
As we dug into it, we found out that they hadn't sent the keyboards to the test lab because they couldn't finish building the circuit boards. They couldn't finish building the circuit boards because, while he factory's purchasing team thought they had received a large order of our RJ45 connectors, in fact the Chinese customs authorities had received them.
It took a while for the whole story to come out, but it sounds like what happened was that the customs declaration for the Taiwan-made RJ45 connectors wildly undervalued the product, so much so that Chinese customs took one look at the packages, said "Nope!", and impounded the goods.
We don't know why the customs declaration was so wrong, though we suspect that someone, somewhere was trying to be "helpful" to minimize taxes paid on the parts. We got somewhat cranky when we found all this out, and while we don't know exactly what went down, the factory reports that they have 3600 RJ45s sitting in their warehouse now.
As we mentioned above, the first round of FCC/CE pretesting didn't happen when the factory said it would. Once we understood what was going on, the factory scrambled to buy a small number of RJ45 connectors to build test keyboards to submit for lab testing.
For reasons that still aren't entirely clear to us, the factory didn't send sample keyboards to the test lab until April 24.
On the 26th, the test lab told us that they were short some documentation: electronic schematics, an electronic block diagram, a copy of the Bill of Materials, a datasheet for the LEDs and an "Operational Description." We turned all of that around in a couple of hours, though we had to guess on the operational description and nobody had ever asked us for a block diagram for the keyboard before. The test lab promised to have results back in two business days, though this week's May Day holiday meant we didn't see the first round results until Tuesday.
We almost passed.
Of course, in the realm of certification, "almost passed" means "failed." The Model 01 had just a little bit too much electromagnetic radiation in the 35-115Mhz range.
Because this sort of thing is pretty common, and a little bit of a dark art unless you have a lot of experience, we designed the Model 01's circuit board to let us add slew rate limiting capacitors on the USB data lines.
One of the services the test lab offers is engineering support to help resolve exactly this sort of issue. The next step is for them to evaluate our proposed solution and make their recommendations for a remediation. It sounds like we'll have their feedback by the end of the week and then they'll test the solution next week.
If the next round of pretesting looks good, we can start building the electronics for the PVT units in a little over a week. We'll turn around and use some of those PVT keyboards for the official CE certification and FCC Declaration of Conformity.
The factory currently has enough keyswitches in stock for the first 500 keyboards. (They've counted them twice.) Due to a mistake by their purchasing department, the rest of the switches didn't get ordered until February. Matias, our keyswitch manufacturer, initially thought they'd be delivering all of our switches sometime in March, but that didn't end up happening. They've committed to delivering us at least 1000 keyboards worth of keyswitches per month from here on in.
The delay is related to an improvement the they made to reduce the amount of "wobble" in the switches at the end of last year. The new switches feel a lot nicer, but due to the tighter tolerances, their manufacturing operation has taken some time to come up to speed. The team at Matias is bending over backwards to get us switches as quickly as possible and right now we believe that we'll have covered 90% of pre-orders with June's keyswitch delivery.
Keycap injection molding
When we last wrote, we told you that the keycap factory was reworking all of our keycap stems, so they didn’t grip the keyswitches so hard that they ripped the keyboard apart when you try to pull one out. That work was supposed to be completed by, roughly, the first week of April. It ended up stretching another two weeks. We got test keycaps in the mail at the beginning of last week and didn't get the factory's test report until the middle of last week. By our measurements, 48 of the keycaps required somewhere between 1.3-1.5 kg of force to remove, with the rest at 0.9 kg to 1.2 kg. 0.9 kg is way, way too loose. 1.2 kg would be workable, but isn't what we'd expected.
The factory's measurements all come in a bit lower than ours, with 20 of the keycaps at less than 1 kg of force being required to remove them and no keycap requiring more than 1.2 kg of force to remove. We think the difference can be chalked up to us using a fancy digital force gauge and them using something a bit more analog.
After a bit of negotiation, the factory agreed to have the keycap factory fix the keycaps that they measured as taking less than 1 kg of force to remove. They'd promised us an estimate of the time required to to modify the tooling last week, but we're still waiting on it. Thankfully this kind of modification is relatively quick and easy. To get the keycaps to fit more tightly, the plastic key stems need to be a hair bigger. And to do that, the factory needs to shave a little bit of metal off the injection molds. (To reduce the force required to remove the keycaps, they had to cut big holes in the injection molds, weld in new steel and remachine the molds. That was the bit that took so long last time.)
Keycap laser engraving
In mid-April, the keycap engraving supplier sent updated samples back to our factory. They'd done a pretty good job of getting the legends and paint in the right place, but even from the photos, we could see that there were a few labels that were still misaligned. The factory promised to have the laser engraving supplier take another pass at it. That new version of the keycaps was due to be returned to the factory last Wednesday, but never showed up. The factory is now promising that they'll be done sometime today.
We're starting to suspect that the only way these issues are going to get ironed out is to have Jesse go sit at the laser engraving supplier's office and work with their engineers. If that happens, you can expect a bunch of photos in the next backer update.
In the last update, we told you that the factory said they'd resolved the issues with the center bars deforming a bit as they came out of the injection molds. They sent us samples. They had, in fact, resolved the curvature issue. From there, we pointed out a few issues we saw with the finish on the samples and asked them to spend a bit of time cleaning up the surface of the injection molds. At the beginning of last week, we got new samples in the mail. We're happy enough with the cleanup work they've done to declare the center bars "done"
Short RJ45 cables
One of the small, but important details we've been working through has been the RJ45 cables we use to connect the two halves of the keyboard.
The longer cables were pretty easy to sort out, but the shorter cables have been unexpectedly difficult. Most cable vendors didn't seem to get why we needed our short cables to be so short. (They have to fit inside the little cutouts between the two halves of the keyboard.)
We specced 12 or 13cm long cables, end to end. We got samples back that were half that length and a couple that were double that length. Some of the samples had connectors that were so long they couldn't fit inside the keyboard. One or two were so stiff that they put serious lateral pressure on the RJ45 jacks. That could lead to the jacks becoming less reliable over time. One sample had connectors so short that they got stuck inside the jacks. One sample had shrouds on the cable that were so long that the connectors wouldn't even fit into the jacks.
One vendor has made us a new set of sample cables that look like they're going to be just right. They match some samples that were made by a different vendor last year. Last year’s cables fit great and worked well, but weren't particularly well made. We expect to have the new vendor’s version in hand sometime next week.
USB Type C Cables
Last time around, we mentioned that we were looking at possibly forgoing the "right angle" Type C cables if we couldn't find a vendor who made something that seemed high enough quality. In the end, all the right-angle Type C cables we found that were well made had gigantic plastic housings around the connectors that prevented them from getting a solid connection to the Model 01. All of the good-looking right-angle cables were either mechanically or electrically wrong.
We've ended up deciding to go with a straight Type C cable from a vendor our factory has worked with before. It's still 1.5 meters long and our teardown shows it to be well made. It has a 56K pull-up resistor in the right place, which means it's not fraudulently claiming to support USB Power Delivery, so it shouldn't melt when you use it to charge your phone.
In the end, it's a pretty boring cable. We're ok with that.
On the software front, there's lots of great stuff happening.
Algernon has been hard at work on Chrysalis, the cross-platform desktop interface for managing, programming and flashing the Model 01 (and other keyboards that run Kaleidoscope or have similar serial communications protocols.)
Over the past month, he's built functionality to autodetect a supported keyboard when you plug it in, flash new firmware on the keyboard over the USB line using @noopkat's avrgirl, the beginnings of a graphical LED theme editor, as well as a few more bits. He’s released a first “alpha” version which includes a mock keyboard, so you can try it out, even without a supported keyboard.
Since Algernon has been focusing on the GUI, the firmware itself hasn't seen a ton of work over the past month. (It doesn't hurt that we have implementations of pretty much every keyboard feature we've reached for.)
The other exciting development on the firmware front isn't exactly Keyboardio related. About a week ago, Wez Furlong popped up with patches to Kaleidoscope that let him add support for keyboards with ARM microcontrollers and Bluetooth connections. One of his first targets is Adafruit's nRF52 Bluetooth board.
There's still a fair amount of refactoring to go before it's clean and easy to use Kaleidoscope with ARM chips and Bluetooth connections, but we're really excited to see this happening. If more keyboards can run Kaleidoscope, more intrepid hackers will be building cool plugins to make your keyboard better.
Our current best guess is that Jesse will be in Shenzhen in about two weeks to oversee the PVT run.
Sometime right around then, Kickstarter backers should expect to get email asking them to confirm their shipping addresses and keycap preferences. While we already have shipping addresses for post-Kickstarter preorder customers, we’re planning on sending them surveys, too. We know it’s been awhile since folks ordered and we want to make sure your Model 01 doesn’t end up shipped to the wrong place.
PVT assembly should take one week. After that, most of the PVT units will be express-shipped to customers. We don’t have a firm methodology for who gets the PVT units. Our plan is still to send those units to early Kickstarter backers located in the United States who confirm on their surveys that they’re happy to get some of the earliest production keyboards, and to put them through their paces, but we’re going to play it a little bit by ear.
A few will PVT units will be held back for reliability testing. The factory thinks that will take about a week and a half. If reliability testing of the PVT keyboards looks good, it's full steam ahead.