Day 590: Two steps forward, one step back.
TL;DR: The PVT run didn't happen before Chinese New Year; Our factory fired an underperforming supplier; We've approved the final versions of some parts.
Hello from Oakland,
We've made a bunch of progress over the past month, but we're behind where we hoped to be.
The major issues right now are the keyboard baseplates, availability of the APA 102C LEDs we're using, and Chinese New Year.
Like we did last month, we've structured this update as a "part by part" update, roughly paralleling Jesse's nightly calls with the factory in China.
The factory sent "golden sample" wooden enclosures in their most recent delivery. The changes since the previous sample were primarily in how tightly they fit the baseplates. Up until this set of samples, every single enclosure had fit the baseplates so tightly you needed to apply force to get them to mate. Because wood can shrink or expand a bit depending on weather and temperature, this meant there was a chance that the size of the baseplates could force the enclosure to crack as it contracted.
Last time around, the wood milling shop tried adding an extra 0.2mm gap between the edge of the enclosure. After that proved to be insufficient, we asked them to take another stab at it. This time, the samples showed up with a full millimeter gap between the baseplates and the enclosure. Assembly didn't require any pushing or twisting, and the baseplates and enclosure mate beautifully.
We signed off on the enclosure samples and asked the factory to place the PVT (pilot run) order. We expect it to be delivered a couple weeks after the end of Chinese New Year.
As we understand it, the wood milling factory has also obtained and started drying the wood for the mass production run.
The plastic baseplates weren't supposed to be difficult. They're pretty standard injection-molded parts.
You might remember an update last year where we talked about the factory trying to speed up the production process by starting the tooling process before we'd signed off on the baseplate design. The sample baseplates that came back from that tooling were riddled with small issues. Some of the problems were "normal" problems you'd expect to see for the first samples out of a "T0" injection mold. Bits were sunken or warped. The surfaces had blemishes. The parts were a little bit warped. The plastic was too thin in places. Some of the issues were more serious. Internal support structures didn't quite connect as they were supposed to. Features were missing.
We went back and forth with the factory as we updated the baseplate design to remove the ill-fated flip-out feet. That took...a lot longer than it should have. The factory was working to minimize the cost of tooling changes and didn't believe us about a couple of the structural issues we were worried about.
Finally, sometime in November or so, the subcontractor making the baseplates started in on the tooling changes. They blew their first (self-imposed) deadline. And their second (externally imposed) deadline. And then they delivered sample baseplates that were comically bad. Structural supports didn't line up with each other. Places where they'd added extra plastic to reinforce a circular structure were offset from the plastic they were supposed to be matched to. Half of the changes to the right hand baseplate...simply weren't made.
Over the course of the next month, the subcontractor doing the baseplates modified the tooling at least 3 more times in an ultimately futile attempt to deliver usable samples. Sometime around the first week of January, our factory told the baseplate supplier that they had one more chance to make everything right or they were getting fired. The manager of the subcontractor promised to deliver perfect samples on that Friday. And then on Saturday. And then on Monday. The samples showed up on Tuesday or Wednesday. That night, the factory told us that they were going to fire the baseplate supplier and had already started getting quotations from new suppliers.
The final samples from the subcontractor were...closer to what we'd specified, but were still bad enough that the factory thought they were never going to be workable. Our factory had a strong incentive to make this supplier work out--they ended up out of pocket for the deposit paid on the defective tooling.
Our factory ended up sharing three competing bids for the baseplate tooling with us the week before China shut down for the Chinese New Year holiday. The pricing for all three options was within about 10% of each other (and the original quotation). They also all shared the same disappointing tooling lead time: March 15-20.
While disappointing, an 8 week lead time for steel tooling isn't entirely crazy. It's actually pretty normal. And with Chinese New Year eating up 40% of that time, it's nearly miraculous.
We asked for, but never received, information about the three suppliers’ capabilities that might help us with the decision. Ultimately, we didn't push too hard on that because there wasn't a whole lot we'd be able to do with it.
Our salesperson promised to push on all three vendors to reduce the cost of the tooling and to try to cut down the lead time. We told her that while we'd be happy to see a cost reduction, we'd be happier to pay the full price or even a little extra if they could reduce the lead time. Two of the vendors said there was no wiggle room on the lead time. The third said they could deliver it earlier, during the week of March 5. The factory recommended we choose that supplier. We smiled and nodded and ran with their selection.
One of the things we received in the most recent sample package from the factory was an updated set of rubber feet samples for the baseplates.
Our rubber feet are simple compression-molded silicone feet. And they're custom-made for the Model 01. Since they're custom made, one of the things we have to choose is the adhesive we use to stick them to the bottom of the enclosure. The samples we got the last time Jesse was in China just didn't stick very well. Even though we've designed the baseplate to help protect against the rubber feet shearing off, they still came off far too easily.
We asked for a few samples with different adhesives. The factory's rubber feet supplier provided three options, two of which we rejected since they didn't actually stick all that well. The third option, 3M's "200MP" adhesive, does a great job of holding the rubber feet onto the baseplates.
With that selection of the adhesive, we've signed off on the rubber feet for the enclosure for PVT.
Sliding dovetail center bars
Each Model 01 will ship with two bars you can use to attach the two halves of the keyboard to each other. One is flat and the other offers a 7.5 degree tent. To make sure they're sturdy and that they won't crack, the center bars are made of glass-filled nylon. In our torture testing last year, we found that the glass-filled nylon has just enough give that it will let you snap the keyboard apart before the bar breaks. (We do not recommend testing this.)
Over the past couple months, the factory has modified the tooling for the tented center bar to remove the old flip-out feet and has updated the bottom sides of both bars to add a Keyboardio logo and a subtle texture, similar to what you'd find on the plastic of a thinkpad.
(When you're injection-molding plastic, you can choose to make the plastic shiny or to have a texture. Shiny plastic can look nice, but it shows abrasion and wear pretty easily. It's also much more time-consuming to create the molds for--to make shiny plastic, injection molds have to be hand-polished to a mirror finish and to be touched up regularly.)
Even before they sent the box, the factory warned us that the sample center bars we'd receive just before Chinese New Year had a problem. Unlike previous versions, they were badly warped. This is probably due to incorrect settings on the injection molding machines or insufficient cooling before the parts come out of the mold. The shop boss at the factory apologized that he wasn't able to correct the issue before Chinese New Year and promised to tackle it after they get back from the break.
Rails for the bottom of the baseplates
The center bars slide onto two rails that screw onto the bottom of each baseplate. To help you connect the center bars, each rail has an arrow indicating how to attach the center bar. In previous versions, the arrows were backwards. The samples we got just before Chinese New Year were finally correct.
The one issue we ran into with the rails was that the smaller of the two rails is just a little bit too narrow to lock snugly in place with the center bars. We sent the factory a video and they'll be tackling this by tweaking the center bar, rather than the rails.
In the most recent box of samples, the factory provided an updated sample plastic stand. This version fully resolved all of the issues we had with the stand bending or flexing. The only known issue with this design is that the hole for the tripod screw is a little bit too big, leading to the stand sliding around a bit more than we'd like during assembly.
The factory duplicated the sliding-around issue and modified the size of the hole in the drawings, after which we signed off on the design of the plastic stands. The stand tooling should be ready before the baseplate tooling.
Rubber feet for the plastic stands
The most recent samples of the rubber feet for the plastic stands are die-cut silicone. We found them to be a bit slippery on a desk. We've asked the factory to provide us with another option that has a bit more texture. They were to have included it with the last set of samples, but the contractor doing the rubber stamping closed for the holiday before delivering them.
Screw for the plastic stands
The factory has sent us a variety of tripod screws to mount the stands to the baseplates. Most of the options were a steel screw with a molded plastic handle. After a bit of initial confusion, they even fit in the baseplate.
All of the plastic-handled screws felt cheap and we ended up rejecting them. Finally, the factory managed to source a black-anodized unibody metal screw that's reasonably grippy and compact enough to fit where we need it to.
For a while, it looked like we were going to have to fabricate custom tripod screws. Jesse specced out our "ideal screw" and got a half-dozen bids, before the factory found a screw that looks workable. We're pretty sure we're set, but if we're not, we've now got a bunch of other options.
We're expecting the "golden sample" of the wooden stands to show up in a final pre-Chinese New Year package tomorrow. The factory has warned us that some of the wood may be thin enough that there's a chipping issue. At this point, it looks like we might not need to use them for PVT. We may end up selling them as an aftermarket option later if they meet our quality standards.
The metal keyplates were finished in November. The stamping factory needs about a week's lead time to churn out thousands of them.
The factory sent us samples of unpainted and laser-engraved keycaps in the last package of samples. They've fixed some small structural issues with the insides of the keycaps, primarily related to the shapes of the keystems. We haven't yet done a key-by-key evaluation to sign off on the plastics, but should be doing that while the factory is on break.
This month, the factory finally delivered a formal report on the force required to remove each keycap. It ranges from 2 to 2.5 kilograms of force per key. That's enough force that your keycaps shouldn't be falling out, but is a little bit higher than they'd initially promised. The factory still owes us a report on how many times keycaps can be removed and put back in without damaging any of the keyswitches. We've explained to them that it's important that customers be able to swap out their keycaps without destroying their keyboards.
The keycap factory has managed to make the plastic a little bit more translucent. Ideally, we'd like it a little bit clearer, but the current coloring is definitely workable.
Keycap laser engraving has been a bit of a saga. The laser-engraving factory has screwed up in a variety of ways, ranging from swapping the left and right hand key labels to modifying the text on the keycaps. Thankfully, the actual quality of the laser engraving seems fine.
The laser company will take another shot at it when they reopen after the holiday, but we suspect that the only way we're going to figure out what's going wrong is to have Jesse visit them in person and get them to agree to let us set up the engraving templates ourselves.
We've mentioned in previous backer updates that the factory had pushed us toward local versions of some components. In some simple cases, that's fine. In others, it's less fine. In pretty much every case, we were willing to try their versions. Generally, the reasons they were pushing the local versions of things related to availability. Some of the components we specified were hard for them to find in China.
One of those components was the Taiwan-made RJ45 connector for the inter-hand cable. The local version was...almost the same. It was just different enough that it wouldn't fit comfortably on the PCB. And the plastic was...almost as sturdy.
In the end, we found that the locally made RJ45 connectors were just a bit too fragile, so we introduced the factory to the salesperson from the original factory in Taiwan and they've placed the orders for PVT and mass production.
The change in RJ45 connectors meant that we ended up having to slightly update the PCB design. More on that below.
Despite the vendor's best efforts, the folks who make the SK9822 can't figure out why our design is quiet with genuine APA 102C LEDs and makes a ton of acoustic noise with their "SK9822" clone.
The best they've been able to offer is that the quality of power delivered over USB isn't good enough for them and that we can mitigate the problem by adding a _lot_ of extra capacitance to our board in a way that we're pretty certain would make it badly non-compliant with the USB spec. They did end up delivering a custom spin of their LEDs with an additional patch wire inside the package. It took them about a week to deliver a few hundred samples, as they promised. Sadly, it didn't help. They've taken sample boards to the company that designed the IC inside the LEDs, but we're not holding our breath.
At the same time, we've been working with APA in Taiwan, who make the LEDs we originally specified. As we mentioned in a previous update, they're pretty heavily backordered.
Last week, they told us that they'd managed to shake loose enough stock for the first nine hundred keyboards and could deliver them immediately after Chinese New Year if we placed the order immediately. We instructed the factory to place the order and they did so. We're supposed to get a confirmation of the lead time for the remainder of the LED order late this week, but we expect it to be sometime in late March. This is less than ideal, but is likely to fit nicely with the baseplate tooling timeline.
USB Type C Connectors
Since the Type C connector is going to see more abuse than most other parts of your keyboard, it's really important to us that it be something that's expected to take a beating.
We originally specified USB Type C connectors from JAE, a respected Japanese brand. The factory had a lot of trouble sourcing them and pushed local connectors at us pretty hard. Unfortunately, none of the local connectors they showed us came with...the names of the factories making them or test reports proving that they could live up to the 10,000 plug/unplug cycles they're supposed to support, we ended up rejecting them.
Finally, we asked the factory to just call JAE in Hong Kong and find a reseller who could do business with them. The first attempt at that ended up with a quotation of "we might have enough stock" and a price roughly 2-3 times the cost of buying the connectors retail in the US. For a while, it looked like the easiest solution was going to be ordering reels of USB connectors from DigiKey, sticking them in a suitcase and flying to China. Thankfully, a second round of discussions with JAE in Hong Kong shook loose a reseller with stock on hand and a less extortionary price.
The reseller has now put a hold on the inventory we'll need for mass production and the factory will send a purchase order just after the holiday.
The factory already has the Matias keyswitches for PVT in stock. We're in the process of working with Matias in Canada to make sure that the switches for mass production are ready for us when we need them.
Because we needed to modify the PCB to fit the RJ45 connector, we took the opportunity to clean up a few small issues around the placement of the programming connectors inside the case.
The factory's regular PCB fabrication house quoted an absurdly long lead-time for sample boards. Apparently, they really don't like doing prototype boards and only do them for our factory as a favor. Seeed and DirtyPCBs are our usual go-tos for prototype PCBs in a hurry. Sadly, while both companies are located in Shenzhen, they're both set up to ship boards out of China...exclusively.
We ended up asking some Shenzhen-based friends for a recommendation of a quick-turn PCB shop that can deliver domestically in China. They recommended HQPCB and SZ-JLC as two fast, cheap options. HQPCB is convenient because the entire ordering flow is done over the web. SZ-JLC, we're told, prefer to call you to talk about your order.
We provided both options to our factory, along with some pricing guidance. We told them that we expected that 10 pieces of the left board and 10 pieces of the right board could be delivered within 24 hours for a total cost of about $160. We told the factory that as long as they placed the order and the cost was under $220, we'd be happy to cover the cost. In the end, the boards cost $175. The whole thing almost fell apart because someone in the factory's purchasing department saw that the total price was 10% over our initial estimate and they tried to stop the order from being placed. Thankfully, our salesperson overruled that decision.
Last week, the factory assembled sample PCBs and we should have them in hand tomorrow afternoon for testing.
Two vendors have promised the factory sample RJ45 cables. Both were to deliver the week before Chinese New Year. Neither did. Thankfully, RJ45 cables are easy to source, even in custom colors and lengths.
USB Type C Cables
The factory sent us two sample USB Type C cables. As it turns out, both were from vendors Jesse found during a previous trip to Shenzhen and introduced to the factory. The vendors the factory had approached on their own had asked for money to make tooling for the cable in advance of providing samples.
One of the cables was more attractive looking, with a pretty rounded molded Type C connector and connectors that look as if they might be gold-plated. This is a cable we first bought off-the shelf in the markets. The other cable was...blocky. It was custom made for us.
Both cables worked fine and we were leaning toward the pretty-looking cable, until we got to the destructive testing phase.
When we cut open the pretty cable, we found that it was just a four-connector USB 2.0 cable...and that it wasn't particularly well structured inside. On top of that, its electronics were wrong. The USB Type C specification makes a big deal out of how a cable is supposed to signal that it's not set up for 3A fast-charging with a 56k resistor. The resistor inside the pretty-looking cable was...10k, which wasn't what we were looking for.
The blocky-looking cable was provided by a vendor Jesse had met in Hua Qiang Bei, the electronics market in central Shenzhen. When he first asked this vendor if he had a 1.5M USB A to right-angle USB Type C cable, the vendor whipped out a notepad and started sketching what he thought we meant. It took about 5 minutes to explain what we wanted. The vendor ended up calling up a bilingual friend to help make sure he understood. He asked for a business card and said he'd send us email when they'd made a sample.
Lo and behold, a week later we got pictures of a sample cable. The price was reasonable and the leadtime for 5000 pieces was about 3 weeks. We asked about the internal structure of the cable and got pictures of the sample cable cut open.
When we got the sample from our factory, we cut it open and were happy to discover an actual USB 3.1 cable with reinforced shielding. When we tested the resistor, we were happy to discover it was a nice spec-compliant 56k.
We'll be having the factory contact the second vendor after the holiday to start to figure out production.
The factory has been working with a packaging design company. They'd owed us a complete initial proposal the week before Chinese New Year, but weren't able to deliver. They say we're the top priority once they're back in the office after the holiday.
Once we agree on a design, the lead time for custom boxes is about two weeks.
The factory sent us samples of the screwdriver they'd sourced. The butterfly logo looked good. There was only one small problem. While the screwdriver was able to open the keyboard enclosure, it was the wrong size to be able to handle the Model 01's interior screws. This issue is, of course, easy to fix. After the holiday, the factory will reach out to the screwdriver supplier to get something with the right size Philips head.
It's now the beginning of the Chinese New Year holiday in China and most suppliers are already closed until sometime around Feb 10. At that point, staff will start returning to work as they return to Shenzhen from their hometowns. This is typically the time of year when there's a lot of employee turnover, leading to factories needing to staff back up as they get up to full speed.
Jesse's planning to head to Shenzhen for the first two weeks of March, with a goal of supervising the PVT run we had thought was going to have happened this month. As you might imagine, the delay is intensely frustrating to us, though less frustrating than shipping a shoddy product.
The Model 01's firmware has seen a bunch of work over the past few weeks. Since we don't expect to have much to report on the hardware front due to the holiday in China, we're planning to talk about that in next month's update. We, did, however want to share a quick work-in-progress video of an LED effect @algernoone is working on for your keyboard: