Funded! This project was successfully funded on September 16, 2012.

Update #2

Test Fixture Development

12 comments
1 like

Thanks to everyone who's backing Teensy 3.0.  Your support means a lot.  I've tried to balance responding to questions and continuing to develop this project, so we'll be ready to ship boards this month.  It's looking like we'll begin shipping the 1st batch within a week of the kickstarter end data.

Here's an update on the behind-the-scenes work going on.

I've spent the last several days building and programming the "bed-of-nails" test fixture that we'll be using to test every Teensy 3.0 board.  The hardware was designed many weeks ago, but the PCBs only recently arrived.  Here it is:

From a usage perspective, the test is simple.  Just press the board onto the spring-loaded pins, and within a second either the green or red LED lights up.  Hopefully it's the green one!

Over the last few days, I've heard comments from people who've been burned by other kickstarter hardware projects that failed to deliver.  Manufacturing and testing a new product in volume is a challenge.  I've been doing this for many years, but so far never published anything about how we test Teensy 2.0 and other boards.  Hopefully sharing these details about the test process might help other projects?

The most interesting part is probably the "nails" section.  Here's another photo:

The pins are part numbers S-100-K-3.5-G and S-50C-TL-3.2-DG-S.  Both can be purchased from Mouser Electronics.  The 2 boards were ordered from Laen's OSH Park   http://www.oshpark.com/  Laen's PCBs work particularly well, because they're from a USA-based manufacturer with much higher accuracy drilling than other low cost services.  For placing the test pins, this matters greatly.  Here's a test board Laen shared with me, where you can really see the difference:  (larger image: http://www.pjrc.com/teensy/beta/pcb_compare.jpg )

Edit: Just to be clear, these photos where provided by Laen from OSH Park (a summary appearing on Hack-a-Day might imply I made these boards, but they are in fact from Laen).  Laen lives here in Portland and he brought these test boards to a local Dorkbot meetup.  I saw them with my own eyes.  Indeed the purple board has much higher drill accuracy.

Here's some more info from Laen:


  • Those are supposed to be 13 mil drills with a 7 mil annular ring.    
  • Seeed averaged 17 mil holes, and an average offset of 4 mils. They also extended the soldermask all over the place.  All 10 boards passed with no shorts or breaks.  Soldermask-to-copper registration was about 7 mils.  Seed requires 8 mil drill-to-copper clearance, and rejected my first board which had only 2.5 mil drill-to-copper clearance.
  • iTead averaged 12 mil holes but had an average offset of 7 mils, and coated all of them with silkscreen.  7 out of 10 of my boards had shorts, even though I'd paid for "100% E-Testing".  To their credit, they offered a full refund.  Seed requires 6 mil drill-to-copper clearance, and rejected my first board which had only 2.5 mil clearance.
  • OSHPark averaged 12 mil holes and had an average offset of under 1 mil.  Soldermask was correct, and had a soldermask to copper registration of 2 mils.  The fab is allowed up to a 3 mil drill-to-copper clearance.  In practice, I've never seen it more than 2 mils off.

To build this top section, the 2 boards are bolted together with standoffs.  I also temporarily add standoffs on the top side, which makes for easy soldering all the pins at exactly the same height.

Each pin is inserted from top side.  The PCB holes are a very close fit, so the larger head on each pin won't fit through the holes.  It has to inserted from the top.  Then the whole thing is turned upside down, where the temporary standoffs hold the top board at the correct height.  Then the pin is soldered first on the bottom side.  Here's how that looks:

This top section is just the pins and 2 LEDs.  There no electronics.  The pins are very strong and last a long time, but if something does go wrong, this assembly is pretty much impossible to service.  When I started using this approach, soldering the pins between 2 PCBs, I quickly learned it's best to dedicate 2 boards to the pins.  Usually little changes end up being needed in the electronics, so it's best to keep the electronics separate so you can unplug the (expensive) pins if the circuitry needs work.  If you later decide to build different electronics for the test, the (expensive) pins can be reused.

For Teensy 3.0, I went with 2 boards for the electronics.  Here's the next board:

Those 7 chips are 74VHC595 shift registers.  Almost every pin is driven by those shift registers.  Most of the test consists of driving all the pins with known patterns (1 pin low, all others high, and then 1 pin high, all others low, repeated for every pin).

On the right hand side is the board presence detection circuit.  When the fixture is waiting for a new board, all the ground pins are connected but all the power pins are off.  The detection circuit applies 44 mV to one of the power pins, and looks for 22 mV or more on another power pin.  If there's interest, I could post a schematic and description.  It's just a simple 2-opamp circuit.  The detect pin is placed about 1/16 inch lower than all the other pins, so the board is detected when that last pin touches and all the others are already engaged.

There's more parts on the back side the deserve a little mention.

Every signal pin has 1K resistor in series, which limits the current in case the PCB under test has shorts or other unforeseen errors.

There's a MOSFET transistor controlling every power and ground pin.  Before test, all the ground pins are on, all the power are off.  Once a board is detected, all the power pins are turned on.  The first test initiates communication with the board, and then turns off power to all but 1 power pin, and cycles through each one to verify each is able to power the board.  Then all the Vcc pins are turned on and that process is repeated for every ground pin, so verify each is connected.

After all power pins are checked, the signal pins are verified by driving a huge number of patterns and reading them back from the processor.  Teensy 3.0 has 5 analog-only pins (4 analog-only inputs and 1 for an external reference).  To test those, analog signals are created by low-pass filtering PWM.  The sections where resistors and capacitors are mixed are 2-pole filters.  The first stage is 1K and 100nF, and the second stage is 10K and 10nF.  That's approximately a 100 us settling time, so it only takes about 0.5 ms to establish a new voltage.  The PWM used is 500 kHz carrier with only 16 steps, which is plenty of resolution for applying test voltages.  The goal is not to test Teensy 3.0's full analog resolution, but to verify that absolutely every pin is connected and none are shorted.  As long as all 5 can be driven to distinctly different voltages, the test is valid.

The bottom board is probably the least interesting, just a Teensy++ 2.0 (modified for 3 volts) controlling the entire test, and a 3.3 volt power supply.

While this test fixture follows the same basic design as the one we use for Teensy 2.0, it's still a substantial project in and of itself.  I've been working the last couple days to get the code finished.  When working on this sort of thing, you can't keep holding a board on those pins.  Here's the "test for the tester" board.

Yesterday I finished all the code which sequences the pins and turns the power on/off to each Vcc and GND pin.  Today's work is reading them back from this board, so I can compare and make sure every single bit matches on every test.

This test fixture is actually 1 of 2 tests.  The 2nd test uses a USB cable, a little USB signal switcher and power measurement circuit, and a Linux-based computer (the same hardware we use today for Teensy 2.0 will work for 3.0).  It programs the default firmware, measures the board's current with and without the LED on, and requires the user to press the pushbutton... so the final bits of hardware that can't be tested on the bed-of-nails are all fully verified.  If there's interest and I have time, I could post another update about that 2nd test.

When we get the first batch of boards back, hopefully in just a little over a week, every single one will be fully tested.

Again, thank you for backing Teensy 3.0.  :-)

jensa likes this update.

Comments

    1. Missing_small

      Creator John Beale on September 11, 2012

      Years ago, in the telecom world we did thermal cycling tests which often captured such a problem, as thermal creep would tend to separate any cold-solder joints over time. Obviously, thermal chambers are out of the question for a low-cost product like this. Maybe a mechanical jig that applies a bit of flex to the board so un-stuck pins lift free? Or a vibration test might do it. Hardly seems worth the trouble in this case, though.

    2. Paul.small

      Creator Paul Stoffregen on September 11, 2012

      Thank you, to everyone here for the great commands, and of course for backing Teensy 3.0.

      I'll try to get another update written up about the USB test. Just last night I finished the code which reports all the pins back to the test fixture, so this one is almost wrapped up.

      @Adrain - I know of one case where that appears to have happened, but I was not able to get the board back for testing. I don't know of any way to test for such a failure. So far, experience with Teensy 2.0 is those types of problems are extremely rare. Of all the hardware troubles, fault USB cables (from 3rd parties) are by far the most common issue.

    3. Fb_profile_picture.small

      Creator Adrian Freed on September 11, 2012

      What a lovely illustration of best practices in product development!
      I am curious whether you have seen a situation when a board tests OK but
      one or more pins on the large IC's lift later on because they weren't soldered right but were touching
      the PCB pad during test? I have seen this on occasion from other suppliers....

    4. Images.small

      Creator Helena on September 10, 2012

      Thank you so much. What I love about Kickstarter is how much you learn, and when people like yourself share such valuable insights, those of us who are hobbyists learn a lot! We tend to struggle learning it all ourselves, and when we hit problems, we just do not have the foundation to work our way out of it. This is so helpful. Thank you again.

    5. Mark%20leavitt%20-%202011%20-%202.small

      Creator Mark Leavitt on September 10, 2012

      Paul, it's great to see how an engineer with a passion for building high quality goodies -- at a very reasonable price -- thinks and works! Glad to be backing you and this project.

    6. Tr2.small

      Creator Thomas Rufer on September 10, 2012

      Thanks for the insight. Nothing too new for me since we develop medical devices (Electronics) on our own, but i have to admit that you did a GREAT job there!
      With testing and with documenting (and also with images to be honest)!
      Unfortunately only a few (this is the first if i recall correctly) cover the testing this deep in their updates.

      Well done! Proud to be a backer! :)

    7. Missing_small

      Creator Andrew F on September 10, 2012

      Did the Seeed PCB order just completely give up on drilling any of the holes? o_O

    8. Missing_small

      Creator John Beale on September 10, 2012

      Absolutely beautiful writeup and photos. I built a larger mixed-signal bed-of-nails test fixture at a previous company and while it worked (after some months), it did not look nearly so clean! I gather for such a small board you don't need board-edge guides or other mechanical placement jig (?) Paul wrote me a detailed personal reply to my questions even before I became a backer. Much appreciated, although I'm almost embarrassed he took time from the project to do so. Anyway, impressive all around.

    9. Missing_small

      Creator Tyler on September 10, 2012

      This was very interesting to read! Never thought about or read about testing something like this. Thank you very much for posting this.

      This test harness seems like an excellent open hardware worthy project all by itself!

      I'd suggest keeping the updates (or at least the question answering) going daily until the project kickstarter closes, then shift to once a week or once every two weeks for the updates after that.

    10. P1010014.small

      Creator Nick Johnson on September 10, 2012

      Excellent post! I've read a bit about bed of nails testing before, but never anything in so much detail. I'd love to see the board detection circuit, for what it's worth.

      Also, is it just me, or does the "test the tester" board look disturbingly like electronics that have been infected with Coryceps?

    11. Jadud-bio-pic.small

      Creator Matt Jadud on September 10, 2012

      Dear Paul,

      Let me say that 1) this all looks awesome, and 2) don't let us pressure you into spending more time on reporting than you should.

      I have seen other projects where they commit to a regular, but reasonable, update schedule. For example, once every two weeks. This allows you to focus on the project itself, but lets us know when you will be communicating an update. In a hyper-blogging culture, this sounds like forever... but in the world of hardware and software, it is plenty-often.

      I would hope other backers would say the same: focus on the project, and keep us in the loop... but not too much.

      I backed your project because you have a clear history in the Arduino/open hardware community, and I don't doubt your ability to deliver this project. That, and I wanted a Teensy 3.0. :) It is because of your history with the community and commitment to excellence that I'm not at all worried about whether things are happening... and I'd rather have regular, bi-weekly updates than day-to-day, blow-by-blow updates that take you away from the important work of developing, testing, and shipping product---even if that is later than you originally expected.

      Keep up the excellent work,
      Matt

    12. Profile_square.small

      Creator Randy Glenn on September 10, 2012

      I'd love to see some details about the second test - this was really interesting to read!

1,572
Backers
$70,874
pledged of $5,000 goal
0
seconds to go
  • Pledge $22 or more
    You selected

    538 backers

    One Teensy 3.0 board. Includes USA postal shipping. Please add $6 for international.

    Estimated delivery:
  • Pledge $30 or more
    You selected

    280 backers Limited (70 left of 350)

    One Teensy 3.0 board from the first batch. Includes USA postal shipping. Please add $6 for international.

    Estimated delivery:
  • Pledge $41 or more
    You selected

    374 backers

    Two Teensy 3.0 boards. Includes USA postal shipping. Please add $6 for international.

    Estimated delivery:
  • Pledge $69 or more
    You selected

    227 backers Limited (23 left of 250)

    Three Teensy 3.0 boards, one from the first batch and 2 more after the first batch. Includes USA postal shipping. Please add $6 for international.

    Estimated delivery:
  • Pledge $79 or more
    You selected

    67 backers

    Four Teensy 3.0 boards. Includes USA postal shipping. Please add $6 for international.

    Estimated delivery:
  • Pledge $108 or more
    You selected

    33 backers Limited (42 left of 75)

    Five Teensy 3.0 boards, 1 from the first batch, and 4 more after the first batch. Includes USA postal shipping. Please add $6 for international.

    Estimated delivery:
  • Pledge $150 or more
    You selected

    39 backers Limited (16 left of 55)

    Seven Teensy 3.0 boards, 2 from the first batch, and 5 more after the first batch. Includes USA postal shipping. Please add $6 for international.

    Estimated delivery:
Funding period

- (12 days)