Exclave: Hardware Testing in Mass Production, Made Easier
bunniestudios.comthe original test specification was literally “plug the device into the USB port of [your] laptop, and type in this AVRDUDE command to load code
At the outfit I work for this is pretty much what they were doing for one of the products when I started. It wasn't that they didn't know better (they had other items in mass production); it was that the product used to be such a low volume that typing in CLI commands to program and test it was (almost) good enough.
I spent months automating that to the point of "plug cable into device and watch for Pass/Fail" indication on screen. Now that we're shipping thousands per month it make a huge difference :-)
Testing is a weird thing. The points you test are a function of what you expect may fail in manufacturing. No point in testing something that's extremely unlikely to fail, unless it's required by regulation.
e.g., FDA (Food & Drug Administration) Good Manufacturing Practices state that if you don't build something using a documented and repeatable process then you must test 100% of product coming off the line. If you have a tight manufacturing process, then you can decide what level of testing meets a good balance of safety and cost.
People spend their entire career as Test & Manufacturing engineers and it is quite a complex and interesting field.
>Reputable factories will test 100% of every product shipped.
Frankly, this just isn't true. It depends on your expected defect rate, how your defect rate drifts over time, the cost of doing 100% inspection, and effects of product failures. If you're making safety-critical gear, yes you will probably inspect 100%. If you're making injection molded clamshells for connected consumer products, you're probably only doing inspection in batches.
I guess it depends on the industry but it is somewhat common for electronic components. There are a lot of failure modes in parts, picking, soldering, and assembly. The economics widely favor doing testing at manufacturing time (versus something like at a distributor or dealing with customer RMA which may come with other externalities like brand damage) because boards can be reworked in the factory and you avoid other waste like packaging, shipping, and tariffs.
Back in the day extremely elaborate and massive bed of nails in circuit test rigs would poke the leads of a through hole circuit board to do continuity testing and basic functions checks. Now a lot of that can be done with JTAG circuitry, a serial bus many ASICs implement and chain together to provide less invasive way to similar tests. Xray, thermal, optical scanners are often used in various parts of the assembly. And of course plenty of humans. Test rig design is a big part of any manufacturing process and may be used to verify overall system function like button presses and connector fidelity.
As a separate but related thought, in VLSI: verification design, for both manufacturing and BIST, can easily eclipse the rest of the design in some ASICs in terms of cost/effort/complexity.
Software engineering is kind of the odd duck in that it is frequently a shoot first ask questions later pursuit in most organizations (funnily, even commonly in places like semiconductor companies that invent and progress new formal methods). There are plenty of decent tools like Lamport's TLA+ that aren't widely used in addition to more familiar aspects of code test.
For another option, see OpenHTF from google, though this targets quite complex devices: https://github.com/google/openhtf
I am starting to use OpenHTF at work. I was not aware of Exclave. Seems like I have interesting homework to do. This blog post by Bunnie was a very nice read.
I love Bunnies stuff. Reading about his reverse engineering work on the Xbox was one of the reasons I chose to study computer science. Excellent article, wish there were more people out there like him.
Fantastic, thank you Bunnie & crew, well done.