Hardware Product Testing Strategy –
for complex or mission-critical parts & systems

Some hardware products aren’t tested very rigorously or completely (e.g., a box of toothpicks or a bag of balloons) because they aren’t critical to the operation of the environment they’re going into or the process variation on the manufacturing process is so tight that it’s not necessary to test every part.  Other products, like parts of airplanes, parts of automobiles, industrial electrical components and systems, medical devices, industrial equipment and machines are a different story.  This is the world we work in at Viewpoint, developing automated test systems for complex and/or mission-critical (or sometimes just mission-important) industrial components, sub-systems and systems.  These products require a pretty serious testing strategy during design, manufacturing, and sometimes after the product has left the plant.

We’ll answer these questions:

1. How should my strategy differ between design validation, manufacturing test, and post-production field test?
2. What are some good hardware testing methods worth considering?
3. What process elements should be the main parts of my test arsenal to help ensure quality product goes out the door?
4. How do I decide on a reasonable level of automation for my scenario?
5. Once the basics are in place, how do I take my testing strategy to the next level?

Testing during design vs manufacturing

During design, the main focus of your testing strategy will likely be to exercise the design across multiple dimensions (e.g., power supply voltage, temperature, vibration, load) to validate that the design will do what you need it to do once you go into production. You want to characterize the performance of the design to help set what your company considers to be reasonable limits for the spec, or, if your spec is customer-driven, to make sure you’ve got enough margin to make sure you’ll meet that spec once the part goes to production and you’re dealing with all sorts of process variation. You want to understand the limits of the design, sometimes operating the UUT (Unit Under Test) to failure, with various reliability/durability and environmental tests.

Once you transition over to manufacturing test, the focus in strategy switches to verifying a subset of the UUT’s performance and functionality. This is where testable requirements become more important. The focus is generally on the key parameters that indicate the product is of sufficient quality (e.g., pressure tests, controller functioning, capacitance, power quality, switch-over behavior, voltages, waveform check, motor motion profile, etc.) to be shipped to a customer. Since not every aspect of the product is tested, there’s an inherent trade between the risk of shipping bad product vs. production throughput/cost that you will make. The end of line testing system should be designed to minimize that risk while hitting the fastest throughput given ROI cost constraints.

Testing in the field post-production

Testing in the field is for scenarios where a product was tested at the factory, but then after being shipped it needs to be re-tested, either due to repair, refurbishment, annual maintenance check, or concern of damage.  In this scenario, sometimes the same test equipment that was used in the factory is available.  Other times the options are limited, due to factors such as:

1. Harsher environment (e.g., open air, lighting, dust, dirt, rain, snow, hot or cold temperatures)
2. Limited power to supply equipment
3. Limited network connectivity
4. Limited time to perform testing

These factors may drive the need for a specialized field tester, separate from the manufacturing tester.

Hardware Testing Methods

Testing methods vary widely based on the specifics of the UUT, but there’s certainly commonality in the test flow.  For the types of test systems we develop at Viewpoint, this often involves:

1. Stimulating the UUT (electrically and/or mechanically)
2. sequencing through various test steps
3. taking various physical property measurements (e.g., vibration, temperature, pressure, load, voltage, current, etc)
4. analyzing the acquired data to calculate pass/fail for various tests
5. and logging, report generation, and/or displaying the results on a UI.

That’s the general flow.  For each base level component/sub-assembly, there are different methods to test different parts of the product.  For example, you might need to test:

• Electric motors
• Pumps
• Compressors
• Valves
• Chains
• Shafts
• Bearings
• Pistons
• Cams
• Actuators
• Heating and cooling parts
• Electronic controllers

Test methods might include power usage, unbalance vibration, shaft alignment, pull-in current, run-up time, pressure tests, power quality, voltages, etc. We describe test methods for a variety of classes of UUTs in our product testing methods guide.

Hardware Testing Process Strategy

Here are the biggest process elements to set yourself up for success:

1. A test engineer needs to be part of the design team from the beginning. It’s not that you’ll drive the design, but you can be there to influence aspects relating to testability.  The design team may be thinking about testability, but they don’t live the reality of test day in and day out.  Relationships matter.
2. Manufacturing test plan. This is the high level document that specifies how the UUT will be tested during production.  It should be derived from the system requirements.  It will feed the test procedure and work instructions.
3. Manufacturing test procedure / work instructions. These are the details of the manufacturing test plan that an operator will follow step-by-step to test the product.

Automated Hardware Testing – Deciding on levels of test automation during design validation

There are five main criteria to help decide the appropriate level of automation during validation:

Soak and ramp profile

If the UUT needs to sit in a particular state for a long period of time (like hours, days, or weeks), there’s an opportunity for automation.

Number of dimensions & set points to sweep (voltages, loads, speeds, temperature, etc.)

Sweeping even relatively small numbers of set points, say 4 x 16 x 2, is still 128 opportunities for human error in either setting parameters or taking measurements. Often, using Design of Experiments (DoE) techniques can reduce the number of measurement points made in each dimension. Some DoE techniques allow for changes to more than one variable (or factor) at a time. And, systematic errors can be reduced through proper randomization of parts used in the measurements. Review https://en.wikipedia.org/wiki/Design_of_experiments and https://asq.org/quality-resources/design-of-experiments for more detail.

Engineer/technician Safety

When high voltage, high current, or fast-moving or powerful equipment are involved, there are automation opportunities.

Product Safety

When the product is expensive and testing conditions might harm or destroy the unit, there are automation opportunities.

Quantity of units to be validated

People are good at thinking and learning, but not the best at repetitive tasks.  We get bored, distracted, and fatigued.  It’s kind of funny/ironic that we used to rely so heavily on people to do repetitive tasks in factories. How many parts is enough? Using statistical tools like Gauge R&R is common. See https://en.wikipedia.org/wiki/ANOVA_gauge_R%26R for more detail.

Automated Hardware Testing – Deciding on levels of test automation during production

There are three main criteria to help decide how much automation makes sense for your test environment during production:

High-mix vs. low-mix & low volume vs. medium volume vs. high volume

There of course are exceptions to this, but generally when production volumes exceed ~1,000 units/year (we’ll draw the line here for high volume), it’s generally pretty straightforward to decide to automate just about anything and everything related to manufacturing and test.

When production volume for a single testable product line falls below this level, the decision on what to automate becomes a bit trickier.  At this level of production, the dominating factors on decisions to automate seem to mostly be related to:

1. Increasing production volume demand (not necessarily surpassing the threshold mentioned above, just non-negligible increases in demands).
2. Liability for shipping bad product
3. Strategic corporate initiatives (based on core philosophies or competitive advantages)

Note that labor costs weren’t mentioned as a motivating factor, which seems a bit odd.  But it turns out that running a true cost-benefit analysis is too hard for most companies, so they usually don’t go this route for justification.

Operator/technician Safety

When high voltage, high current, or fast-moving or powerful equipment are involved, there are automation opportunities, just like during the validation phase.

Mission-critical product

In this case, there might only be a handful or a few dozen of these parts produced, but if the parts are truly mission-critical, automation may facilitate the rigor required to test the part to the levels needed to establish the trust required for these parts, and automatically produce the reports to justify that trust.

How to get to the next level of product validation

Once you’ve got the basics of automated product validation in place, it can be challenging to get to the next level because often times you just don’t know what you don’t know until you start running tests.  You need serious amounts of flexibility from a hardware I/O perspective, test sequence recipes, and data acquisition standpoint.  At this point, improvements are very application-specific.  If you want to chat about your specific scenario, you can reach out here.

How to get to the next level of Production Test

Once you’re doing the basics from a manufacturing test standpoint, there are generally improvements that can be made for low and medium volume production. The improvements are facilitated by going paperless and increasing automation from an assembly and test standpoint, and are best-suited to those interested in:

1. Reduced human error
2. Improved traceability
3. Increased productivity

If you’re interested in learning more about how to achieve these sorts of improvements, check out StepWise

• Product Validation & Production Test System – For complex Mission-critical sub-system
• Hidden Factory Assessments Lead to Waste and Cost Reductions
• Increasing Test System Automation for Existing Tester to handle Production Volume Demand Increase.

Next Steps

Does your testing strategy include an automated test system? Want to learn more?

Deep into learning mode?  Check these resources out:

• 9 Considerations Before you Outsource your Custom Test Equipment Development
• Test Automation Best Practices – for automated hardware testing
• Product Testing Methods – for industrial hardware products
• Custom Manufacturing Test Equipment for testing in production | End of Line Test Equipment
• Hardware Test Plan – for complex or mission-critical products