Test Automation Best Practices

Utilizing custom test equipment for automated hardware testing of low/medium-volume manufactured parts

What not to automate | What to automate | When to automate

We’ll go over some tips and thoughts on why it matters when considering custom test equipment for automating testing of low/medium volume manufactured parts.

When not to automate testing – Examples of what’s usually not worth automating for low/medium volume test:

  1. Part load/unload with a test fixture: Many parts today have too many physically complex connections (e.g., small and dense cabling) to be automated by machine.
  2. Challenging test points: some parts have test points that are physically challenging to reach by an automated machine.
  3. Very high mix production: there are so many different types of parts to test that finding commonality in the measurement and testing procedures is practically impossible. Automation succeeds best when the test tool can be utilized across many part and/or part types so the cost of automating is dispersed over a large number of parts.
  4. Short run of a fairly simple part: the cost of developing an automated test system will likely overrun the profit from a short run of some parts that are easy to test by hand. Contrast that situation to the production of some components for a completely custom satellite, of which only 2 units will be produced. In this case, you will want an automated tester for design validation, since the cost of the satellite part is likely to be very expensive.
  5. A part is about to go end-of-life: similar to the prior item, the profits from the remaining number of units to be produced is too small to gather enough profit to pay back the development effort.

What to automate – Aspects worth considering for test automation (why you should care and corresponding tips):

Consideration: Automated sequencing of the setups and measurements associated with the testing of a single part.
Why it matters: Reduces the manual labor and error involved in moving from one test to another, such as equipment settings, test point connections, and recording of results.
Tip: What type of test executive is best for you: fully custom, COTS framework, or COTS product? The use of configurable COTS components (hardware and software) is usually the most cost-effective. If you can find a COTS device that satisfies your test needs, use it.
Consideration: Automated selection of the test sequence based on part model.
Why it matters: If the test equipment is used to test many different types of parts, time is saved when the test system can quickly identify the test sequence needed to test the type of part now at the test system.
Tip: Part marking codes, barcodes (utilizing barcode scanners or OCR), and microcontrollers inside the unit under test (where the test system reads a part type ID from the micro via some comm interface (e.g., RS-232, I2C, USB, Ethernet)) can all be used to identify the part type.
Consideration: Automated archiving and report generation of the results of each step in the test sequence in addition to overall part pass/fail disposition.
Why it matters: This reduces errors and time needed to record results and create reports. Furthermore, archiving results from each test step aids in understanding design and production issues, whereas the simple pass/fail disposition does not.
Tip: For parts needing extensive validation documentation (typical of regulated environments, such as FDA and FAA), automated report generation very often offers a huge time savings. We’ve seen situations where automated report generation alone justifies the automation of the test system (e.g., Case Study – Production Test of Large Uninterruptible Power Supplies).
Consideration: Automated Part handling
Why it matters: Reduces errors in mounting and connecting the part to the test equipment (e.g., putting a part in backwards when there is symmetry or nest insertion alignment precision challenges). Sometimes automated handling is faster than a person. It’s usually only used in manufacturing test, not often in product validation.
Tip: Building a machine to handle your part can be costly as compared to having a person do this work. For low/medium volume manufacturing, part handling is rare.
Consideration: Design of test steps for reuse
Why it matters: If you test several part models that have similar test steps, think about abstracting them to a higher level so you can reuse aspects of step configurations and code. This reuse saves time in development and debug.
Tip: If you’ve never designed for reuse, the first attempt usually abstracts at too high a level resulting in one huge test step that does everything.
Consideration: Measurement abstraction
Why it matters: Common measurement types are often used in testing parts, whether of similar models or very different. For example, measuring a static voltage is used in almost every test. For example, your test might need to measure the DC power supply level and the strain on a support beam. Both ultimately result in a voltage reading, even though they measure different properties. This abstraction saves time in development and debug.
Tip: Most companies have a limited set of products models, so the number of types of measurements are limited as well, making this approach more manageable.
Consideration: Hardware abstraction
Why it matters: Certain measurements often use the same type of equipment, such as a DMM or an oscilloscope. However, for example, if the DMM in use today becomes obsolete, having a hardware abstraction layer (HAL) in your test application means you don’t have to rewrite the application in order to support a new DMM. Only the software driver below this HAL layer needs to be developed. All the code above the HAL does not change and not additional debug is needed. This abstraction saves time in development and debug.
Tip: Take a look at IVI for pre-developed HAL models for common instruments, such as DMM, oscilloscope, power supply, and so on.
Consideration: Hardware sharing
Why it matters: If the installation and removal of UUTs takes a significant amount of time relative to the actual test time, consider having one or more test fixtures to connect the test equipment to the UUTs. Then, while installing one UUT, the test equipment can be measuring another UUT. Test equipment costs can be significantly reduced.
Tip: This sharing of test equipment implies the use of switches to automatically switch connections between the test equipment and multiple UUTs. Switches have electrical characteristics that must match the measurement requirements. And, especially if they are mechanical switches, they can wear out.

When to Automate – When do you choose automated testing over manual testing?

At the core, this is driven by the “why” you want to automate in the first place. The most common reasons to automate testing in low/medium volume manufacturing include:

  1. Decrease test time
  2. Human error reduction
  3. Product quality
  4. Labor utilization improvement
  5. Safety of both personnel and product

Decreasing Test Time

Whether for production throughput or getting through design validation, time savings is a major driver for automation. Determine how long each aspect of your test sequence takes. This includes:

  1. part load
  2. assembly steps
  3. fixture, harness, or probe motion time
  4. measurement/switching settling time
  5. measurement time
  6. calculation/analysis time
  7. part unload

Lining these up in rank order will give you a good idea if which components dominate your total test time. You’ll need to make sure the aspects that dominate can actually be improved with automation that’s reasonable for your setup. For example, if your test time is dominated by part placement or assembly steps, robotics may help. But, if a robot can increase your production rate by only a small number of parts per shift, then the extra profit your company makes on those extra parts should be balanced against the cost of the robot. On the other hand, if your test time is dominated by a test tech probing various points and setting up various instruments, basic test automation may help significantly.

Human error reduction

Not that machines never make mistakes (they do), but humans are generally more error prone. Humans get distracted. They get bored. They get pre-occupied. A good example is reading a value off of a multi-meter incorrectly and recording that wrong value on a test results sheet. If the wrong value is within the specified limits, but the actual value was outside the limits, then you may have a false pass on your hands, and you don’t want that part leaving the factory! Or, if the opposite occurs, you could end up scrapping or wasting time troubleshooting a good product, which is obviously bad for your company’s bottom line.

Product Quality

Automated testers help improve product quality mainly in two ways. First, the quality perceived by the end-customer increases when a tester detects a functional problem before the troublesome unit is sent to the end-customer. This usage of a tester doesn’t directly increase product quality, rather it identifies the good units and culls the rest.

The second way that product quality increases with a tester is through better design. By reviewing the test results of all units, the product design team can begin to understand where the product design needs improvement. The ultimate goal of design is that no units ever fail during assembly, making the tester superfluous. In reality, that objective can never be achieved, and testers will always be useful for at least a final pass/fail assessment.

Labor utilization improvement

People are good at performing tasks where creativity and judgment are important. Machines and computers are good at algorithmic repetitive tasks. By giving a test engineer and test operator an automated test system, the best attributes of both machine and human are used.

We’ve seen some of our manufacturing clients boost production by up to 3X while driving down cost or production per part by 30% through the use of sensible automation. Simple automation can often have dramatic effects. For example, enabling the test operator to retrieve a trace from a scope automatically by activating a foot pedal while holding two probes on test points, rather than having to clip the probes and then press buttons on the scope, can speed up a test step by 5X.

Full automation enables a single test operator to do the job of many by having an N-Up tester that allows an operator to load one part while the other(s) are being tested.

Safety

Personnel safety and harm is especially important when the test requires an operator to probe in areas of dangerous high voltage, high current, or moving parts. By using hands-off automatic connections of measurement sensors to test points, such as by proper fixturing or part handling equipment, the human can maintain safety and reduce risk while benefiting from the use of automation. Furthermore, if part failure may be catastrophic and cause harm to the operator, then automation keeps the human away from the part during testing.

Product safety should also be considered when the part is expensive and inadvertent destruction during testing is possible. By adding automatic shutdown capability to the test system in the event of certain limits being exceeded, part damage and possible human harm can be averted. For example, if a part has excessive current drawn on the power rail, the test system can detect excessive power being drawn by the part and shutdown rapidly enough to prevent a bursting power supply due to overheating.

Next Steps

If you need a custom automated test system and either want some help with part of it or you just want someone else to take care of the entire test system development, feel free to reach out to us. Automated test system development is what we do.

Yes, I want to chat about my test automation needs »

Deep into learning mode?  Check these out: