How to improve a manual testing process

For low-volume/high-mix manufacturing

Go paperless – save results electronically
Drive consistency – use a sequencer application
Track your WIP – use IDs to trace part movement and history

There are 3 main reasons to improve your manual testing processes:

Production cost reduction
Product quality improvements
Training and skills improvement for your production operators

Manufactures that perform manual steps generally produce parts in low volume, and often with a high mix of part types. Sometimes the equipment used is old and possibly obsolete.

Test steps are often performed in conjunction with assembly steps at the same station, so that a part can be verified as it is being assembled. The improvements suggested below discuss both assembly and test (A&T), but the improvements apply equally well to a test-only process.

Improving a manual assembly & test process for low-volume/high-mix parts is usually thought to be expensive and thus have a long payback period. Or, in the case of old equipment, companies may think it’s wasteful to make updates to the system, even if incremental, since a new A&T station will be needed soon anyway.

Articles around Machine Learning (ML) and Artificial Intelligence (AI) sometimes exacerbate this perception by depicting the need to make massive, company-wide changes to your manufacturing process and people. The effort can seem so daunting that none will take the step. However, there are a few affordable, proven changes worth considering for low-volume/high-mix situations including:

Go paperless – save results electronically
Drive consistency – use a sequencer application
Track your WIP – use IDs to trace part movement and history

These changes are used to collect data associated with 3 areas:

the A&T station,
the parts, and
the operators.

The data can be used to expose waste and potential improvements in the A&T process.

Go paperless – save results electronically

Storing the assembly and test results in an electronic file allows searching and statistical analyses. These results would include both part performance information (e.g. electrical and mechanical readings), and operator performance (e.g., time to complete, part variability).

Why it matters

Production, quality, and design engineers can create a Pareto chart of issues to identify the easiest and/or most cost-effective steps to reduce the occurrence of those issues. Production managers can use these data to identify ways to increase production rates and reduce rework.

Companies often underestimate the cost and time savings of automating the analyses of the results datasets. We’ve seen many cases where the bulk of the payback in improvements to manual processes has been from:

eliminating the hours spent each week manually processing data for their operations team to scrutinize and
the time to generate their customer’s reports.

Further, if you manufacture regulated products, these data can simplify audits because the history of each unit manufactured can be quickly and consistently located.

What to know

The electronic data file is intended to hold the records from each part assembled and/or tested. With a manual test system, people usually assess the measurements to decide the pass/fail status of the various functional aspects of the part, such as the power supply voltage or the force from a hydraulic actuator.

If the process is simply for assembly, operator notes about assembly difficulties or concerns can be helpful in locating common issues. Some companies use predefined codes to make the operator’s job easier in selecting common issues, while a free-form comment field can also be useful for everything else.

Information describing the performance of the A&T station and the operator is useful to assess opportunities for improvement. Start/stop times for each step in the procedure highlight steps where automation could potentially speed up the step, thus increasing the production rate and lowering the cost of production per hour. These start/stop timings will also show which operators might need additional training for specific steps which take longer for them than other operators. Station usage percentages relative to available hours will show when equipment is being underutilized or when more may be needed to meet production goals.

These data can be used to calculate an Overall Equipment Effectiveness (OEE) measure for each production station across part types and operators. The idea with OEE is to grade a manufacturing unit for the following 3 items:

Availability: fraction of scheduled time that the operation is available to operate.
Performance: speed at which the process runs as a fraction of its optimal speed.
Quality: measured by the first pass yield as a fraction of all parts made.

Multiplying the value of these three parameters gives the OEE, which ranges from 0 to 1, or 0%-100% if scaled as a percentage. A value of 100% is very hard to achieve in practice. Values between 70% to 90% are more typical. This metric can be useful as an overall view into the performance of the production assets: both equipment and people.

The data file format can be almost any type if you have the tools and/or expertise to automate the aggregation and extraction of data in the file(s). Common practice puts data into a database, but we’ve seen companies use Excel, with columns being all the parameters captured and the rows being the part serial numbers and/or date/timestamp. Typical parameters across the columns are:

part serial number,
start/stop time for each step,
operator name,
assembly comments,
measurement results and their associated pass/fail criteria and disposition,
overall pass/fail disposition of the part,
rework steps and comments,
and so on.

Drive consistency – Use a sequencer application

Without a following the procedure consistently, the electronic data collected in the previous recommendation may be hard to assess and interpret. A software application can drive consistency by sequencing through the steps in the A&T procedure. This sequencing application will serve 2 primary functions:

to prompt the operator about the next step to execute and
aid in recording the outcomes of each step into the electronic results file.

The purpose of this application is to direct consistent actions and outputs.

Why it matters

Analyzing the results file mentioned in the prior section requires consistency in the steps performed during the A&T operations. Any analysis methods that you develop to extract information to improve operations will need data formed consistently. Operators won’t be able to take shortcuts and any such shortcuts, if approved, can form an updated sequence. This formal suggestion process is a disciplined way to collect input from operators which can then be rolled out to everyone.

This consistency also simplifies (1) training new operators on procedure(s) and (2) training existing operators on new procedures.

What to know

This application need not require making any modifications to the existing A&T station equipment other than adding a PC. However, since a PC is present, making measurements might then be automated via (a) data acquisition from sensors and (b) reading results from programmable instrumentation, rather than having an operator transcribe readings from meters and displays. Then this application can write those data from each step into the results file, minimizing transcription errors by hand-typing directly into a file.

Developing the sequencer application is almost certainly custom since your process and product are very likely different than others. However, most companies make a family of similar products, and this application can usually support multiple product types, allowing development costs to be spread out.

Depending on the level of automation, the application can be very straightforward to develop: from a commercial off the shelf (COTS) product, such as NI TestStand, to a customizable open-platform, such as Viewpoint’s LabVIEW-based StepWise, to completely custom solutions, such as some Excel scripts or an in-house developed sequencer application.

If you choose to automate the analysis of results, this application needs to work together with the data in the results files. Have a discussion between the analysts that produce the reports, the operations team and/or clients that consume them, and the sequencer developer to assure that the data are written to the results file in a way that can be consumed effectively.

Beware of “vendor lock-in” by a COTS application that uses proprietary data file type(s), undocumented file content, or non-customizable capabilities. Such a situation will require a complete re-do of your efforts. Even if you use an open platform, a certain amount of rework will almost certainly be needed if you switch to another application platform, open or not. This rework is simplified if configuration and results files are accessible and/or convertible by other applications. For example, if these files use XML or JSON, most other applications can use them, likely after a conversion from the old to the new format.

Track your WIP – Use IDs to trace part movement and history

You likely already serialize your parts with unique IDs. Identifying each part with a unique ID during capture of the electronic data connects results to a specific part. Furthermore, assigning unique IDs to each station and operator couples the electronic information to the locations the part travels through and people that impact the part.

Why it matters

With IDs captured, the status and location of each work in progress (WIP) is marked for use by the operations team as the parts move through production. For example, you can know when part X has gone through station Y. By searching for status on part X in the electronic data, you might see that this part has been through station Y twice, once on initial assembly and test by operator A and, after some rework at station C, a subsequent one by operator B some 28 hours later. Having IDs in the electronic files brings clarity to how and why parts move through your factory.

What to know

For products that contain subassemblies, you may want to include one or more fields in your electronic records that link the IDs of the subassemblies to the final, rolled-up product. These connections let you trace the manufacturing history of all the subassemblies in a product. This trail of device history is also beneficial if your products and manufacturing procedures are audited by a regulatory body.

For example, if a particular product was returned due to a failure in the field, note can be made of the subassembly that caused the issue. Also, review of the test timestamp and the pass/fail status of that subassembly can help sort out any warranty claims. Finally, if an auditor wants to see how you verified that a particular part was ready for shipment, you can show when, where, and how it was manufactured simply by searching the electronic records for the history of the assembly and any subassemblies.

Is the effort worthwhile – Deciding how much to spend

Deciding how much to spend to improve your A&T process is hard. Ideally, the decision will include an understanding of the costs to:

assemble and test each part,
rework any parts that have assembly problems or fail test criteria,
generate reports for the operational team (amortized across the number of parts made for each report generated)
generate reports for the clients (might be amortized too if your clients can accept a “standardized” report or perhaps each of your clients requests a customized report)
handle parts returned by clients because of poor quality
train new A&T operators on the existing procedure(s) or existing operators on new procedures.

Finding numbers for some or all these costs can be hard because either no one in the company knows them, even as estimates, or the calculation is too complex due to the large number of part types made.

Why it matters

The process of computing these numbers is an enlightening exercise on its own and will almost certainly help you rank areas for improvement.

What to know

Once all the numbers or estimates are available, convert them to a per-part cost as best you can and then add them. From there you can start to identify achievable costs savings per part. For example, if rework is a large relative cost, after you collect results from many parts you will be able to catalog the most prevalent failure mode which can point to solutions such as redesign or a discussion with your supplier about inspecting their parts before shipping to you.

The second aspect that is difficult for many manufacturers is comparing these cost savings with the cost to improve your manual A&T station. If you have the experience, capabilities, and bandwidth in-house, then you can estimate the time and cost to improve the station. Or, you can engage with an experienced test system integrator to work with you to estimate the cost of improving your A&T station. Either way, convert this cost into a payback period based on the savings identified above.

Even more importantly is a discussion to understand your product types and quantities. Especially for low-volume manufacturers, combining A&T improvements across common part types can help amortize the costs by improving several A&T stations at once. The commonality of the parts allows a large base of effort to be spread across several parts with minor customizations applied to each specific part.

Next steps

If you’d like to chat about how to improve your manual test system, feel free to reach out:

Yes, I want to improve my manual test system »

In learning mode? Check these out: