Glitches in plant data systems often bring production to a complete stop. When that happens, how do you respond?

Your equipment is down. Production supervisors anxiously watch the maintenance electrician punch soft keys. The controls engineer, off to one side, feverishly leafs through a battered book of ladder logic diagrams while idled operators sip coffee. Overlooking the scene from a glass-paneled office, manufacturing engineers glance nervously at their watches. The plant manager arrives with the corporate consultant who will determine if this particular plant management team will launch the new Manufacturing Resource Planning (MRP) system. This latest corporate venture - newly integrated with the plant equipment control system - will send real time data to corporate headquarters, parts suppliers, and customers. So everybody, everywhere, immediately knows this plant is not turning out product. As you might imagine, the tension level is high.

Unfortunately, nobody in maintenance knows that today's problem lies with the newly installed barcode system, part of Information Systems' domain. This system is in the hands of a network technician working by himself in a tiny room stuffed with equipment racks. There, among the spaghetti cabling and blinking lights, he is desperately checking routers and running diagnostics in hopes of fixing the problem.

How do you resolve such a situation quickly and permanently? The answer is not to run diagnostics. What you need is a methodical approach that allows you to solve root causes of problems.

The weak link. Seemingly small, isolated mistakes in network installation or maintenance quickly turn into networked disasters, which usually show up as I/O communication errors. Thus, you need to be familiar with the networks that link your equipment and control centers. This is less daunting than it might seem, because even a complex control system consists of simpler subsystems (see Sidebar below). Let's begin our troubleshooting there.

Visual inspections. To ensure the network is doing its job, check the cable installation, termination, grounding, and polarity. Visual inspections are the quickest way to accomplish this.

Inspect the cable installation. Did the installers of the original system and any subsequent upgrades choose the most direct route to minimize cable length? Doing so reduces noise, signal loss, and interference. Ensure network cables aren't in the same wireways as power-handling cables. They should run in dedicated wireways that carry only low-voltage or communication cables - rerouting could solve your network problems. Also, keep network cables far from sources of electromagnetic noise: arc welders, gas discharge lighting equipment, carbon arc furnaces, high-voltage and/or current switching equipment. When you do rerouting, tie-wrap network cables closely to supporting equipment - but make sure those tie-wraps don't dig deeply into the cable jacket. Avoid strain on connectors by leaving slack in the run.

Check network cable terminations. Often, you can trace a network problem back to a missing or improper cable termination. Terminate (with a resistance) both ends of a bus network and any unused tap or line from a hub in a star or ring network. This prevents reflections - as long as the resistance for each termination is correct and the connections are secure.

Look at network cable shielding. Should the shields of coaxial cables in a network connect at some point? Should you ground them? That depends on which protocol (Ethernet, Token Ring, Field Bus, etc.) you are using. In all cases, never ground shields at more than one point.

Check network cable polarity. Are transmit and receive pairs "swapped" at opposite ends of each cable? If the network supplies DC power, ensure the polarities are consistent at each connection. Verify correct polarity at each device, and verify each connected device has adequate operating voltage.

Testing. You can test for shorts, opens and grounds using an ohmmeter. Test for shorts between all wires with cables disconnected at both ends. All resistance readings should be infinite. Test each wire for unintentional grounds with cables disconnected at both ends. Again, all resistance readings should be infinite. Test each circuit for opens with the two circuit wires jumpered at the far end. All readings should be zero or close to it. Using the wire color codes, check for reversed circuit polarity, uncrossed transmit/receive circuits, and split transmit/receive pairs.

If possible, use a reflectometer to detect and locate cable and connector defects, as well as unterminated cables. Test for near-end crosstalk (N.E.X.T.), using a cable tester. It will transmit test pulses on one pair and receive on another pair to identify unacceptable coupling between transmit/receive circuits in multi-pair cables.

More advanced troubleshooting. If the installation looks correct and tests don't reveal the problem, you'll need to do some more advanced troubleshooting. The following procedure will help you be successful.

Verify symptoms:

• Personally check the system operation problems;

• Investigate circumstances under which problems appear or first appeared; and

• Break the system down into its components: power supply, I/O hardware, controller, logic/program, or network communication.

Use system interface to locate device or network segment problem:

• Interpret the error messages;

• Verify that devices booted correctly and programs loaded normally;

• Determine which devices or network segments are not communicating;

• Check address assignment tables.

Inspect hardware:

• Field devices may be the problem. Check them for correct operating voltages, adjustment, setup, configuration, and wire connections;

• Inspect cables and cable connections; and

• Check diagnostic lights: Interpret colors and flashes on network cards, modules, hubs, and smart I/O devices.

Analyze network performance. Use network diagnostic instrumentation or programs to identify:

• All network nodes and their addresses and status (whether they are communicating or not);

• Non-communicating network segments;

• High traffic problems;

• Protocol problems, such as collisions and transmission errors; and

• Problems with hubs, repeaters, routers, and switches.

Your best tool. We all know integrated manufacturing systems stop if the data communication network fails. Yet, we sometimes forget about the network between people. When operations and support people work together to solve problems rather than use their energy to point fingers, uptime goes up while costs go down.

Encourage people in operations and maintenance to identify common goals and barriers to reaching them. The synergy from such an environment contributes to the bottom line - a serious consideration when managers must determine which plant to close.


Sidebar: What Constitutes a Control System?

Every control system has four main components, including:

• An input or field device (e.g., a proximity switch), which provides process information to the controlling device or operator;

• A controlling device, which receives input from field devices, makes decisions based on its program, and sends its decision to an output device;

• An output (e.g., a contactor), which receives a decision from the controlling device and makes something happen in the equipment (e.g., start a motor); and

• The cable network, which connects all the pieces and passes data back and forth.