If you ask Adam Wunn how things are going today, he'd say, “Very busy, but the good kind of busy.” That's because not long ago, he worked 70-hour weeks to try to resolve network and station reliability problems.
Wunn is corporate information services manager for Harry's Fresh Foods in Portland, Ore., a growing company that provides fresh foods to restaurants, schools, hospitals and retail delis nationwide. Harry's provides more than 45 tons of fresh food daily to companies like Wal-Mart, Costco, Safeway and Albertsons.
“Our production and administration is tightly tied to the computer networks, and even a few minutes of downtime can cause significant problems with our timetables,” Wunn said. “I can say that, in this company, downtime is not an option.”
Although not an option, downtime for Harry's was an unfortunate reality. During the five years that Wunn has been with the company, he's seen substantial growth in the size and scope of its computer networks. In 1998, Wunn started documenting hardware and operational failures that seemed outside of the industry norm. The situation that emerged was one of an alarming number of untraceable odd crashes and abnormal computer behavior within Harry's networks.
“We had issues all along, but they became worse as we upgraded to newer computers and added much more equipment in our manufacturing plant,” Wunn said.
The failures ranged from disruptive workstation crashes, requiring a quick reboot, to serious server crashes, requiring as much as eight hours of recovery time. In a few extreme cases, disastrous hardware failures led to both downtime and expensive repairs and replacement of the destroyed hardware.
Wunn suspected that power problems might be the culprit, so he purchased and installed TVSS and UPS devices for the network — an investment of almost $4,000. To his dismay, the system failures continued unabated and, in some instances, even the TVSS units failed in a shower of sparks and black smoke. By mid-2000, Wunn was getting serious about finding answers. Harry's was in the early planning stages for a new 75,000-sq-ft production facility, and the problems, which were costing time, money and productivity, needed to be understood and solved.
In early 2001, a chance encounter led to a solution. Harry's operates a retail deli in Portland and, coincidentally, the point of sale (POS) system at that location was experiencing a few problems of its own. Dennis Hadley of Enterprise Information Solutions, the Seattle-area POS company that provided the system, wanted to resolve the problem quickly. He arranged for Keith Nelson, a representative for a manufacturer of power quality solutions, to investigate power at the site. During his visit, Nelson identified a few routine installation errors and noted some neutral to ground noise on his instrumentation. He advised Wunn to simply install power conditioners to eliminate the noise. He did, and the problems were instantly resolved, but Wunn still had the following questions:
Where do these power problems come from?
What specific power problems affect computers?
Are there other considerations for networks?
What are the most likely symptoms?
What solutions should be used?
Nelson used some basic diagnostic tools to help answer Wunn's questions and illustrated the concepts of power line noise and voltage impulses. The TVSS and UPS devices were not effectively addressing his issues. In fact, they were contributing to the problems instead of fixing them.
Nelson explained the complex set of circumstances that develop in the networked environment when a collection of interconnected servers, workstations and peripherals make it easy for power disturbances to enter and exit the network. Then he discussed the potential for the development of ground loops and the value of such time-honored wiring practices as dedicated/isolated electrical circuits.
By late summer of 2001, it was obvious the deli's power problems had been resolved. Everyday crashes had been reduced to just three in six months since the installation of the power conditioners. It was time to find out whether or not the same technology could improve performance at the corporate facility.
Not wanting to duplicate the unsuccessful TVSS experience, Wunn decided to further investigate the issue. He spent hours doing research on his own, gathering information from the Internet and studying power quality case studies. Finally, he scheduled a meeting with power quality experts from Portland General Electric (PGE).
At the November meeting, PGE addressed two key topics: how to handle the existing power problems and avoid future problems in Harry's new building. PGE representatives discussed the necessity of providing adequate capacity and stressed the importance of properly installed wiring. They recommended the installation of a TVSS device at the main service entrance as a way of keeping externally-generated transients from entering the facility, and confirmed Wunn's suspicion that basic TVSS devices installed at the computer were insufficient for Harry's critical applications.
Wunn decided that proper electrical wiring, while extremely important, would only lay the foundation for a comprehensive solution.
“Even with careful planning, wiring is never perfect,” Wunn said. “Later wiring revisions may not always be done correctly, and we can't police every dedicated plug. The deli was a good example. It had dedicated wiring but still experienced a multitude of crashes until power conditioners were installed.”
Wunn then approached corporate management with his proposal to install power conditioners on all network components. Due to the cost and their previous experience with TVSS, Wunn and management decided on a carefully staged approach. In December 2001, they began gradually installing power conditioners while simultaneously comparing network performance to the historical failure data that Wunn had charted since October.
Harry's corporate network consists of 11 servers, 67 workstations, 12 workgroups, and multiple laser printers connected via a 10/100 Base T Ethernet network. Almost all of the computers are Mac-based. Parts of the network are housed in two separate buildings, which are connected via a 10-megabit wireless link.
Two months of data documented 229 separate failures, about 25% of which had occurred in one building and 75% in the other. While the specifics of each failure varied significantly, they could be categorized into four broad types:
Type 1 — Server or workstation crashes or freezes requiring rebooting or reinstallation of operating systems.
Type 2 — Operational problems like unexplained error codes and application failures.
Type 3 — Application software failure requiring reinstallation of the application software.
Type 4 — Hardware failures requiring component repair or replacement.
“We experienced a number of strange problems, including odd colors showing up in Word documents and applications refusing to do simple things like saving a file,” Wunn said. “We'd reinstall the software and things would be fine for a while — then the problems would reoccur.” At one point, Wunn even pursued an issue all the way to a senior Microsoft engineer, who was completely frustrated when he couldn't solve the problem.
Fortunately, the majority of failures were not of a destructive nature. Disruption of Harry's network operations was far more prevalent.
Within two weeks of the installation of the first power conditioners, Wunn witnessed dramatic results. System failures immediately fell by 55%. A backup server, which had crashed as often as six times a week, began to operate continuously. Several users who once called Wunn with daily problems had not needed his help at all. With these encouraging results in hand, Wunn completed the installation of additional power conditioners.
Harry's selected a low-impedance power conditioner with ground conditioning capabilities that incorporates an isolation transformer, a surge diverter, a noise filter and an inductor. The inductor is placed in the safety grounding conductor on the input side of the transformer's neutral to ground bond. The inductor prevents power disturbances in the safety grounding system of a facility from flowing in the “loops” that may be formed by shields and grounds used in interconnecting data cabling.
Wunn said he chases fewer power problems these days. “I know that back in the ‘bad’ days, I had dozens of problems a day that I could not solve,” Wunn said. “Out of desperation, I called in outside help.”
Wunn learned a lot about power quality during the process. He said it is critical to educate corporate management as well as users. “There were many meetings and even more questions,” Wunn said. “Getting to a solution was an evangelical process. I also feel that most of the power community is really misinformed about the need to use power conditioners with computers. They think UPS and power strips are all that are needed. I think in the next five years we will see more and more techs and executives become educated about the problems and how to solve them correctly.”
Since Harry's installed the power conditioners, overall system failures have been reduced by 90%. “Today when problems arise, they usually have a cause and solution that are not power-related,” Wunn said. “My staff's efficiency is up dramatically, and the stability of our network, servers, and client computers is excellent.”
Ver Mulm is vice president, director of marketing, at POWERVAR, Inc., Waukegan, Ill. You can reach him at dennyv@powervar.com.