You worked hard to convince upper management that the existing UPS system supporting your facility was nearing the end of its life and needed to be replaced. After spending several years of planning and investing hundreds of thousands of dollars, the system was finally replaced with a new state-of-the-art unit. The new UPS system was thoroughly tested, and everything worked as designed — job well done.
A few weeks later, thunderstorms begin to roll through the area, and there are electric utility power outages. Your phone rings. The building lost power, and the UPS could not support the critical load until the generator came online. Your heart sinks. This is a nightmare scenario. However, you have been doing this for a while. This is not the first time a UPS unit you’re responsible for has failed. You know there’s a reasonable explanation for this system failure, and the UPS manufacturer will identify the problem and correct it. There will be hard questions from management and end-users, but everyone understands these things happen. It just needs to be fixed as soon as possible.
A technician checked the batteries and downloaded the events log. After a thorough review of all information, there was no identifiable cause for the failure. The unit just turned off. Over the next several months, more electric utility outages occurred. Sometimes, the UPS would support the load; other times it wouldn’t. The mystery continued.
Background
The original UPS unit was typical of its time — a 480V, 300kVA double-conversion unit with input and output transformers. It was wired in a 3-phase, 4-wire plus ground input and output configuration. This meant the system had an input and output grounded conductor, often referred to as a neutral. These systems worked very well and allowed for different input and output configurations.
The new system did not have input or output transformers and was wired in a 3-phase, 3-wire plus ground input and output configuration. There was no neutral connection, which meant it could not support line-to-neutral loads. As most of these systems feed power distribution units or other transformers, this is not a problem. Transformers are separately derived systems providing a neutral that will support any single-phase loads.
Troubleshooting
To identify the problem, a load test on the UPS system and generator was performed. The critical load was shut down and disconnected from the UPS output. A resistive load bank was connected to the UPS unit and artificial load added to the system.
The input circuit breaker to the UPS automatic transfer switch (ATS) was opened to allow the system to operate as designed. During this test, the system operated as expected. The UPS unit operated on batteries while the generator started and came online. Once generator power was at an acceptable level the UPS unit continued to support the load and recharge the batteries. No issue could be found that would prevent the UPS system from working correctly.
Over the next few months, several more power outages occurred. Sometimes, the system supported the load; other times the UPS unit would shut off, resulting in the loss of the critical load.
The UPS manufacturer installed power quality analyzers (PQAs) on the input and output of the UPS unit to capture data during an outage. The hope was they would capture something that would point to the underlying problem.
While waiting for the PQA to record events during an outage, our company was contracted to review all of the data collected to date. The customer wanted a new set of eyes to review everything before they made the decision to replace the unit.
One of our first tasks was to review the event logs of equipment in the critical power electrical system. During that review, we learned that if an outage lasted only a few seconds, the UPS would record the events as an electric utility outage but would continue to operate. However, if the event lasted longer than approximately 7 seconds, the UPS system would shut down. The only alarms provided by the UPS system were UPS AC INPUT FAILURE and INVERTER OFF.
One issue involved with troubleshooting complex systems relates to the number of people involved and their level of expertise. There is often a learning curve between the subject matter experts — each learning what the other knows and being sure they speak the same language. If a certain amount of respect is not established early in the process, it can lead to delays as people try to protect their company. There is also the problem of educating stakeholders who are not subject matter experts in this area, but often need to brief upper management.
This became a delay at this site because of a history of a ground fault problem. Certain contractors would sidetrack important conversations by revisiting this old problem. They wanted to be sure that “blame” was never assigned to them or the equipment they sold. However, the ground fault problem had been solved several years earlier and documentation provided regarding the solution. The facility manager explained this had not been an issue for several years.
As part of our investigation, we had planned to perform a detailed site inspection and review of the installation for code compliance and industry best practices. This would establish a baseline and prove there were no installation issues, hopefully putting to rest the concern that grounding and bonding of the original electrical infrastructure was done correctly.
During our review, we found that the neutral (grounded) conductor was still connected from the main switchgear and generator, through a 4-pole transfer switch to the input and bypass switchgear of the UPS system. The neutral was not run to the UPS input or output, but was instead connected to the maintenance bypass panel. From the maintenance panel, it ran to the UPS distribution panel. From there, it was also connected to Panel EL (Fig. 1).
The fact that neutral wires being installed and connected in the system was not in and of itself an issue. In a new installation, the neutrals would not have been run to save money. But because this was a retrofit — and the wires were already there — it would have cost more to remove them.
We also checked for current flow on the ground and neutral conductors. Since the UPS system is a 3-wire system — and would not allow for line to neutral loads — there should be no current on the neutral conductor. Ground current should only exist during a fault. However, we did find current on both neutral and ground conductors. We traced the source of this to Panel EL, which contained several single-phase circuits with neutral conductors. Later, these loads were found to be emergency lights.
Before we verified and removed the single-phase loads, the site experienced a power outage that resulted in the loss of the critical load. This time the PQAs captured the event. The data collected on the output of the UPS unit was indicative of a phase-to-ground fault on an ungrounded system. Figure 2 shows B phase potential go to zero with respect to ground and A and C phases increase in potential with respect to ground.
After several conversations with the UPS manufacturer, we now understood how this specific unit handled line-to-ground faults. When the load was being supported by the inverter and bypass power was available, the inverter would go into current limit mode and transfer to bypass. The overcurrent protection devices would then operate, protecting equipment and personnel. If the load was being supported by the inverter while operating from batteries, logic would turn off the inverter.
We established that during electric utility operation the line-to-neutral loads were supported through the neutral conductors (where available) and through the UPS input and output equipment grounding conductors (EGCs) when the neutral was not available. When the UPS was on batteries, the logic sensed the ground current as a ground fault and shut down. The system did not shut down on every outage because of a small delay in the logic to be sure there was a legitimate ground fault, which prevents nuisance trips.
The load of the emergency lights was only several amps and not large enough to operate the overcurrent protection devices. The UPS logic, however, was much more sensitive. When operating on batteries, it sensed this current as a ground fault and would shut off the system to protect equipment and personnel.
Several weeks after the emergency lighting circuits were moved to a non-UPS panel, an outage occurred that lasted long enough for the generator to start and carry the load. The backup power system worked as designed.
Recommendations
After the cause of the problem was identified, several additional recommendations were made to prevent this from occurring in the future, including:
1. All single-phase loads should be removed. This is required for the installation to meet the UPS vendor requirements and local and national codes. The UPS manufacturer had inquired several times if there were any line-to-neutral loads connected to this system and were assured there were not.
2. All neutral wires be de-termed and secured in all panels and the UPS ATS. If any line-to-neutral loads were added in the future, they would not operate immediately. This would indicate to the installers that something was wrong.
3. All panels on the UPS output ahead of the transformers should be clearly labeled that no neutral loads are allowed.
Conclusion
There were delays in identifying and correcting the problem at this site. The first was the intermittent nature of the problem or the appearance of the problem being intermittent. Some of these delays were due to the number of vendors and contractors involved in the investigation. Although each vendor and contractor wanted to solve the problem, they also believed it was not their issue; therefore, they should not pay to solve it. The UPS manufacturer, in this case, did work diligently to assist in solving the problem.
The history of the ground fault at the facility distracted from identifying the root cause. Had everyone understood the ground fault problem had been corrected several years earlier, identifying the load drop issue may have moved faster. Finally, the troubleshooting method used hid the problem. By shutting down the load and removing it from the UPS system during testing, the problem was also removed. This allowed the system to operate correctly.
It’s best to use the site’s load for testing and troubleshooting, when possible. Had the actual load been used during testing, the problem would have been identified earlier. There would have been an outage to the critical load during testing, but the customer could have been prepared for it. Since they chose another option, they experienced several outages at inopportune times because of the length of time it took to find and correct the issue.
When replacing any system, extreme caution should be taken to ensure the existing infrastructure will support it and that the new system will function correctly.
DeLauter is the operations manager for Static Power Conversion Services, Inc., in Columbia, Md. He can be reached at [email protected].
