When those of us who work in the critical power industry talk about data centers, the “giants” typically come to mind, such as banks, credit card companies, colocations, and social media platforms. You’ll find countless articles written about large Tier IV infrastructures; however, many of your customers are not billion-dollar data centers. But that doesn’t mean they’re not just as important. These customers support payroll, email, and phone systems. Some are communication systems for first responders, and all are imperative to everyday operations in society. When their processes are lost, it negatively impacts public welfare.
Battery failure
One case in point involved a customer that experienced a loss of power to its critical load in August 2017 when a line of thunderstorms moved through the area, resulting in an electric utility power outage. When a battery failed in the UPS system, it brought operations to a standstill.
This customer had a 200kVA uninterruptible power supply (UPS) system supported by a single string of 800Ah VRLA batteries installed in 2011. In 2016, our company recommended the batteries be replaced due to age. In March 2017, we recommended the plant (all strings, jars, and cells that make up a battery system) be replaced because internal ohmic values documented during the semi-annual preventive maintenance program showed a significant increase in value. Table 1 is a comparison of the baseline readings from 2011 and those collected in March 2017.
As the thunderstorms moved through, and the critical load lost power, a technician was dispatched to determine the cause of the failure. When he arrived on-site, he found the UPS system in bypass mode and the critical load supported by electric utility power. The technician recorded the internal ohmic value and voltages of all jars, noting that several had failed (Table 2).
The technician also noted several jars had swollen so badly that they could not be removed without cutting the cabinet apart (Photo). To repair the system and return it to operation, all of the batteries would need to be replaced. A new cabinet would also be required, increasing the cost and complexity of the plant replacement.
When batteries are in stock, they can be ordered and installed within several days. Unfortunately, this particular battery had a lead time of 90 days. To compound the problem, the cabinet was built specifically for this model, and there was no alternative with the same form and fit. The only option to return the UPS system to service was to install a temporary plant rented from a local battery vendor.
The cost to replace the batteries in March would have been $40,000. After the failure (in August), with the expenses of rental batteries, a new cabinet, and additional labor, the total cost exceeded $70,000, which doesn’t include financial impacts from the loss of critical operations.
Preventing failure in the first place
So how can these situations be avoided? Test equipment can be used to record and save a battery’s float voltage and internal ohmic value readings. This data can then be compared to baseline readings, the plant average, and manufacturer guidelines. If the new readings begin to stray too far from the baseline readings, then you can take action before the system becomes unreliable.
Unlike vented lead acid batteries, generally called flooded or wet cells, visual inspections of the plates are not possible with valve-regulated lead acid (VRLA) batteries; the cases are not clear. Instead, we must rely on test data when evaluating their health.
There are two basic ways of determining if a VRLA battery is failing: its float voltage and its internal ohmic value. Each jar has a specified float voltage range. If the voltage reading is below this range, then it’s a sign the battery is weak or failing. If the internal ohmic value has increased from its baseline reading or is higher than the plant average, you’ve got problems too. The Institute of Electrical and Electronics Engineers (IEEE) recommends that when any cell’s internal ohmic value increases by 30% to 50% from its baseline reading, a deeper investigation should be completed. This includes a performance test, cell replacement, and even complete replacement of the plant.
A general guideline is to replace VRLA batteries every five years. However, the length of time is dependent on the specific battery model, the history of that model, and the operating environment of the plant. IEEE 1188-2005 also recommends performing a battery capacity test approximately once a year. If the capacity has fallen below 80% of the manufacturer’s rating, then the plant should be replaced.
Making maintenance work for you
It’s important to note that this customer performed semiannual maintenance on the UPS system, which included recording voltage and internal ohmic values as recommended by the manufacturer and IEEE. The problem is it did not replace the batteries when they showed signs of degradation. If they had replaced them when recommended, a critical step in maintaining system reliability, then this outage would have been prevented. Is there anything else this customer could have done without spending the money for a Tier IV infrastructure or even Tier II arrangement?
Several factors can increase reliability in addition to performing routine maintenance as recommended by a manufacturer and IEEE, including: using multiple strings, installing a monitoring system, or using a modular battery topology. The options discussed here, while useful, are not a substitute for a solid preventive maintenance program, including replacing batteries when at end of life.
The major disadvantage of a single string of batteries is the multiple single points of failure. If a battery string is thought of as a chain, each cell is a link in the chain. Many modern UPS systems require 240 cells in series to make a battery string. In our analogy, that would be 240 links. If one cell fails, the entire string fails — just as if one link breaks in the chain, the entire chain is broken.
If two chains are used and a link breaks, the other will continue to work. If multiple strings of batteries are in parallel, when a cell fails, the other string will maintain operation. Of course, this redundancy comes at a cost. Multiple strings of batteries have a larger initial investment and maintenance cost, including the replacement at end of life. They also have a greater footprint, which reduces space for IT equipment.
Another option that can assist in maintaining a battery plant’s readiness is a permanently installed monitoring system. Depending on the system used, it will continuously watch each jar’s float voltage and take sample internal ohmic readings at set intervals. During discharges, the system will monitor each jar for inconsistency. When any values fall outside of preset parameters, alarms can be generated and sent via email, SMS, or to a building monitoring system. These systems can be expensive and are most effective when purchased during original install or when replacing the batteries.
An alternative option growing in popularity over the past decade is modular batteries. Modular batteries are configured with small battery jars inside a larger container. Several of these containers or modules make a string. Because the jars are smaller, multiple modules are paralleled in a cabinet, often in the same size footprint as a single string of traditional VRLA batteries.
Modular batteries possess additional advantages over traditional battery solutions. The UPS systems contain a built-in monitor that records applicable data during a prescheduled self-test or during a discharge. When the system finds a weak module, it will generate alerts allowing the suspected module to be identified and replaced.
Modular batteries can be replaced relatively quickly as compared to a traditional VRLA plant. For example, a UPS system loaded to 80kW with 10 minutes of battery backup could have its battery replaced by one person in as little as two hours. Additionally, the system could stay online, allowing the critical load to be protected by the UPS system during the replacement. A more traditional VRLA plant would require two people and approximately two hours to complete the battery plant replacement. The system would also need to be bypassed, placing the critical load at risk.
There are two major disadvantages to modular battery design. First, they are typically 1.5 times the cost of traditional battery solutions. Second, you must stay with the original manufacturer. Thus, this eliminates the competitive bidding process when replacement components are required.
As with all things, there are advantages and disadvantages to each option. Each needs to be weighed to decide what is best for each particular installation and customer.
Data centers supporting large customers spend the money to deliver high reliability in their critical power systems. The smallest outage can cost millions of dollars in revenue. They know this and design, build, and support their infrastructure in ways to provide it. But a majority of UPS system owners do not need Tier IV redundancy, and many do not have the budget for this type of system. It’s our responsibility as engineers, electrical contractors, and service companies to inform our customers of options that could improve the reliability of their system within their budgets.
De Lauter is an operations manager with Static Power Conversion Systems, Inc. in Columbia, Md. He can be reached at [email protected].
SIDEBAR: Terms to Know
It’s helpful to understand the various terms that are used when discussing batteries and their condition. Certain words are often used interchangeably, which might lead to confusion.
- Cell – The smallest electrical structure of the jar comprising plates, separator material, and acid to produce a voltage potential.
- Jar – A single container holding the cells often referred to as a battery. Typically a valve regulated lead acid (VRLA) jar consists of six cells, producing 12VDC.
- String – The minimum number of jars connected in series to form a DC voltage to support a system. For example, 40 jars connected in series will result in a nominal 480VDC string.
- Plant – All strings, jars, and cells that make up a battery system. If two or more strings are connected in parallel, all would be in the battery plant.
- Jar float voltage – The voltage of an individual jar when the plant is fully charged.
- Internal ohmic measurement – A measure of the internal impedance, conductance, or resistance of a jar or cell used to help determine the overall health of the jar or cell.
- Baseline – An initial set of recorded data that can be used as bases of comparison.
- Valve-regulated lead acid (VRLA) – A lead-acid jar that is sealed except for a valve that opens to the atmosphere when the internal gas pressure in the jar exceeds atmospheric pressure by a pre-selected amount. These units are often referred to as maintenance-free or sealed batteries.
