When a 1,000kW engine gen-set burned up at a cogeneration plant, the owners were left with much more than the cost of repairs. They also received several days worth of stiff utility penalties for failure to provide promised power. Hoping to pass off these costs to a third party, the plant's insurance company called for a forensic investigation. This investigation uncovered some surprising results.
Since the relay failure that caused the fire was due to "wear and tear," as opposed to a design or construction defect, the investigation cleared the relay manufacturer of any responsibility. However, improperly designed and installed oil piping and controls left the consulting engineering firm and installation contractor with an embarrassing liability.
When responding to the alarm, firefighters found the gen-set, which stands more than 5 ft tall and measures about 20 ft in length, engulfed in flames. Once extinguished, they discovered limited damage; affecting only various hoses, building wiring in conduits above the fire, and external wiring.
As I started my investigation, the staff directed me to the service deck above the burned engine. There was a trail of char and oil on the plywood floor originating at a small tank. This oil had traveled to a hole in the floor penetrated by the engine's exhaust pipe. When the oil contacted the exhaust pipe, it ignited on the hot surface, spreading the fire to the engine's rubber hoses and control wiring.
After asking the operating engineer the purpose of a small 15-gal tank, I found it served as a "day tank" and contained lubricating oil for the crankcase of the engine. A float-operated needle valve in the engine crankcase controlled the amount of gravity flow from the day tank to the engine, maintaining a set oil level in the crankcase. A vent at the top of the day tank allowed the free flow of oil. There was a 1-in.-dia overflow pipe that exited the tank at the top and proceeded down to a "used oil" storage tank outside the building.
As the engine operates, it loses lubricating oil. As the day tank gradually drains, another float operates a switch that signals its need for more oil. A 1.6-gal-per-min positive displacement gear pump (located 12 ft below on the engine room floor) starts at the signal from this float switch and then replaces that oil. This pump draws lubricating oil from a large "new oil" storage tank outside the engine room, supplying it through another 1-in.-dia pipe to the day tank until the level rises enough to cause the float to turn off the switch and stop the pump.
The unburned oil path on the service deck clearly indicated the source of the oil was the air vent in the day tank, located about 15 ft from the hot engine exhaust pipe. Outside the engine room, within a containment berm, lay the two large tanks. Plant maintenance personnel reported about 230 gal. of oil were missing from the "new oil" tank (the main lubricating oil source), and there was no measurable rise in the oil level of the "used oil" tank, which should have received an overflow from the day tank. No oil had flowed through the overflow pipe; that system had failed to perform as intended. Instead, the excess oil left the day tank by means of the vent opening and overflowed onto the engine via the opening in the floor. The obvious questions were: Why was there excess oil, and why didn't the overflow pipe remove the surplus oil?
An unusual starter, which included a disconnect, overload switch, and separate contactor, controlled the 208V, single-phase gear pump motor. Operated by the day tank's float switch, a small relay controls the coil of wire creating the magnetism that activates the contactor. This way, only a minimal safe amount of current flows through the float switch. This small relay comes enclosed in a clear plastic housing with delicate electrical prongs on one end that plug into a socket. Due to their size and appearance, we sometimes refer to this type of relay as an "ice cube" relay.
The day after the fire, plant maintenance personnel found the motor contactor's coil still energized through the "ice cube" relay: even though there was no power from the float switch, and the pump was not running.
I electrically and mechanically checked each component of the lubricating oil pump controls. I opened the day tank and checked the functioning of the float ball, arm, and linkage to the switch. I checked the operation of the switch contacts as I moved the float arm up and down. Everything functioned normally. I even checked if someone might have erroneously connected the float switch wiring to the grounded side of the control circuit. Although I found the device wired properly, I checked the wiring to the float switch for shorts and grounds. There were none. Next, I went downstairs to the oil pump's contactor and overload switch, checking each for wiring errors or signs of malfunction. Again, each checked out.
What caused the lubricating oil to overflow? Intending to do some circuit tracing to learn what ultimately caused the fire, I carelessly stuck the ice cube relay back into its socket. I jumped at the spark it made when inserted and the loud clunk of the motor contactor pulling in. This puzzled me. I had just left the float switch open. How could this relay activate the contactor? And why wasn't the pump running? Several similar tries confirmed my discovery.
Looking at the relay's plastic base, between normally open terminals 1 and 4, I found burned and carbonized plastic. I measured the resistance between these terminals, which was less than 30 ohms. The resistance between all other terminals was essentially infinite. What did this mean, and how might it have contributed to the fire?
High ambient temperature and heat from contact resistance caused the initial breakdown of the insulation. The carbonized plastic then allowed a small leakage current to flow continuously, through the pump motor contactor's coil. Initially, this current was not enough to activate the contactor, but it did facilitate further deterioration of the relay's plastic insulation over time. It takes less current to hold the contactor in the closed position than to initially pull it in. On the evening of the fire, the leakage current flowing through the carbon would finally be enough to hold the contactor in, even after the relay dropped out. As oil flowed into the day tank, the float switch control went into the OFF position, but the now effectively shorted ice cube relay caused the pump to continue operating.
The oil kept coming. It quickly rose beyond the level of the 1-in.-dia overflow pipe. That pipe proved to be insufficient to accommodate the incoming oil due to a combination of the restrictions caused by the pipe length, number of elbows, and the flow resistance of two check valves. The relentless pump easily filled the day tank to the top of the air vent, then pushed oil onto the floor. Finally, I'd found the cause and origin of the fire. Now, there were just a few loose ends, such as: Why did the lubricating oil pump stop operating, even though there was voltage at the starter.
The overload (O/L) switch responds to heat, normally caused by motor current flowing through it. However, the heat could be from any source. The pump continued to supply oil through the day tank to the fire, until its thermally activated motor overload switch tripped out: not by a motor overload, but by the radiant heat from the fire! The O/L switch has a manual reset. Thus, the starter's contactor could be in the closed position but with the O/L switch tripped out, there was no power going to the pump.
The engineering design drawings for the day tank indicated overflow and vent pipesbut omitted layout or size specifications. Often, the mechanical contractor must handle these details. Logically, it seemed like a 1-in. pipe should have been able to handle any overflow, but no one had ever tested it. Unfortunately, this became expensive oversight. The plans and physical evidence could not indicate which party was responsible.
The relay had three sets of contacts: each rated to accommodate the current drawn by the pump's contactor. However, installers only used one set. They should have allowed the connecting wires to reach through the adjacent pins, enabling all three sets of contacts: to conduct the current in parallel and reduce the heating, arcing, and wear. Had the installers followed this simple solution, they could have prevented the fire. Had someone periodically unplugged and visually inspected or measured the relay's insulation resistance, they could have identified the defect and rectified the situation by simply replacing it.
