Four months later, it tripped again?
As I said in the video-clip, I hope you see what is unfolding. What we call "problem-solving" is a misnomer -- we have not solved anything. All we do when we "solve" a problem is delay the time it takes to reappear! The deeper we dig, the more time we have before the next failure. In the above case, the effort expended gave them 4 months before a similar trip.
It's as if we cut off the top of the iceberg below the waterline, only to have the remainder of it bobble up and resurface. The deeper the submerged cut, the longer it takes to resurface. But just as an iceberg will always reappear until it's gone, the situation will always reappear until its is entirely eliminated.
Let's continue the example. The same people that responded to the first failure responded to this second failure. The operator, maintenance crew, machine shop, and area supervisor knew what to look for this time. They quickly confirmed that it was another high vibration trip. They disassembled the compressor, shipped the components to the machine shop, and confirmed another bent shaft. However, this time the area supervisor quarantined the compressor, including the oil reservoir. When he checked the oil, he found substantial amounts of water in present in the oil. Apparently, the oil pump had been ingesting and distributed water along with the oil, providing inadequate lubrication to the unit. Because of this water in the oil, the high speed shaft and impeller, rotating at 30,000 RPM, heated up and eventually bowed from the heat. The supervisor had just found the fifth domino.
But now the supervisor was determined. He looked for a source of leakage into the oil reservoir to explain the presence of water in the oil, but could find none. Finally, he walked out to the maintenance shed on the other side of the plant (he hardly ever got out that way) and looked for the barrels of oil which were used to replenish his reservoir. He could not find any oil drums inside, so he went to the back of the building. There, totally exposed to all the weather, were about 25 different drums of oil. It looked like a dump zone rather than a storage area, but he walked over to inspect the "situation." To his disgust, he found that many of the oil drum lids were missing! And many other lids had been replaced by wads of paper towels. Only a few drums had their original lids intact.He found the oil drum that was used to replenish his compressor. It was one of the barrels without a lid. He siphoned some oil to get a sample for the lab, but there was no need to go to the lab. Since he got his sample from the bottom of the barrel, it was almost totally water. He had just found another domino (see drawing).
By this time the supervisor was livid. He tried to contact the person who was responsible for the storage of the oil, but he couldn't find anyone who would admit responsibility. Finally, in total desperation because of the mess he found, he traced down the oiler who had been lubricating this compressor. After sharing what he found with the lubricator, the supervisor told him that from now on he was to get the oil from a new drum which had been placed right next to the compressor. The supervisor decided to purchase and store the oil by himself to assure that this would never happen again.
Finally, after purchasing new components and rebuilding the compressor, they restarted it knowing that they had solved the problem once and for all!
This is a true story that occurred in the mid-1970's. To my knowledge, that compressor has never failed in that manner again. There's only one problem. In 1982 the plant was shut down, closed, and then dismantled. It was loosing too much money. Although the compressor was kept running, many other equipment experienced increasing failure rates. You see, the situation had never been totally resolved, nor even understood. The iceberg had never been totally removed -- it simply resurfaced over and over again in other places. As it grew, it totally destroyed the profitability of that plant. They did not dig far enough! They did not address the underlying, latent causes. They only postponed the inevitable.
I was just getting started in this field in the 1970's. I was not as aware of what was going on behind the cloak of the unknown as I am now. However, because I was very familiar with that site, I'd like to make some retrospective comments about the remaining causal factors of that situation.
Firstly, the oiler who had been refilling that compressor reservoir knew about the open oil drum all along. This facility only had 2 or 3 oilers, and they were the only ones who dealt with the oil drums behind the maintenance shed. Although they might not have been responsible for storing them outside, they were certainly responsible for and knowledgeable of the numerous open lids.
I say this not to try to find blame, but to point out one of the most important facts of things that go wrong; that the latent, or underlying causes of all failure are known ahead of time. They are not known by everyone, of course. For example, the area supervisor did not know of the open oil drums ahead of time. But the oilers did! The oilers also knew they were doing things too quickly in general, taking shortcuts. In fact, if you think about it we all know of things which ought not to be happening.
We all walk around in full knowledge of the latent causes of all our failures.
- What we call "problem solving" is a misnomer. All we do when we "solve" a problem is delay the time it takes something similar to appear.
- It is vital to "freeze all the evidence," before it evaporates (oil in the oil drum had originally been drained so that it could not be checked).
- The latent, or underlying causes of all failure are known ahead of time.
- We all walk around in full knowledge of the latent causes of all our failures.
