Insane Hydraulics
Bold and audacious blog about fluid power
I did an interesting assistance call in November which, as usual, involved a piece of mining equipment. I want to go through the troubleshooting process and show you how, despite my "expertise", equipment knowledge and state of the art diagnostic tools, I still found the malfunction only by sheer dumb luck.
So, I got a call from our "beloved" mine asking me to have a quick look at a malfunction that "seems to be hydraulic, although we are not one hundred percent sure..." How very original!
I soon discovered that the piece of equipment "in question" was an old friend of mine - the GEHO paste pump, exactly the one that helped me make one of the silliest mistakes of my "industrial life".
For those of you who don't know - the business end of such a system is essentially a positive-displacement two-piston pump that shoves paste fill through a series of pipes into mined-out galleries, and any pumping problem becomes a serious issue because it slows down production and can result in pipe blockage, which is an adventure in its own right! A very expensive one!
But we don't care about paste, do we? It is the hydraulics that runs the two cylinders that matters, and I must tell you that this is a very well designed and executed closed-loop drive with intelligent control. The PLC makes sure that the two cylinders operate synchronously in a manner that reduces the paste pressure pulsations, which means not only shifting the rods back and forth, but also following a proper deceleration and acceleration patterns according to their position, and adjusting it when they run out of sync.
Moving a differential hydraulic cylinder with a closed-loop pump is already challenging, but achieving synchronized movement coupled to precise positioning, controlled acceleration and being able to do so with four hydraulic pumps (that's right - four!) is nothing short of impressive!
The problem was - the paste pump would work OK for a random amount of time (ranging from a couple of hours to several days) and then one of the cylinders would slow down, and since the control system was always trying to synchronize - the whole pumping action would slow down or even stop - which made any work impossible.
Before I dive into the troubleshooting process, let me make a gist of the system, so that there's no technical doubt about what it is that I am trying to troubleshoot here.
The Hydraulic System:
The two hydraulic cylinders and the control cabinet are mounted on the ground floor and the HPU is installed on the first floor inside an enclosed container. Each cylinder is moved by a closed-loop. Each closed-loop transmission consists of two pumps - a 500 cc and a 355 cc Rexroth A4SGs with proportional electronic controls, governed by VT50xx amplifier modules, mounted in the cabinet below. The pumps are mounted in tandem, and each tandem has its own electric motor. The pumps are cross-connected. (The first pump of the motor one makes the loop with the second pump of the motor two). Each loop has a flushing/relief manifold. And... that's it - take the size out of the equation and the hydraulic skeleton is actually not that complex!
Troubleshooting Aids Available:
First - the PLC screen with tons of useful information. All the key parameters are here - loop pressures and temperatures, piston position, valve actuation, inputs, etc...The only thing that I would point out as a flaw of sorts would be the very low refresh rate. I, personally, like to see the values refresh several times per second, and in that case, it's more like once a second - but I am being nitpicky here...
Second - the four VT50xx amplifier cards driving the pumps are conveniently placed inside the cabinet for quick verification. These cards, aside from the normal status LEDs plus the red LED that immediately warns you about problems in the swash-plate position feedback wiring, have these nice diagnostic sockets on the front panel that allow you to instantly verify the input signal that a card is receiving and the position of the swash-plate that the card is reading - all with but a simple multi-meter. Very convenient!
Third - the pumps themselves have visual indicators of the swash-plate angle. So - when in doubt whether a pump is on stroke or not - all you have to do is go and have a look.
In short - it can't get better than that! A troubleshooter's dream!
The Environment:
It is important to mention that a paste plant as a whole is a very noisy place, because of the vacuum filtration system, which creates constant hissing noise - very loud indeed!
Back to the troubleshooting now.
We started the pump, saw it work problem-free for about on hour - and then it started doing its funky business, as one of the cylinders suddenly slowed down...
I must tell you now that whenever I hear about an intermittent hydraulic problem - I already know that the malfunction will never happen when I am around. Ever seen that gag from the 80s when a TV repairman would enter a house and the TV would instantly show a perfect picture, and as soon as he closed the door behind him the image would turn to static? Well - that's me with hydraulic equipment. (If you struggle imagining what a TV repairman and static are - you and I were born in different centuries...)
Still - I was glad to see the pump fail because "if it's broken - you can fix it". So I told the techs around me - "Not to worry, lads, for I'll troubleshoot this in a heartbeat!", and sprang into action.
Before anything I wanted to know how the pumps on the failing side "felt". With multi-meter in hand - I measured the VT card diagnostic outlets:
Pump one - control signal 3,6V - beautiful! Swash-plate feedback - minus 3,6V - wonderful! The card gets the right signal and the swash-plate is right where it is supposed to be. (Full stroke was 6V if you must know). OK then, pump number two - control signal 3,6V, swash-plate position minus 3,6V, once again perfect.
But I knew better, right? And as an experienced tech who double-checks everything - I ran upstairs and looked at the swash-plate angle indicators on both pumps - and what I saw was indeed exactly what the PLC was telling me - both pumps were about mid-stroke. Yet the cylinder was very-very slow... Bummer.
The next thing that I did was confirm to the instrumentation personnel and the electricians standing- by that all pointed to a hydraulic problem. Then I started thinking:
"...Charge pressure's OK, the high pressure's not that high - so something must be by-passing somewhere!..."
And there were only two places where something like this could happen - it could be either the huge manifold that housed the reliefs and the loop flushing or the actuator itself.
Since one of the cylinders was barely moving, even with the pumps at about 50% displacement - I used the hand technique - i.e. put my hand on the return side connectors - and saw that they were cold, so I knew the cylinder was not by-passing, at least not obviously. Then the only other place the bypass could happen would be the valve manifold - so I went back up to the HPU and started putting my hands on the manifold body trying to detect a hot-spot... Controversial, right? I mean - safety-wise? I agree! So - kids - don't do what I do! I know what I can put my hands on and most of you probably don't.. (Do re-read the disclaimer if you haven't already).
Now - where was I? Oh yes - I was manifold-hugging... A couple of years ago this same system had a malfunction with similar symptoms that was also diagnosed by hand - one of the relief valve pilots became loose and lifted from its seat, creating a leak that caused the respective logic element to by-pass the main loop causing one of the cylinders to slow down - and at that time I could feel one of the sides of the manifold hotter than the other. Apparently, this time I was facing a similar issue...
But this time it was different - all the symptoms were there, I was certain that the charge pressure was OK, I was certain that both of the pumps were on stroke, I was certain that the main cylinder was not by-passing, so the only place in the whole loop where it would physically possible to bypass anything would be the valve manifold (well - in theory, there could also be a problem inside one of the pumps, of course, - but far less likely, and to be completely honest I didn't even want to think about it - given the amount of work it took us to remove the pumps last time they failed...) - and yet - I didn't see the main by-pass symptom - the local temperature increase. The manifold was pretty warm, of course, but I couldn't detect any noticeable hot spots...
So I thought to myself: "...Hey, my hands aren't precision instruments, so I guess I am just not feeling the temperature gradient!.." Plus - since the pressure was so low, maybe 50-60 bar - probably it wasn't enough to create a noticeable hot spot on such a large heat conductor - so I advised everybody that we should stop the paste pumping process when possible and make preparations to disassemble the relief pilot/logic element assembly because most likely the problem was there.
So far so good. So, here I was, waiting patiently next to the electric board, chatting casually with the electricians as we were waiting for the shutdown to complete - as you imagine, even with a malfunctioning paste pump you can't just turn it off - there's a protocol that needs to be followed to make sure the paste doesn't block the tubes and all.
But the thought that something was OFF didn't leave me. Somehow, even at such a low pressure the sheer flow the monster pumps were dishing out even at half displacement should have been enough to heat the manifold to a "hand-detectable" degree. So, I decided to go back upstairs and give the manifold "one last hug". At that point, by the talk that I could hear on the production radio channel, I knew the HPU would stop at any second. So I ran upstairs and once again "inserted myself" into the HPU. To be able to touch the manifold I had to physically stand on the electric motor of the failed side, and as I was groping about the manifold in my attempts to pinpoint a hot spot - I heard a strange vibration through my feet. It was pretty low frequency and changing - increasing its pitch, as if a turbine or something heavy was revving up - you know that brum-brum-brum-brum noise (not sure how to write down the increasing frequency here).
I looked up and saw a puzzled facial expression of the maintenance tech that went up with me - I could tell that he felt and heard it too. The ambient noise inside the HPU was so loud that you couldn't talk, but his eyebrows were clearly saying - did you do this? And my face said the same thing - did you? And then it stroke me - the prime mover, dummy! So I jumped off of the motor, grabbed a flashlight and pointed it into the gap between the motor housing and the fan grill while pressing my bud-protected ear against the lifting eye on the motor's body.
And - yes sir! I saw the fan rotate, but pretty slowly! In fact, I could make out individual fins, and also my ear heard rum-rum-rum instead of the normal z-z-z-z- that every technician who had been around four-pole three-phase motors knew by heart. To double-check myself - I pressed my ear against a similar eyelet on the second motor - and heard the z-z-z-z all right. And the next second the system shut down.
Another thing I noticed as I put my hand on the working motor and then on the faulty one was the temperature difference between them. Although these motors are pretty efficient and don't heat that much, especially at relatively light loads, I could still feel the "bad" motor stone-cold, while the other one was warm. Now I knew for a fact that the motor had not been turning. I passed the information to the electricians, and sure thing - they discovered a malfunctioning part in the power board, that would let the motor start, but then would cause it to occasionally stop.
Some Qs and As now:
Why were the swash-plates moving when the main motor was stopped?
On that system, the pump servo pressure supply was common to all four pumps, so there was always pressure in the swash-plate positioning mechanism.
How lucky was I?
Very. If I hadn't climbed on top of the electric motor exactly at the right time when it suddenly decided to rev up - I would probably spend the rest of the day dismantling valves and logic elements from a 300 kg manifold and wondering why I wasn't finding anything wrong.
What else?
I wonder now if the fact that the swash-plates were tilting and the pumps were cross-connected turned them into motors - and let them rotate the motor the other way around, creating a kind of rotating by-pass. I think this is possible and would have been so cool to document, but unfortunately, as soon as I saw the slowly turning motor the system shut down - so I couldn't confirm it! Would have made a cool video!
So - yes, if your prime mover is an electric motor, and the hydraulic system is not working - spend a second to check if the motor is actually turning before doing anything else!