When we started wave three of the Deviation Analysis, we knew that, in each case, Dydramat dust was only found in production rooms 507 and 508. Next, we discovered it was found during the regular pre-production inspection. This inspection takes place just before the start of production on the first scheduled production day after the normal cleaning and maintenance days. During the inspections on 8, 16 and 24 February and on 3 March they found the Dydramat dust more or less evenly distributed all over the floor and on the production equipment. Although the Dydramat dust was barely visible, a thorough vacuuming produced from a low of 31 grams to a high of 68 grams. Analysis of each vacuum bag indicated that in each case the dust was virtually pure Dydramat. Fortunately, no Dydramat dust made its way into the Microcyn production equipment. If it had, they would have had to shut down production.
We confirmed that since the Dydramat dust was found in rooms 507 and 508 on 8 February, the only time Dydramat dust has been found is during the regular pre-production inspection. We know that because since 9 February Product Quality has been checking all active Microcyn production rooms on an hourly basis. They have found no trace of Dydramat dust. The only time it is found is during the regular pre-production inspection.
Whenever there is a problem like this, lots of Likely Causes start to fly around, mostly from people not directly involved in the Deviation Analysis. Based upon the information we had at this point in the Deviation Analysis, we could conclusively rule out the following possible causes that were flying around:
- Incompetent operators are letting Dydramat dust fly around.
- The sealing equipment in the clean-up room doesn’t work right.
- The production equipment in question was used in Dydramat production during the previous cycle.
Tip of the day: When you do a good job of gathering, sorting and organizing information, you can rule out Likely Causes that do not make sense.
Next time we’ll look at the true Likely Cause and how we proved it.
Last time we determined there was documented Dydramat contamination in Microcyn production rooms on 8 Feb, 16 Feb, 24 Feb and 3 March. That was a good start. So, we continued our Deviation Analysis by determining if just some, or all of the Microcyn production rooms were contaminated.
Production floors 3 and 5 are dedicated to Microcyn production in 10 functionally identical production rooms.
(Click image to enlarge)
The investigation quickly found that production rooms 507 & 508 were the only Microcyn production rooms with the Dydramat contamination.
Floor 4 is dedicated to Dydramat production. As the Dydramat contamination is limited to rooms 507 & 508, we quickly ruled out anything directly related to floor 4 as the cause. In addition, we knew something about floor 5 itself was directly related to the contamination problem.
So, we compared floor 5 to the other floors in the production building. Here’s what we found:
- Each floor has its own Heating, Ventilation and Air Conditioning (HVAC) equipment.
- On each floor, except floor 5, there is one HVAC system for the entire production area.
- All HVAC systems are zone controlled, allowing for HVAC being provided to each production room independently as needed.
- About 2 years ago, the HVAC on floor 5 was upgraded to provide better HVAC control.
- Floor 5 now has 3 independent HVAC systems. One services the offices. Another serves what is now the Production Equipment Clean-up Room and production rooms 505, 506, and 509. The third HVAC system is dedicated to production rooms 507 and 508.
The above got us to focus on the HVAC equipment. Standard procedure is to turn the air conditioner off when there is no sanitation activity or production in a production room. Because there is no heat being generated by the equipment and because people are not continuously working in the area, air handling is accomplished by turning on the exhaust fan in each production room. This significantly reduces your electrical cost and is only a minor inconvenience to the people staging the sanitized equipment in the room in preparation for the next production run.
Next time we’ll continue to look at the Deviation Analysis and how we used the information above.
When we showed up on 8 March, Pharmco had recently completed construction on floor 5. The construction converted the US FDA Certified Clean Storage Room into the new Production Equipment Cleaning Room.
Although the construction was scheduled to be completed by 29 January, it was not completed until
5 February. This was due, in a large part, to some modifications to the Heating, Ventilation and Air Conditioning (HVAC) system on floor 5. The modifications involved the removal of the air lock which was required for the US FDA Certified Clean Storage Room. The US FDA Certified Clean Storage Room was entered once or twice a month at most. The Production Equipment Cleaning Room is entered hundreds of times a day.
To go through the air lock people would enter the air lock, close the door behind them and wait for the HVAC system to create a negative pressure in the air lock. This negative pressure would then release the air lock door to the US FDA Certified Clean Storage Room. The people could then enter the US FDA Certified Clean Storage Room. When the people were ready to leave the US FDA Certified Clean Storage Room they would push a button in the Clean Storage Room and wait until the HVAC system created negative pressure in the air lock. This would release the door allowing them to return to the air lock.
The HVAC experts decided to remove all the air handling equipment from the air lock. This meant that the entire HVAC system for the new Production Equipment Cleaning Room needed to be reworked. This extra work delayed the completion of construction by about a week.
Several people blamed that construction on the Dydramat contamination in the Microcyn Production Rooms. We could eliminate that as a Likely Cause for two reasons. First, all the Floor 5 Production Rooms were sealed in an FDA approved manner during the entire production process. Second, the construction was completed by 6 February, yet there was documented contamination on 8 Feb, 16 Feb, 24 Feb and 3 March. Yes, construction might explain the contamination found on 8 Feb. But there is no logical way to blame construction for the contamination found on 16 Feb, 24 Feb and 3 March.
Ruling out construction as a Likely Cause opened people’s minds so we started to gather more information. Next time we’ll start to look at the Deviation Analysis we they did.
This situation happened at a company we’ll call Pharmco Pharmaceuticals. Microcyn is the biggest selling and most profitable drug Pharmco offers. Their second biggest selling, most profitable drug is Dydramat. They also produce a wide variety of other drugs.
Production at Pharmco, as well as at all other manufacturers of ethical drugs, is driven by Food and Drug Administration (FDA) requirements. To comply with the FDA’s requirements each production area is run for four days and then has four days devoted to cleaning, maintenance and product change-overs, if required.
After the four days of production, the equipment is stripped and cleaned, has all normal maintenance done, and is made ready for the next production run. This process normally takes two days. The third and fourth down days are used for major maintenance work and non-standard product change-overs.
Each major product has two separate production areas. For example, Microcyn is produced on floors 3 and 5. The production areas on floors 3 and 5 are virtually identical, see below. (click image to enlarge)
Production is scheduled so that the 4 days of production on floor 3 are the 4 down days on floor 5. The 4 days of production on floor 5 are the 4 down days on floor 3. This allows for continuous production. The Production Teams move from floor to floor producing product. To promote flexibility, all the Production Teams make all the products Pharmco produces on a rotation basis.
When we were called in, Pharmco was faced with a potentially devastating problem. Several of the Microcyn production rooms were contaminated with Dydramat dust. Microcyn is designed to increase blood pressure in older people. Dydramat, on the other hand, is designed to lower blood pressure. If Dydramat accidentally got into the Microcyn supply, a potentially fatal reaction could occur, resulting in a major law suit.
Next time we’ll start to look at what was going on at Pharmco when we arrived.
Last time we found the cause of the SS III right movement swing cylinder hoses pulling out of the fitting at the block end of the hose. So, what next? But, first, a little background.
Do you remember the title of the first post in this series? It was, “These hoses have lousy crimps.” Before we were called in, this client was sure the problem was bad crimps on the hydraulic hose fittings. To them, that was the only explanation. How else could a hydraulic hose pull out of a fitting? Not to be pointing fingers, but, they came to this conclusion without gathering any information!
After less than an hour of data gathering we could rule out bad crimps. Refer to our second post in this series for details. So, the first tip of this series: drawing a conclusion without any data gathering often leads to taking costly, time consuming inappropriate action.
Now to the fix. Do you think that making the hose specification longer and leaving it at that will truly solve this problem? Hope you said, “No!” This incident led Mighty Mountain Manufacturing to take a close look at all their hose specifications. To their surprise, about 65% of their hose length specifications were not optimal. So, they fixed them all! Now the second tip of this series: finding the cause of a Deviation often brings to light an opportunity!
Next time we’ll start to look at another real-life problem upon which we worked.
The first Deviation Analysis session, among other things, gave us the following:
- Something changed on or about 18 July
- Whatever sponsors this deviation happens in Operations Test
- There must be something different about the SS III when compared to the SS I & SS II as it is the only one with the failures.
We decided to look at “something changed on or about 18 July’’ first.
We found that on 10 July a shipment of hoses was received from a new supplier. That lines up quite nicely with the first failed hose. Mighty Mountain Mining keeps a 6 day + /- safety stock on hand. They replenish stock in a loose First in First Out (FIFO) stock rotation. The hoses are restocked from the back of the bin and withdrawn from the front of the bin.
That led us to look at the hose specs. We found that the hose in question had the following spec: (click image to enlarge)
Further investigation determined that the old supplier was always on the “long” side of the spec and the new supplier is on the “short” side of the spec. At this point we did not know if hose length was a factor, but, it is something we noted for future reference.
Next, we looked at “whatever sponsors this deviation happens in Operations Test.” We found that in operations test each function is driven to its extreme limit 10 times. We determined that the failure occurs anywhere from the 2nd to 8th test of the swing cylinder. We knew this was significant and also noted it for future reference.
Next time we’ll look at “There must be something different about the SS III when compared to the SS I & SS II as it is the only one with the failures.”
Last time we looked at “something changed on or about 18 July” and “whatever sponsors this deviation happens in Operations Test.” This time we’ll look at, “there must be something different about the SS III when compared to the SS I & SS II as it is the only one with the failures.”
The team dealing with “there must be something different about the SS III when compared to the SS I & SS II” found that the SS III has a hydraulic block that is different from that of the SS I & SS II. Click link to see the diagrams troubleshootinglogic.com/?p=5319.
As soon as we saw that the SS III had relocated the position on the hydraulic block of the swing cylinder ports we knew we had something upon which to follow up. One of the team members pointed out that the specification for hose length had not changed when the ports were relocated. A quick visual inspection of an SS III made it clear that there was tension on the hose at the block fitting. When a hose on the lower end of the length specification was installed, there was enough force to pull the hose out of the fitting after a few full swings in Operations Test.
Next time well discuss the “fix”.