Wednesday, August 29, 2012

#FDA Inspection Readiness: Inspect the Inspector @fdazilla

So get this.  That FDA 483 Store has this uber-awesome feature for inspection readiness:

You can inspect the inspector.

FDAzilla has a page for each of the 772 FDA inspectors involved in their 1,000 Form 483 Reports and cross-referenced two key pieces of information about them:

  1. Which cGMP site they audited
  2. Who else was with them when they conducted the audit
Which means if an FDA investigator shows up in your lobby, you can search their name on the FDAzilla 483 Store and see:

fda 483 investigator inspector

  • See who else they've inspected
  • See who they've worked with
  • Buy and research their 483 observations to see their core expertise
FDA inspectors have core areas of expertise.  Some focus their work in key geographic areas.  The more you know about your inspector, the better you are able to pass the inspection with minimal 483 observations.

This FDAzilla 483 Store is the only tool I know of that lets you inspect the inspector.  

Anyone know of anything better?

Compile Error in Hidden Module modAddin (Excel 2003, 2007)

You're here because you can't stand seeing this:

modAddin excel compile error in hidden module

anymore.  Your molars are ground flat and deleting the Forms folder didn't work for you.

You're in luck:  a guy who goes by GJBenzie wrote in and provided a solution that worked for my Windows Vista 32-bit and here it is:

Try This for Getting Rid of modAddin Errors

1. Go to Start > Run

2. Type cmd

Start Run CMD
3. At the prompt, type:

regsvr32.exe /u C:\Windows\System32\MSCOMCTL.OCX

cmd unregister mscomctl.ocx 4. Hit Enter

mscomctl unregistered 5. Then type:

regsvr32.exe C:\Windows\System32\MSCOMCTL.OCX cmd register mscomctl.ocx

6. Hit Enter

mscomctl registered

Did that work for you?

If you use OSI PI, check out: Want to never deal with this problem again?

Tuesday, August 28, 2012

1,000 #FDA 483s!?! @fdazilla

Check this out:

FDA 483 Store

It's basically a place where you can search and buy one-thousand FDA 483s reports.

 Here's what a search for Amgen's 483s look like:

fda 483 store/></a>
<br />
<br />
You get the goods:<br />
<br />
<ul style=
Click into the 483 report and see the details:
  • Number of observations
  • Inspection Time Range
  • Investigators

fdazilla 483 store individual 483
Add To Cart and checkout with PayPal.
It's strange because when you're running cell culture production in GMP environments, QA sort of gets these communications and plan their inspection readiness strategy around these 483 reports.

Next thing you know, you're at a change management meeting and you're trying to address all these new requests and can't figure out the drivers for them...

Well, here they are: 483 observations that FDA inspectors make of other plants will also apply to you.

p.s. -  This is, of course, in addition to their FREE 483 offering I wrote about a while back.

Monday, August 27, 2012

Steam Sterilization Part 2 (aka Use The Right Steam)

So in the Steam Sterilization Basics post, we established that sterility is achieved when we blast microbes on the process surface with steam.

When the steam touches the cool process surface, it condenses thereby releasing the energy of the latent heat of vaporization, 540 kcalories per liter of condensate.

On top of that, condensate occupies less space than steam such that it draws a vacuum pulling more steam to the cool process surface.

Well, there are three types of steam and two of them are the wrong types for sterilization. A good illustration is as follows:

Figure 1: Water as it is heated

Wet Steam (aka supersaturated steam)
Water vapor at boiling point - liquid water present

Dry Steam (aka saturated steam)
Water vapor at boiling point - no liquid water present

Superheated Steam
Water vapor at a temperature higher than the boiling point.

Wet and Superheated Steam Don't Sterilize

Wet steam exists at boiling point, but does not effectively sterilize because it already contains condensate, it cannot condense and therefore does not possess the latent heat of vaporization required for effective sterilization.

Superheated Steam is too hot to condense and when it comes in contact with the process surface, it just cools to a lower temperature. The heat transferred to the process surface may achieve kill, but not reliably.

Dry Steam Is The One You Want

Dry steam is the steam that collapses easily into condensate when in contact with cooler process surfaces because it is at the boiling point, thereby releasing the latent heat of vaporization.

The effective sterilization of bioreactors depends on dry steam condensing on the process surface until the temperature equilibrates, and this means operating the bioreactor at the boiling point of water, which if steaming at 15 psig is121oC.


Using wet steam? Nope, not enough energy.

Run the SIP hotter than boiling point? Nope - no latent heat of vaporization.

You achieve sterility by delivering the latent heat of vaporization with dry steam, which can only exist slightly above the boiling point.

After that, time it takes to kill the bugs depends on how many bugs are there.

Further reading:

Sunday, August 26, 2012

Steam Sterilization Basics (aka Why You Suck at SIP)

The principles of steam sterilization (SIP) get lost in the shuffle of running a modern pharma production facility.  I think the reason for this is that the specifications get written down way back when, it gets coded into the automation and all that gets validated you sort of forget about it.

...Until you start getting bacterial contamination of your bioreactors.

What most people think is that you can sterilize a bioreactor by heating the process surfaces up to 121 degrees Celsius and hold that temperature for 30 minutes.

As a newbie engineer, that's what I thought.  I'm looking at my Bioprocess Engineering textbook right now and all I see are equations for death kinetics:
kd = αe-E0d/RT
There are mentions of using steam for sterilization, but you'll never understand until you get one of these biotech old-timers who were clamping tri-clovers since before you knew how to read.

Steam Basics

Question: How much energy does it take to heat a liter of pure water from 21o Celsius to 100o Celsius?

Well, if one calorie is defined as the amount of energy it takes to heat one gram of water one degree Celsius, then to heat 1000 grams of water (a.k.a 1L of water), it takes one kilocalorie (kcal).

And if we have to cover the distance of 79oC, then that is:

79 kcal required to heat 1L of water from 21oC to 100oC.

Question: How much energy does it take to make steam from 100oC water?

There's this thing called the latent heat of vaporization. It refers to the extra energy it takes to turn something from a liquid into a gas. So even as you're adding more energy (heat) to the system, the temperature remains the same.

For water, this is 540 kcal/kg. This means:

540 kcal is required to turn 1L of water into steam.

Key message: It takes seven times the energy to make steam from water (at 100oC) than it does to boil water from room temperature.  The reverse is true.  When steam condenses it releases that same energy.

Steam Sterilization

This latent heat of vaporization of water is key in killing bacteria on your cool process surface. Here's why:

When the steam first hits that process surface, the steam loses energy as heat transfers to the process surface.  When the steam loses enough heat, water begins to condense.  And when water condenses, 540 kcal of energy (per liter of water condensate) pummels the organisms that are there.

Also, condensing steam forms a vacuum that draws more steam to the area.  This constant bombardment of disruptive energy ends only when the process surface temperature equilibrates and condensation stops.

This microbial holocaust is achieved through the two properties of steam:
  1. 540 kcal of energy is released per liter of condensate formed. 
  2. Vacuum drawing more steam to the area

Time & Temperature Myth

Clearly to make steam from water, you need high temperatures.

Clearly, it takes time for the bioreactor to reach these high temperatures.

But these are not the only factors.  You need the pressure for steam formation.  You need water to make steam.  And all this depends on how "infected" your process surface is to begin with.

There are a lot of variables in making steam sterilization (SIP) effective: 121oC for 30 minutes isn't all it takes.

Stop Sucking At SIP

See also:

Thursday, August 23, 2012

Biologics Manufacturing - Great Amgen Video

Here's a really informative video on biologics manufacturing.

Or if you prefer to watch this on YouTube, start at 15 seconds in and you can stop watching at 3:28 where the talking points begin.

It's the same old talking points that I've addressed:
My questions for innovator drug manufacturers are as follows:
  1. Is your Master Cell Bank single-cell sourced?
  2. You have never upgraded your cell line (v2.0) for an existing product, correct?
  3. Are you willing to publish the CofA testing requirements of your clinical material and abide a regulatory body approving/rejecting drugs according to whether or not each batch meets/fails those CofA specs?
Your YES response to all these questions will stand me corrected.

All the propaganda aside, nice video.

For more information on biologics manufacturing, see Amgen's biotech learning resource.

Wednesday, August 22, 2012

Bioreactor Contamination Failure Modes

When it comes to bioreactor contamination, there are exactly two failure modes:
  1. Failure to Kill (the bugs)
  2. Failure to Keep (the bugs) Out
The way it works is this:

For the volume of space (say a bioreactor) you want sterile, you start with a clean tank free of residue.  You do this by executing a Clean-In-Place (CIP).

Once your reactor is clean, you seal off orifices of your envelope.  Once this envelope is sealed, you sterilize it with steam (SIP), creating a sterile envelope.  Once the interior of this envelope bug-free, you try to keep it that way.

Failure To Kill

"Failure to Kill" is a deficiency in creating a sterile boundary.... the failure to reach sterilizing temperatures, thereby leaving viable microbes inside of the envelope.  Steam-In-Place procedures can fail for a variety of reasons, including:
  • Mis-calibrated temperature probes
  • Failure to maintain sterilizing temperature
  • Failure to reach sterilizing temperature
  • Dirty Surface
  • Inaccessible Process Surface

For validated GMP systems, the sterilization procedure is usually automated with several critical manual steps.  The faithful execution of the sterilization procedure will rarely result in the "Failure To Kill."

Some microbial contaminants are harder to kill than others.  Gram-positive bacteria, which contains a hardy outer shell - "peptidoglycan" - can withstand higher temperatures for longer than gram-negative bacterial.

Also, gram-positive bacteria have the ability to form-spores where they reduce themselves into viable, but dormant form.  The spores lie dormant until more favorable environmental conditions emerge.

Most sterilizing procedures are validated to inactivate spores, but if your bioreactor contaminations are predominantly gram-positive, hard-to-kill, spore-formers only, then look for "Failure To Kill" as one of the components of the bioreactor contamination.

Failure To Keep Out

"Failure to Keep Out" is a deficiency in maintaining the integrity of the sterile boundary.  The non-moving-parts of a sterile boundary includes the physical vessel, elastomers, valve diaphragms, and sterile filters.

During contamination responses, people often visually inspect elastomers and valve diaphragms for nicks and cuts as there is wear and tear on this material over the course of use.

Filters are often suspected and post-use integrity tests are done to ensure that the filter integrity was not breached.

When more media or feed is needed, the sterile boundary may be extended.  This is when a second envelope is created near the bioreactor, sterilized and then opened next to the bioreactor.

Maintaining the integrity of the sterile boundary often requires positive pressure.  The failure to maintain an outwardly flow can often result in the deficiency of maintaining integrity.

In fact, when steam is rapidly cooled, the pressure will drop suddenly often creating a vacuum that can suck "bugs" into the envelope.


There are exactly two failure modes that cause bioreactor contamination.  While failures can be a combination of both, it is important for the bioreactor contamination response to simplify and clarify these two modes so that the true root cause is easily enunciated and therefore found.

Consult A Bioreactor Sterility Expert

Tuesday, August 21, 2012

OSIsoft PI Server To (Re)Support Batch By 2013

As previously discussed:

PI ModuleDB is Old-And-Busted; PI AF is the New-Hotness.

And with PI Server 2010, OSIsoft has decided that PI AF is the path to the future.

What's going on is that they are building a new way of describing your plant's assets using Microsoft SQL Server as the guts of the software.

Since PI Batch is intimately tied to PI ModuleDB, it leaves PI Batch users in a lurch.

Well, be left in the lurch no more.  PI EventFrames is coming:

Old-and-BustedNew Hotness
PI BatchDBPI EventFrames

In this LinkedIn discussion, Todd Brown and Buck Bard talk about 2013 as the timeframe for when EventFrames get support.

tl;dr - Biopharma PI users, keep holding your horses.  No need to put in that Change Order for the upgrade, yet.

p.s - Until 2013, PI ModuleDB and PI Batch still work.

p.s.s - Join the PI Professionals LinkedIn group if you work with OSI PI.

Monday, August 20, 2012

PI DataLink: Cannot Insert Trend

There's this feature in PI DataLink where you can insert a PI Trend:

It's a weird feature.  As in... if you wanted to see a trend, why not use PI ProcessBook?

See the PI ProcessBook Best Practices For BioPharma.

Whatever the case, if you click that link and nothing happens, then you have a problem with your PI DataLink.

It's a symptom whose disease is solved by deleting either of these two folders in order to fix the problem:

  • For Windows XP:
    C:\Documents and Settings\%USERNAME%\Application Data\Microsoft\Forms
  • For Windows Vista, Windows 7:

It's actually the same exact root cause as those Compile error in hidden module: modAddin errors.

Set Your Default (OSIsoft) PI Server Already

This is true.

Your PI BatchView QuickSearch isn't going to search.

You might get yourself a COM Error 80040408.

And you might lose a minimum of 90 minutes of your life shagging down what a COM Error has to do with NOT having a default PI server.

The first thing you do is go to your PI Connections. On the menu, select Tools > Options:

You're next, you're going to get this dialog:

In that center area where it says Default Server Settings... select those drop-down menus and make sure that your favorite PI server shows up there.  (In this example, it's

Thursday, August 16, 2012

PI ProcessBook - Best Biopharma Practices 3

ProcessBook can open 2 types of files:

  1. .PIW are ProcessBooks
  2. .PDI are Displays
And frankly, I can't think of a good reason to have .PIW files anymore.

ProcessBook (.piw) files are mostly navigational.  You create this glorified outline of essentially text links:

And that's it.  Your users are looking at an outline.  Big whoops.

In its heyday, I bet this knocked the socks off a ton of people.  But we have the internet.  Even the desktop browser seems old compared to these snazzy phones and tablets.

The world has grown up in a damn hurry and for PI ProcessBook to compete, you basically have to go with PI displays.

My recommendation is to create remarkable PI displays for more than just engineers and operators.  Here's that same PIW file made into a PI display:

processbook area overview

So instead of a static text-linked outline, make a display where you can see the entire area.  In the above figure, there are only 5 units in use.  Now, users aren't wasting their time clicking into the display for a unit that has nothing going on.

And this is important when you want to engage infrequent-PI users.  Not everyone has a job where looking at the process is a daily task.

Moreover, these area overview displays are a low buck, high bang activity...especially for management.  Managers like to see the things they're paying for... the easiest way to show off their investment in PI is to create remarkable displays that they can relate to.

The more you market their investments to them, the more they're willing to plow cash back into your projects.

It's a win for your users.  It's a win for your management.  And it's a win for you.

Previous best practices:

  1. Standardize On Colors For Key Parameters.
  2. Use White/transparent Background For Presentation-Ready Displays.

Wednesday, August 15, 2012

PI ProcessBook - Best Biopharm Practices 2

The thing about visualizing process trends is that at some point, you're going to have to make a decision with it.  And when you make a decision with it in the cGMP environment, you're going to need someone to review the decision.  Perhaps this decision is big enough that you're going to need to document it.

Or perhaps, this decision needs a little CYA and you're going to have to present it to the plant management.  Whatever the case, there comes a time when you're going to have to take what you're seeing out of PI ProcessBook and put it in a Microsoft Word document or Powerpoint slide.

This is where the core philosophy of my ex-boss "Jesse B." comes in handy.  Jess is a big believer in one-stop or "few-stop" shopping for data.  You go to one or two places for data and spend the rest of your time analyzing.  OSIsoft PI server is the ideal stop for plant information.

The corollary to "One-Stop Shopping" is "No Additional Editing."  This means having the PI display formatted correctly where "correctly" means no additional work has to be done.

Face it, when you're facing a big decision and you've spent all your bandwidth on analyzing the data, the last thing you want to do is mouse around trying to get the background color right, or putting lipstick on the proverbial PI display pig.

Here are a few things to help you out:
  • Use a white or transparent background for your PI ProcessBook displays.
    All Word documents and most Powerpoint slides have white backgrounds
  • Use dark colors for primary tags
    The tags that need the most attention should have dark to contrast against the white background
  • Use lighter colors for noisier tags
    Noisy tags like jacket temperature or controller output will do well with lighter colors
Have a look at this display and you can tell immediately that this can be copied and pasted directly into a memo with no additional editing:

You're shaving 5 to 10 minutes off every presentation you make.  Over the course of a week, we're easily talking an hour or two of your life you're never going to have to spend on making your memo look pretty.

Monday, August 13, 2012

PI ProcessBook - Best Biopharm Practices 1

There's this guy I knew who dropped out of the Air Force Academy. He was telling me that I'd never last (probably true). For example, he was saying that the amount of conformity was insane. Everyone had to fold their bed the same way. Their underwear in the same drawer, shampoo in the same drawer. Had to pack their bags the same way.

And the reason, I'd imagine, is in the heat of combat, you need some ammo from your buddy, that you know where he's packed it and can get to it without asking. And the sooner you enforce this discipline, the better off you are.

The same goes for PI ProcessBook displays. The great thing about ProcessBook display is that once you know how to create stuff, people go crazy and create a bunch of displays using the default colors. There are some folks who will change their own colors, but overall, it's a jungle out there.

The bad part of this is that displays are usually shared. The way people actually use ProcessBook is by pulling it up to show someone.

Maybe you're a Manufacturing Sciences plant support guy and you need to show an operator what's going on and why he needs to switch to controlling off the other probe.

Maybe you're showing your boss why you think this batch of media is contaminated.

Maybe you're presenting data to the VP of the plant a real-time status update and you needed a screenshot for your Powerpoint presentation.

Whatever the case, the moment you show someone your ProcessBook display, they're spending the next 5 to 10 seconds reading the labels to figure out what is going on. Meanwhile, you've started explaining what's going on, dragging that cursor left and right.

The way to fix that is standardizing on your colors:

For cell culture displays, I've proposed:
  • pH and related trends - red
  • dO2, Air, O2 Sparge - blue
  • Temperature (Vessel/Jacket) - green
    Red and blue represent hot and cold, so a neutral color like green makes sense.
  • Agitation - black
  • Volume - orange
    The color of media is orange
  • Pressure - grey
If you have a vibrant PI user community at your organization, standardizing on parameter-color is going to be a tough sell.  But if you can pull it off, it will be a great stride towards organizational efficiency.

Tuesday, August 7, 2012

KnowledgeBase Is Moving

Public service announcement.

The Zymergi KnowledgeBase, which as of yesterday lived at:

has been moved to:

You don't need to do anything as the old links will automatically redirect you to the new site.  But just in case your bookmark breaks or something, that's the reason.

Monday, August 6, 2012

OSI PI ProcessBook Glossary

There's a ton of confusion regarding PI ProcessBook terminology. This short post will help clear it up.
PI ProcessBook
A OSIsoft-written Windows application that renders trends with PI data
PI processbook
Because this is the software people use to view PI data, users often confuse this with PI itself.
A file-type (.piw) that can be read by PI ProcessBook application. Helps organize displays
PI processbook
A ProcessBook is one of two types of files that can be opened by PI ProcessBook (the application) used to organize displays and help users navigate the plethora of files on the file server.
Display (aka "PI Display)
A file-type (.pdi) that can be read by PI ProcessBook application. Typically, users open pre-configured displays to view data
PI processbook
PI display is where the magic happens.
An object on a display that graphically represents PI data
A symbol that plots tag data versus time
PI processbook
The trend where PI data gets plotted. Users interact with the Trend symbol the most to visualize what's happening with the process.
A curve on the trend of plotted PI point data. The above trend has 3 traces.
If you understand the world in Microsoft Excel terms, here's the analogy:

Excel PI ProcessBook

If you open up VBA (Alt-F11) and explore the guts of PI ProcessBook, you'll see that the objects behind PI ProcessBook are named this way.

Further reading:

Wednesday, August 1, 2012

Follow-on Biologics Technology as of 2008

A lot of people wonder why I blog about biosimilars.  I don't have a Ph.D and I actually don't have any vested financial interest in the biosimilars.

Biosimilars have entered the mainstream ever since their mandate was specified in the Patient Protection and Affordable Care Act (a.k.a ObamaCare).

Of the many things that ObamaCare mandated, one of them was that the FDA provide a regulatory approval pathway for biosimilars.  Imagine that... a mandate for government.

In the United States as of the publication of this post, there is no way to introduce a biologic onto the market unless you are the person who came up with it... i.e the "innovator."

That's right.  Pharmaceutical companies research the the drug, show safety/efficacy in the clinic, demonstrate cGMP compliance get the FDA nod to market the drug in the United States.

But when pharma drug patent(s) expire, other pharmaceutical manufacturers are permitted to seek FDA approval to sell the generic version... so long as they meet the rigors of FDA facility inspection and their application shows that they can produce the active pharmaceutical ingredient (API).

How do they know how to make the drug?  The process isn't that simple, but you can figure it out from the aforementioned drug patents.

So why can't this be done for biologics?

The reason this putatively can't be done for biologics is because the active pharmaceutical ingredient is not made by a chemical reactions managed by humans.  The API is made by chemical reactions managed by genetically engineered cells.
  • For small biologics like insulin or hGH, E.Coli or other bacteria is capable of producing.  
  • Large biologics require mammalian cells like Chinese Hamster Ovary (CHO).
So when the FDA approves of the manufacturing procedure for a biologics process, the approval goes for that specific genetically engineered cell line.  Not even the innovator can switch out the cell line without going back to the clinic.  Them's the rules.

It stands to reason the cell line is locked for the innovator, it is locked for everyone else, hence a legal monopoly for biologics in the United States.

But back to the reason I blog about biosimilars.  In 2008, I ran into an old friend (we'll call Morpheus) who worked at a big U.S. biotech company and he asked me if I remembered a mutual friend (who we'll call Neo).  Back in the early 2000s, Neo had left to work for an overseas pharmaceutical company to lead their biologics development.

Morpheus was telling me that Neo's company had produced an exact copy of one of his company's flagship drug.

Me: "What do you mean, exact?"

Morpheus: "I mean it's closer to our clinical controls than our current manufacturing process."

What Morpheus was saying was that the competitor's drug matched the drug used to produce the successful clinical trial better than what the U.S. biotech company was manufacturing in 2008.

What was on everyone's minds at the time was industrial espionage.  After all, it was against prevailing dogma that anyone could reproduce the biologic.  But to replicate the drug better than the innovator... is insane.

I never heard about the allegations of industrial espionage again.

But what I do know is that the term biosimilars is a misnomer.  They ought to be called, "Bioidenticals."

And this was back in 2008.  Who knows how far these overseas companies pushed this technology?

One thing for certain is that the technical hurdles of protecting the biologics monopoly are falling.  And with the ObamaCare mandate for a regulatory pathway for biologics, it appears that the regulatory hurdles are falling as well.

See Also: