Tag Archive for emergency response

A Business Continuity Football Analogy

Football season is just around the corner – and, I love football season.  So, in keeping with the season, I thought I would use a little football analogy for today’s blog.

Imagine you are the head coach for a football team.  You work long and hard in putting together a playbook with complicated blocking schemes, stunts, trick plays, disguised coverages, blitzes, audibles, etc.

Now imagine that you put the playbook on the shelf just waiting for a game to begin.

You wouldn’t do that, would you?  No.  You would give it to your players and expect them to study it in great detail, memorizing their assignments and what is expected of those around them.  You would practice the plays, looking to improve performance and perfect each and every play so that when game time comes, you are prepared.

Now imagine you are the manager of a crisis management program.  Yeah, I think you get the point.

Yet, many organizations do exactly that.  They invest time and money in putting together quite sophisticated emergency response, business continuity and disaster recovery plans but do not distribute them to the “players” to study and memorize, do not routinely practice them under varying downs and distances, and just sit back waiting for the whistle that begins the game.  And, remarkably, wonder why things did not go well when a disaster occurs.

So, my recommendation is, wipe the dust off your playbook, distribute it widely and get down to some serious practicing – maybe not two-a-days, but more than once-a-years – or, prepare yourself for a losing season.

Earthquake on the East Coast

Sometimes reality exceeds the imagination.  Here at Safe Harbor Consulting we have the priviledge of creating and facilitating emergency response and business continuity exercises for a number of organizations.  One of the first challenges we tackle in each case is to select a scenario that is feasible, yet not overdone, realistic and believable.  Up to about one hour ago, creating an exercize for an earthquake for companies on the East Coast of the United States, did not fit that criteria.
How many organizations up the eastern seaboard of the United States had practiced earthquake response plans?  Not many – yet there are several, overdue fault lines all along the east coast, including a few that put New York City at risk.
Know your risks and threats.  Safe Harbor Consulting can also conduct a thorough Risk Analysis that helps identify those risks that may threaten your facilities.
I will be closely watching the news reports to see how folks fared this afternoon.
I hope all of you did and are doing well.

Disasters Through Time

Disasters come in different shapes and sizes with varying short term and long term impacts.  Some disasters are more significant in terms of human loss and suffering and some are more significant in their impacts on businesses and commerce.

Business continuity and disaster recovery planning historically focused on those disasters that impacted businesses and commerce, but recent events, beginning with the World Trade Center attacks, have required business planners to also incorporate emergency response and crisis management techniques that focus on the human element in their programs.

Looking back in time, the first significant business interruption event to occur promoting the concepts of disaster recovery and business continuity was the First Interstate Bank fire in Los Angeles.  This event occurred on the night of May 4, 1988 and really brought the need and justification for disaster recovery and business continuity planning to the forefront of executive board rooms and has served as the first real case study for these practices ever since.

It was only four days later on May 8, 1988 that a Bell Telephone Central Office in Hinsdale, IL caught fire and disrupted phone services to a wide area of IL including much of Chicago.  Within days, we had one event that severely impacted the entire operations of a large financial firm and one event that had a limited, but widespread impact to phone services for hundreds of companies.  And, the practice of disaster recovery and business continuity gained credibility and importance.

Over the next few years there were a number of newsworthy business interruption events caused by hurricanes, snow storms collapsing roofs, fires, and others, but nothing of great significance until the bombing of the World Trade Center on February 26, 1993.  This event impacted a significant number of large firms requiring alternate site recovery into vendor provided disaster recovery facilities that stretched the limits of these offerings requiring some firms to recover in secondary and tertiary facilities greatly distanced from their home location.

The 1993 bombings, of course, was just a precursor to the tragic events of 9/11/2001.  The attacks on the World Trade Center in 2001 highlighted the deficiencies in corporate business continuity programs with regards to the immediate emergency response and crisis management stratagems.  Many impacted firms knew where and how to recover technology and work areas but were not prepared to deal with the immediate needs of responding to a crisis with such severe impacts to its facilities and employees.  This event put the focus on the Crisis Management aspects of a holistic Enterprise Recovery Program.

Like the World Trade Center tragedy, Hurricane Katrina, which devastated New Orleans and the Southeastern United States on August 28th, 2005 challenged both the emergency response and business continuity aspects for many companies over a large geographic footprint. 

Each and every one of these events brought with them unique challenges and varying response requirements.  And, each event, in the post-event evaluations, provided plenty of lessons learned and opportunities for all of us to improve and expand our own disaster recovery, business continuity and crisis management programs.

Of less impact in all areas, but one of my favorite (if that is the right term to use) business interruption event was the Chicago River Flood in September of 1991.  This event occurred on a beautiful end of Summer Day when a construction error while reinforcing a bridge crossing the Chicago River punctured a whole in an old underground highway flooding the basements in many Chicago Loop area buildings.  Some of these buildings housed computer and electrical equipment in the buildings resulting in severe widespread business interruptions.  It just goes to show you, you never know when, where or how a business disaster might occur.

There have been, of course many other events, to numerous to mention in one blogs, but, I think, these events highlight the growth and changing nature of the business continuity, disaster recovery and crisis management industry in the US.  I think it is always good to go back and revisit these events every now and then to make sure the lessons learned from them have not been forgotten over time.

Responding to Riots – Case Study: London

Wow, just watching the news reports and reading the articles it is not hard to imagine how a city gone mad could seriously impact one’s ability to conduct business from the small, local establishment to large organizations with a presence in the area.

The impacts can range from your employees’ commutes being impacted due to blocked roads and shutting down mass transportation services to having your building engulfed in fires started by the mob.  Your customers may be prevented from getting to your place of business, key vendors may be impacted and your competition might be at risk as well – all events that could require some sort of coordinated response from your emergency preparedness and/or crisis management teams.

Has your company assessed the probability of a riot impacting key facilities and or the potential impacts should this occur?  Forbe.com has an interesting article on the potential insurance implications from the London riots with regards to English law.  It might be worth your while to investigate potential insurance policies your firm has covering potential losses due to riots or civil unrest.

I do not believe organizations need business continuity / disaster recovery plans for every type of scenario that could occur.  The strategies for responding to challenges employees face when transportation is shut down could be the same for strikes, hazmat spills, or even fear of being in public during pandemic scares – so, maybe you have a plan to address an immobilized workforce.  The plans you have for any event that damages your facilities may also work in this event, should your facility be in harm’s way of the fires that have been set.

I think it is valuable, however, to identify if you could be at risk to impacts from a riot and have an emergency response plan in place should the risk exist.  And, it may be a good idea to take a look at those insurance policies before an event occurs.

Referring to the Forbes’ article, I don’t care what you call it – if it impacts your ability to conduct business and puts your assets at risk – it is not a good thing.

The Recovery Time Objective Debate Continues

The Recovery Time Objective debate continues over on a LinkedIn discussion board.  Really folks, I don’t know what is so hard to comprehend here!  I think some people are just trying to be difficult as a means to show they are smarter than everyone else.  Me, personally, I prefer the KISS method – Keep It Simply Simple (I know it is usually said another way, but I wanted to avoid labeling people).

Simply put, the RTO measures the time objective for moving from Point A to Point B where; Point A equals the moment when a business process (or technology resource, if used for IT Disaster Recovery purposes) stops functioning and Point B equals the point when the business process (or, you know) must start functioning again to avoid jeopardizing the solvency of the organization.

It is an OBJECTIVE – that word is part of the acronym – why is it so hard to comprehend?

Yes, yes, yes, the event that interrupts the process or service will definitely influence when the recovery process starts, or what recovery tactic you decide to take – but the OBJECTIVE remains the same.  Fine, fine, fine, so you have an emergency response team that is responsible for assessing the damages and determining whether or not to declare a disaster, but the OBJECTIVE remains the same and the clock is ticking.

Hopefully, your proven recovery capability is less than your recovery objective.  In that case, the Recovery Time Objective minus the Proven Time to Recover equals the time your Emergency Response Team has to gather, evaluate the situation, and declare the disaster in order to ensure your RTO is met.

RTO – PTtR = Maximum Time to Declare

Your Emergency Response Team needs to be aware of all of these factors while performing their response tasks.

You do not decide the RTO or the PTtR at time of disaster – it is too late.

The RTOs are established in the BIA process.  The PTtR are established through a series of tests and exercises.

I do not disagree with most of what people are arguing in the discussion thread – I just disagree with the words they are using in the argument.  You are overcomplicating the point and mixing apples with oranges.  Sometimes I think it would be better to just throw out the common terms in use today and come up with new terms at each company that do not have a preconceived notion of what they mean.  Then define the new terms the way you want to use them so everyone in that organization has a common understanding.  That may be throwing out the baby with the bath water, but it might stop me from pulling out what little hair I have remaining while reading this agonizing discussion thread.

Communications, Communications, Communications

You know the old real estate adage about the three most important factors in the value of property? Well, I have plagiarized that saying and apply it to emergency response programs by asking, “Do you know the three most important components of a successful Crisis Management Program?”. “Communications, communications and communications.”

Sometimes I think business continuity and disaster recovery planners forget that fact as they get caught up in providing complicated alternate site solutions and in depth contingency plans.

Now, I don’t mean to imply that detailed continuity and recovery strategies and solutions are not required for enterprise recovery programs – because they are. What I am suggesting is that planners sometimes, in my humble opinion, get too deep into this aspect of the methodology before even considering the logistics of how management will be contacted and informed of a disaster.

I have, on more than one occasion, been responsible for teams of consultants working the business continuity methodology, conducting BIAs, Risk Analysis and Recovery Requirements Analysis, when the company CEO says, “Joe, your team has been here for over a month now, and I still don’t know who calls me at three in the morning when the alarms go off.”

I think job one should be, document the current response process, even if for a non existent recovery solution.

For me, it’s like creating a new football program and designing the perfect play book, implementing a defensive and offensive style of play, trick plays, and situational calls before ever teaching the players the fundamentals of the game and how to block and tackle.

Keep it simple stupid, at first. Then when they are comfortable that they know what to do when the alarms go off, and how the leadership team will communicate to one another, you can start to improve the recovery solutions to position them to survive the disaster.

If you don’t first provide the basic foundation of an emergency response and crisis management program before you get deep into your methodology, you will be punted out of there before you score your first touchdown.

Crisis Management sans Disaster Recovery and Business Continuity

A few years ago I was making a consulting sales call on a small food processing company.  This company had one facility; half of which housed their business offices and half of which was the production and warehouse facility.

The CEO of the company agreed to meet with me as a favor to a mutual friend of ours, but he was already set on the fact that he did not need a business continuity or disaster recovery plan.

“Look Joe,” he says, “it’s easy; if we experience a disaster here we are simply out of business – case closed.  Why do I need a disaster recovery or business continuity plan?  We have such a unique facility and business process, there is no place else for us to go.”

Now, I could have tried to convince him that he was at risk of component failures, a single machine could shut down, or his computer infrastructure could fail, or a number of other, less than total devastation risks that he might want to recover from – but, I could tell he was already entrenched in a defensive posture and was not going to be sold on the need for a disaster recovery or business continuity program.  So, instead, I took a different tact.

“Okay, I understand.  But even in that case, wouldn’t you want to ‘go out of business’ the right way?  Even if your business is completely destroyed, don’t you want to make sure your employees got out okay?  And, wouldn’t you still need to pay them what you owe them?  You might still have accounts receivables outstanding – wouldn’t you still want to collect that money?  You would have bills that need to be paid and debts resolved.  What would you do with your assets that weren’t destroyed?  Bank accounts?  Wouldn’t you want to notify your clients that had unfulfilled orders?  Maybe assist them in finding other alternatives?  How would you notify all of your stakeholders that an incident occurred and you are no longer in business?  Wouldn’t you want to do these things in a timely and organized manner?  Could you achieve that today?  Wouldn’t you still need some of your employees to assist in the proper shut down of the business operations?  Who would you want to help you in these tasks?  Where would you meet?  Do they know who they are?  Do you have the information you need concerning bank accounts; accounts receivable/payable; outstanding orders; customer contact information- stored off-site where you could access it to properly shut down your operations?”

Even with the strategy of shutting down the company – there is a lot of work to do – besides the fact that you want to make sure you effectively respond to the crisis to protect the health and welfare of your employees and other stakeholders of the firm.

So, we agreed, there was no need for the traditional Disaster Recovery and Business Continuity plans – but I did win the privilege of helping them develop, implement and test a pretty robust emergency response and crisis management program for such a small business.

I think he was expecting a fight from me – instead we entered into a beautiful business relationship.

Command and Control Response Structure

In the article, “Turning Disaster Response on its Head”, posted on Continuity Insights Magazine, John Orlando from National Life Group, offers some good commentary on crisis response behavior and how disaster response programs need to be aware of and employ techniques from lessons learned in recent events.

I think, however, that there is a little bit of apples and oranges being compared in the article.  I agree that a lack of a centralized Command and Control response structure will result in a sometimes effective, ad-hoc response from resourceful individuals, but I do not think that that supports the call to abandon the creation of a Command and Control structure at time of crisis.

John shares a number of examples where abandoning a top-down management model has resulted in creative and productive solutions in the corporate world.  The problem with this theory is, all of those examples required time and trail-and-error environments – during a crisis you do not have the luxury of either of those traits.

Throughout his article, John frequently uses the term “harness the power”.  I ask, who is it that is “harnessing the power” if not those in command and control?

What I gather from John’s article is that in planning our Crisis Response / Emergency Response structure we should be aware of the lessons learned from past events where the Command and Control structure was inadequate and ad-hoc organization took over to fill in the gaps; and, how the proliferation of the social network media has added a new dimension to our response communication and information tracking process.  I think all of his examples highlight that the Command and Control structure was not well enough prepared and in place to handle the response – not that the Command and Control structure was the wrong approach to take.

I do not have access to the data or link at this time, but I do remember reading an article years ago where a military study suggested that the best management style during times of peace is the participatory management style, but at time of crisis, the most effective management style is the command and control style.

I love the information John shares with us in his article and I think there is much value to be gained by the Emergency Management professional in reading John’s article, I just don’t draw the same conclusion about this information “Turning Disaster Response on its Head.”

The BIG One vs. The Most Likely One

I think sometimes we get carried away with the potentially sensational nature of emergency management and disaster recovery planning.  I have worked with a number of individual agencies, companies and consortiums who want to do table top exercises for huge, sensational scenarios with borderline Armageddon-like impacts.

Whereas, I am fine helping them prepare for and execute such exercises, I like to warn people that, in terms of business continuity planning, these scenarios are less likely and, believe it or not, potentially less challenging to individual businesses than the isolated and less spectacular building outage due to fire, power-failure or some other similar, mundane event.

Sure the emergency response and triage following an earthquake, tsunami, dirty bomb, etc., will be spectacularly challenging and chaotic, but, if you want to exercise your ability to recover critical business processes, you should consider the following issues.

Remember, in the aftermath of a huge, wide-area disaster:

  • Your customers are impacted and may not have a demand for your services during the crisis.
  • Your competition may be impacted and not in a position to take market-share from you.
  • Given the nature and newsworthiness of the event, the expectation that you are in business will be impacted.  The marketplace will be more tolerant of your downtime.

The greater business continuity challenge is…

  • If the disaster only happens to you.
  • Customers are seeking your services and less understanding that you cannot perform.
  • Your competition is fully functional and ready to take your customers away.
  • You do not have marketplace empathy.

So, again, I simply wish to caution you as you plan your next table top exercise; focus on what you want to test.  If you want to exercise your emergency response posture and the organizations in place to respond to a wide-area disaster, okay, go with the big event.  But, if you want to exercise your ability to get back up and running in a scenario where the demand for your services and competitive environment remain constant, maybe just the good old building fire is the best way to go.

I know these types of exercises are less fun to plan and conduct, but the small, independent business interruption event is still the most likely scenario to occur and, if you are not in position to efficiently and effectively respond to that, maybe you can hold off an simulating the end of the world until you are.

Planning Your Table Top Exercise

When I help organizations plan their table top exercises, the first thing they always want to do is to select the scenario.  And, the first thing I do is say, “Time out.”

Before you start trying to pick the coolest or best scenario to exercise your business continuity, crisis management or emergency response plans with, you must first decide what it is in your program you want to test.

For example;

  • Do you want to test for a specific building, campus or geographic region?
  • Do you want to have the facility destroyed, damaged, or simply inaccessible?
  • Do you want employees or on-site visitors impacted (injuries or deaths)?
  • Do you want your customers impacted – either increasing the demand for your services or decreasing the demand for your services?
  • Do you want your competition impacted by the event as well?
  • Do you want this to be a news worthy event or not?
  • Do you want to exercise your evacuation plans or just the business continuity strategies?
  • Do you want the impact to be short-lived or long term?
  • Do you want to also impact nearby recovery sites?
  • Do you want to impact employee’s availability to work from home?
  • And, the list goes on.

Once you have determined the scope and objectives of what it is you want to exercise, then it is much easier to pick the most relevant scenario.

Writing this blog, reminds me of a funny story that occurred while I was meeting with a client.  Due to our busy schedules we meet for lunch one day to discuss the type of table top exercise that might be right for his program.

While eating, in a very crowded restaurant, lost in the passion for what I do, I was rattling off these types of questions without being cognizant of how our conversation might sound to anyone eaves dropping from a nearby table:

“Well, what is it you want?  Do you want deaths – we’ve done bombs, fires, plane crashes, disgruntled employees … Or, do you just want to prevent access to the building?  We could do a hazmat accident on the nearby highway, or a late night fire when the building is not occupied.  Do want the disaster to be unique to your facility or impact the whole community?  We’ve done isolated events like water pipes breaking or we could do a wide-spread pandemic or dirty bomb.”

My colleague then started smiling and I asked her what was so funny.  She said, “Can you imagine if someone is listening in – you sound like a terrorist or hit squad – they are probably calling Homeland Security right now.”

Yeah, I guess sometimes I can get carried away.