Archive for Table Top Exercises

Marketplace Empathy

Safe Harbor Consulting has been successful in assisting a number of organizations with their Table Top Exercise Programs for business continuity, disaster recovery and crisis management solutions.  One of the first challenges we face in the exercise planning process is to settle on the right scenario for the exercise.

Of course, the first thing we do is to get our client to forget about the scenario for a moment and list those things within your programs that you want emphasized in the exercise.  For example, we ask questions like:

Do you want the scenario to include death and injury of employees and guests?  Or, keep the focus on business interruption?

Do you want to address damage assessment procedures or just have the scenario result in the loss of access to facilities?

Do you want the scenario to result in a long term outage (weeks or months)?  Or, a short term loss (hours or days)?

Do you want the scenario to be an immediate impact and obvious disaster?  Or, an escalating problem that “rolls” into a disaster?

Knowing the answer to these questions will help us land on the proper scenario.

But, this scenario discussion also leads us to talk about another interesting phenomena in business continuity planning that I am not sure I have heard anyone else talk about.  Many times, I find myself trying to talk the client down from those “spectacular” disaster scenarios to scenarios that are more likely to occur and, believe it or not, more likely to offer a greater challenge to your organization.

The phenomena I speak of is a concept I call “Marketplace Empathy”.

One of the factors that will measure your success in responding to and recovering from a business interruption event is how well do you meet the outside world’s expectations?  In those newsworthy, high impact, catastrophic events that impact you, your costumers and your competitors alike, you are not necessarily expected to be up and running the next day, or even weeks or longer.  The marketplace, as a whole, can empathize with your dilemma and will allow you the luxury of time to get back to business as usual.

This will not be the case when your business interruption event is caused by a less newsworthy, low impact event that only impacts you.  If your Call Center is down because of a fire in your telecom office that takes your PBX down you are not going to be granted that same level of forgiveness as when a tornado wipes out the entire town where your Call Center happens to be located.

Marketplace Empathy.

With it, RTEs (Recovery Time Expectations) will expand.  Without it, RTEs will shrink.

With it, the news will center on the event.  Without it, the news will center on your inability to deliver.

I do not believe Marketplace Empathy is a concept that should influence your planning process, but it is something you should consider when planning for and/or executing your Table Top Exercises.

The fact is, RTOs (Recovery Time Objectives) and MADs (Maximum Acceptable Downtimes) are planning targets based on BIAs and other informational input, but the RTEs will be influenced by the scenario you are impacted by and responding to.  When you go with the Tom Clancy-esque type of scenarios in your Table Top Exercise you risk having your participants focusing on the event itself and you allow people to challenge the real need to recover the business when the impact is so great and so many people are affected.

Marketplace Empathy.  Just something to consider when planning your next exercise.

Tabletop Exercises: Are These Enough?

At Safe Harbor Consulting we specialize in creating and facilitating Tabletop Exercises.  In fact, most of our projects and a large percentage of our revenue is earned from performing this service.  We, therefore, understand, appreciate and tout the benefits and values of conducting these exercises and realize the efficiencies and effectiveness of using this non-invasive testing technique.

I do get concerned however, that organizations depend too much on the tabletop exercise as their one and only business continuity, disaster recovery, crisis management testing tool.

Tabletop exercises are great for educating management, employees and others on the strategies and solutions in place.  Tabletop exercises are terrific for ensuring the documentation supporting these programs are complete, accurate and easy to understand.  Tabletop exercises are tremendous for promoting communications and cross-checks between various departments and groups that have different yet coordinated roles in a comprehensive resiliency and recovery organization.

But …

Tabletop exercises do not physically prove the validity, effectiveness and timeliness of most of the physical infrastructure and logistics in place to engage and support a real-life implementation of the solutions in place.

Talking through how you would engage call trees or notification and escalation protocol should not take the place of actually performing call tree and notification tests.

Ensuring people know where to go and how to conduct business in alternate site locations (even those that use in-house, displacement strategies) should not stop you from physically exercising this strategy every now and then.

Reviewing lists of phone numbers of people and agencies to call at time of a crisis, should not prevent you from physically dialing those numbers to verify they reach the intended party and that party understands what is/would be required of them at time of crisis.

Tabletop exercises are indeed, relatively inexpensive ways to educate people, heighten awareness of programs, procedures and protocol.  Tabletop exercises can be conducted with little disruption to the production work environment and little risk of impacting productivity.  And, conducted properly, tabletop exercises can absolutely discover plenty of opportunities to improve your programs and implemented solutions.  I just wish to caution folks that these exercises should not stop you from attempting mock events that more closely simulate a real life response to a potential business interruption event.

I am not saying, “Do not do tabletop exercises”, or even, “Do fewer tabletop exercises”; I am just suggesting that you should strive to include physical tests in your overall exercising process, when possible, to better prepare your company for the eventual business interruption event.

These, “other kinds of exercises” are hard to coordinate, take up time of many employees in the organization, can be disruptive to the production environment and, can be rather expensive to conduct – but, the further validation they provide and the heightened level of preparedness they instill can be worth the investment, every now and then.

Just don’t let yourself become too complacent with the tabletop exercise.  Try to get permission to do more, if you can.  Outside agencies, like fire departments, local emergency management agencies, and the such, usually love to assist and play a role in these exercises – I urge you to reach out to them and see how they might help you raise your exercise bar.

Safe Harbor Consulting would absolutely love to continue to perform a bunch of tabletop exercises – but, we would be even happier if we could assist you with a more life-like simulated exercise to really test your business continuity, disaster recovery and/or crisis management posture.

Test Facilitation – Who Should Be Doing This?

This, of course, is a self-serving question – I won’t even try to disguise that fact – but I wonder how effective it is for Business Continuity and Disaster Recovery Managers to design, administer and facilitate their own tests and exercises.

I used to argue this point years ago when I was a programmer.  I thought it was very important to separate the testing responsibilities from the programmer writing the code.  It just seemed to me, if the person who wrote the program was also responsible for testing it, you wouldn’t be so effective.  They would only test to see that the software worked the way they wrote it and not the way a user might use it – does that make any sense?  I remember once another programmer asked me to help test their program.  I sat down at the input screen and in the fields asking for dollar amounts typed in a bunch of letters – they immediately said – “No, no, no – those are numeric only fields.”  “Yeah, well what happens if I put in letters?”  The program aborted.  Back to adding code checking the fields for numeric values.

In the case of Business Continuity programs I think there are a few conflicts of interest at play here.  I think many planners use tests to highlight program weaknesses and increase awareness of policies and procedures, but, they are also responsible for many aspects of the program and, by human nature, will design the test/exercise knowing what they do well and what they do not.  Maybe, having an outside entity, someone who is not prejudiced by the knowledge of what a particular program’s strengths and weaknesses are, would result in a more legitimate exercise.

Secondly, even though Business Continuity Planners are mostly responsible for preparing their organizations to respond to and recover from disasters, don’t most, if not all, of you also have some role to play in the implementation or management of the crisis during time of the disaster?  If so, how well are you testing your process if you are the one preparing and facilitating the exercise?

One last segment of this commercial – I really do apologize for this, I promised myself I would try to be more subtle in using this blog as a blatant commercial – by using an experienced outside organization to develop and facilitate your exercise, we gain the benefit of the knowledge they have learned in participating in many exercises of other organizations and seeing what they have done well and what they have struggled with.  Some of this experience will benefit your organization as they observe your responses and actions throughout the exercise.  I know that I always find myself, at one point or another in an exercise, when discussing the challenges an organization faced in the post exercise review offering saying something like, “What I have seen another company do in this situation that seems to work for them is …”

I know this often comes down to a question of costs and budgets – what doesn’t – but, I think some planners just aren’t confident enough to have someone else come in and test their program and/or use the exercise as a means to promote themselves and their role in an organization.  For programs that are mature and your real testing objective is to measure just how prepared you and your organization really are, maybe it’s time to bring in an outsider to help administer your exercise.  If so … our phones are open.

Now back to your regularly scheduled show.

Risk Analysis: The Nuclear Power Plant Threat

I am in the process of creating an Emergency Response Facilitated Exercise for one of Safe Harbor Consulting’s prestigious clients who has elected to simulate a nuclear power plant crisis near one of their strategic corporate locations.  My research on this topic has uncovered some rather disturbing information.

Currently, the US standard is to establish an evacuation zone of 10 miles, yet in the wake of the Fukushima, tsunami induced crisis, the US government ordered the evacuation of US citizens within 50 miles of the site.  The Nuclear Regulatory Commission (NRC) suggests that they would do the same should a similar event happen in the US.  Then why not expand the standard evacuation zone that nuclear sites currently are told to plan for?

Furthermore, my research suggest that information concerning the expected time to evacuate from nearby nuclear power plants is based on old and outdated population figures.  This is disturbing to me – what are your thoughts on this?

This web site shows the active nuclear power plants and the population counts nearby.  Realizing how many plants were in the path of Hurricane Irene is pretty scary.  Sure these facilities are hardened and built to withstand most weather and geological threats, but still – a breach at any one of these plants could be devastating.

Now, I do not want to come across as a fear monger – just wondering how many of you include the possibility of evacuation caused by a nuclear power plant compromise as part of your risk analysis?  If doing so, I would use the 50 mile radius precedent established by the Fukushima catastrophe as my measuring stick and not the official 10 mile radius established by the NRC.

Now back to planning the exercise.  Maybe in a future blog I can relate how it went.

A Business Continuity Football Analogy

Football season is just around the corner – and, I love football season.  So, in keeping with the season, I thought I would use a little football analogy for today’s blog.

Imagine you are the head coach for a football team.  You work long and hard in putting together a playbook with complicated blocking schemes, stunts, trick plays, disguised coverages, blitzes, audibles, etc.

Now imagine that you put the playbook on the shelf just waiting for a game to begin.

You wouldn’t do that, would you?  No.  You would give it to your players and expect them to study it in great detail, memorizing their assignments and what is expected of those around them.  You would practice the plays, looking to improve performance and perfect each and every play so that when game time comes, you are prepared.

Now imagine you are the manager of a crisis management program.  Yeah, I think you get the point.

Yet, many organizations do exactly that.  They invest time and money in putting together quite sophisticated emergency response, business continuity and disaster recovery plans but do not distribute them to the “players” to study and memorize, do not routinely practice them under varying downs and distances, and just sit back waiting for the whistle that begins the game.  And, remarkably, wonder why things did not go well when a disaster occurs.

So, my recommendation is, wipe the dust off your playbook, distribute it widely and get down to some serious practicing – maybe not two-a-days, but more than once-a-years – or, prepare yourself for a losing season.

Disasters With Warning and Those Without

In just a matter of days, locations on the East Coast of the United States will experience both a disaster that comes with some advance warning, in Hurricane Irene, and a disaster that seemingly pops up out of nowhere, with no advance warning, in the earthquake centered in Virginia.
For events like a hurricane, organizations should have a checklist of actions to take 72, 48 and/or 24 hours before impact that can prepare them for the potential threat and lessen the impact of the event.
For scenarios like the one that played out yesterday on the east coast, organizations need to be ready in a moments notice to react, respond and recover from the threat.
Luckily, yesterday’s earthquake had limited impact to both domestic and corporate facilities – although there may have been a few laundry bills that had to be paid.  And, hopefully, Hurricane Irene will, likewise, have limited impact on the East Coast.
In both cases, however, there should be plenty of opportunity to learn a few lessons to improve our overall disaster preparedness posture – let’s hope organizations take advantage of this opportunity.
And, even if your company or home does not and did not lie in the path of either of these two events, you can prepare yourself for similar scenarios and … test, test, test your level of preparedness through plan walkthroughs, table top exercises and/or mock disasters.
I hope you fared well in yesterdays earthquake and that Irene decides to pass us by unharmed.  But, I also hope each event allows you to take advantage of the heightened awareness of the possibility of crises to update, improve and better socialize your crisis management, emergency preapredness, business continuity and disaster recovery programs.

Earthquake on the East Coast

Sometimes reality exceeds the imagination.  Here at Safe Harbor Consulting we have the priviledge of creating and facilitating emergency response and business continuity exercises for a number of organizations.  One of the first challenges we tackle in each case is to select a scenario that is feasible, yet not overdone, realistic and believable.  Up to about one hour ago, creating an exercize for an earthquake for companies on the East Coast of the United States, did not fit that criteria.
How many organizations up the eastern seaboard of the United States had practiced earthquake response plans?  Not many – yet there are several, overdue fault lines all along the east coast, including a few that put New York City at risk.
Know your risks and threats.  Safe Harbor Consulting can also conduct a thorough Risk Analysis that helps identify those risks that may threaten your facilities.
I will be closely watching the news reports to see how folks fared this afternoon.
I hope all of you did and are doing well.

The BIG One vs. The Most Likely One

I think sometimes we get carried away with the potentially sensational nature of emergency management and disaster recovery planning.  I have worked with a number of individual agencies, companies and consortiums who want to do table top exercises for huge, sensational scenarios with borderline Armageddon-like impacts.

Whereas, I am fine helping them prepare for and execute such exercises, I like to warn people that, in terms of business continuity planning, these scenarios are less likely and, believe it or not, potentially less challenging to individual businesses than the isolated and less spectacular building outage due to fire, power-failure or some other similar, mundane event.

Sure the emergency response and triage following an earthquake, tsunami, dirty bomb, etc., will be spectacularly challenging and chaotic, but, if you want to exercise your ability to recover critical business processes, you should consider the following issues.

Remember, in the aftermath of a huge, wide-area disaster:

  • Your customers are impacted and may not have a demand for your services during the crisis.
  • Your competition may be impacted and not in a position to take market-share from you.
  • Given the nature and newsworthiness of the event, the expectation that you are in business will be impacted.  The marketplace will be more tolerant of your downtime.

The greater business continuity challenge is…

  • If the disaster only happens to you.
  • Customers are seeking your services and less understanding that you cannot perform.
  • Your competition is fully functional and ready to take your customers away.
  • You do not have marketplace empathy.

So, again, I simply wish to caution you as you plan your next table top exercise; focus on what you want to test.  If you want to exercise your emergency response posture and the organizations in place to respond to a wide-area disaster, okay, go with the big event.  But, if you want to exercise your ability to get back up and running in a scenario where the demand for your services and competitive environment remain constant, maybe just the good old building fire is the best way to go.

I know these types of exercises are less fun to plan and conduct, but the small, independent business interruption event is still the most likely scenario to occur and, if you are not in position to efficiently and effectively respond to that, maybe you can hold off an simulating the end of the world until you are.

Planning Your Table Top Exercise

When I help organizations plan their table top exercises, the first thing they always want to do is to select the scenario.  And, the first thing I do is say, “Time out.”

Before you start trying to pick the coolest or best scenario to exercise your business continuity, crisis management or emergency response plans with, you must first decide what it is in your program you want to test.

For example;

  • Do you want to test for a specific building, campus or geographic region?
  • Do you want to have the facility destroyed, damaged, or simply inaccessible?
  • Do you want employees or on-site visitors impacted (injuries or deaths)?
  • Do you want your customers impacted – either increasing the demand for your services or decreasing the demand for your services?
  • Do you want your competition impacted by the event as well?
  • Do you want this to be a news worthy event or not?
  • Do you want to exercise your evacuation plans or just the business continuity strategies?
  • Do you want the impact to be short-lived or long term?
  • Do you want to also impact nearby recovery sites?
  • Do you want to impact employee’s availability to work from home?
  • And, the list goes on.

Once you have determined the scope and objectives of what it is you want to exercise, then it is much easier to pick the most relevant scenario.

Writing this blog, reminds me of a funny story that occurred while I was meeting with a client.  Due to our busy schedules we meet for lunch one day to discuss the type of table top exercise that might be right for his program.

While eating, in a very crowded restaurant, lost in the passion for what I do, I was rattling off these types of questions without being cognizant of how our conversation might sound to anyone eaves dropping from a nearby table:

“Well, what is it you want?  Do you want deaths – we’ve done bombs, fires, plane crashes, disgruntled employees … Or, do you just want to prevent access to the building?  We could do a hazmat accident on the nearby highway, or a late night fire when the building is not occupied.  Do want the disaster to be unique to your facility or impact the whole community?  We’ve done isolated events like water pipes breaking or we could do a wide-spread pandemic or dirty bomb.”

My colleague then started smiling and I asked her what was so funny.  She said, “Can you imagine if someone is listening in – you sound like a terrorist or hit squad – they are probably calling Homeland Security right now.”

Yeah, I guess sometimes I can get carried away.