Tag Archive for disaster recovery

Preparing for the London Olympics

I am assisting a client in developing contingency plans for their London offices in preparation for the upcoming 2012 Olympics.  We are researching possible risks, threats and disruptions based on past Olympics and past London-area events.  And, believe me, there is plenty of material there to raise a concern.

In this process we have developed two paths of recommendations: Precautionary Strategies and At-the-Ready Contingencies.

Precautionary Strategies are actions we recommend be taken to lessen the possible impacts of disruptions that are likely to occur.  These actions require no triggers to enact; we recommend following this course of action simply as a matter of business during the Olympics.

Precautionary Strategies include:

  • Scheduling work-from-home times where the capability already exists and the disruption to work flow negligible.
  • Rerouting business processes to non-London offices where this can be easily accomplished and does not stress the remote offices work flow.
  • Utilizing facilities outside of the Olympic parameters where possible.  This may include working out of their business continuity work sites if this can be managed at little expense.

Precautionary Strategies will be temporary, low cost, easy to implement and low disruption activities that can remove some of the stress that the Olympic events may cause.

At-the-Ready-Contingencies are more of your typical business continuity solutions that will be engaged only if threats and disruptions occur.  These actions will include identified triggers that must be monitored and tracked throughout the Olympics.  Most of these strategies, hopefully, are already in place as part of their existing Business Continuity Program.

The London Olympics will certainly disrupt the commuting processes to London area businesses and introduces a threat of civil disorder, terrorist activity, and security breaches.  London area companies should be well aware of the Olympic footprint and understand the traffic flows that may interfere with their employee’s commutes.  And, these interruptions and threats will exist for pre-Olympic activities as well as for the Paralympics that follow.

Where possible, we believe London area businesses should evaluate the possibility of minimizing their London-based business activities during this event to lessen the possible impacts to their firms.  Wherever you can take precautionary steps at low expense, it might be advisable to do so.  And, of course, you should brush the dust off your Business Continuity Plans and ensure that your recovery strategies are still viable given the risks this event incurs.  For example – if your alternate site locations are also within the Olympic footprint, you may need to establish a temporary other location during this event.

Certainly, there will be plenty of security provided and lots of precautions and contingencies put in place by the local and national authorities.  Be sure you are aware of these plans and make sure that your plans will work within the parameters provided.

We hope for a smooth and exciting Olympics in 2012 – let’s just hope all of the excitement is on the athletic fields and within the competitive framework of the games.

Conducting a Test – Yes, a “Test”

Yeah, I know, I know … we don’t have “tests” we have “exercises”, because tests imply pass/fail and exercises imply getting stronger.  Yeah, I used to sing that same silly song, too.

I now understand that there are times when you need to test; times when you need to drill; and, times when you need to exercise.  You do need to test your solutions to make sure that they do, in fact, work.  You do want to know if you can “pass the test”.  You may refer to these as validation tests.

I’ll take it a step farther and would even like to see us test the people.  I think it would be great to gather our key business continuity players, managers and employees into a room and give them a regular, school-like, no. 2 leaded pencil, don’t start until I tell you to, and put down your pencils when instructed, actual tests.  Why not?

I would like to ask key players and managers questions that they should know about our Emergency Preparedness, Business Continuity and Disaster Recovery Programs.  These questions might include:

  • If the fire alarm went off right now:
  • What would you do?
  • Where would you find evacuation routes posted?
  • Where would you congregate once outside of the building?
  • Who are your floor wardens?
  • If you received a bomb threat on the phone, what would you do?
  • If you got a call at 2:00 am that the building had burnt down…
  • What would you do?
  • Who would you call?
  • Where would you go to work?
  • Where would you find your Business Continuity Plan?
  • If the Data Center experienced a disaster…
  • What applications that you use would be recovered?
  • In what timeframe?
  • What would be the status of the data?
  • What applications would not be recovered?
  • What business processes would you be expected to continue?
  • What business processes would be temporarily suspended?
  • If you do not know the answers to any of the questions on this test, where would you go to find them?
  • If we experienced a disaster and you weren’t available to participate in the recovery, who has been trained to play your role?  Have you trained them to be successful in this effort?

I could go on, but I want to try to make this an interactive blog and challenge you to post the types of questions you would want to include on this pencil to paper test.  You can do so by posting a comment to this blog.

You can exercise your solutions all you want.  You can physically recover hardware, applications, data, networks, etc., time after time – but there are some things that people need to know when the alarms go off or your ability to execute your plans will be severely hampered.

How do you think your organization would do if given this kind of test?  My bet is most companies would not fare too well.  There are historical cases where adequate recovery capabilities were in place but the people were not educated well enough to implement these capabilities at time of event.  How better to determine our level of preparedness than to give them a test?

You can exercise and get as strong as you want, but if you can’t pass the test … you will fail.

Continuity Insights 2012 Management Conference

The Continuity Insights 2012 Management Conference is scheduled for April 16 – 18 in Scottsdale, Arizona.  And, Safe Harbor Consulting will be there and well represented on the agenda.

We have been slated a terrific spot on the agenda with Joe Flach presenting his break-out session, “Revisiting the BC/DR Planning Methodology” (Session B4) on Monday, April 16 from 11:00 am – 12:00 noon.  Then, on Tuesday, April 17 from 9:45 – 11:00 am, Mr. Flach will be a panel member in a break-out session on “Exercise Facilitation Techniques” (Session G4).

You can register through the Continuity Insights website and enjoy early registration discounts.

Safe Harbor Consulting will also be hosting a Hospitality Suite in the “Talking Stick Resort”, where the conference is being held, and we look forward to meeting and entertaining you there.  We will post the room number for the hospitality suite on the Conference Bulletin Board at the conference.

We are looking forward to some fun in the sun; interesting and educating sessions; and good times with good friends.  Let us know if you are planning to attend the conference so we can be sure to connect in Scottsdale.

Goodbye 2011; Hello 2012

My how time flies.

An entire month went by without a single blog being posted to this site.  What was I thinking!?

The good news is:

  • Business at Safe Harbor Consulting has picked up and I have been busy spending time on other, income generating tasks.
  • I took time off in December to spend with my family and friends and let this blog take second priority for a while.
  • There have been no real big, disaster events demanding my attention and prompting me to write a blog.

The bad news is: 

  • I have been experiencing a little writer’s block and have been challenged to come up with new topics to write about.
  • I have received little feedback from blog readers on my postings and have wondered about the effectiveness of this tool.
  • Business at Safe Harbor Consulting has picked up and I have been busy spending time on other tasks.  (Sometimes, news is both good and bad.)

But alas, it is time for the New Year’s resolutions to kick in and I must, at least, try to fulfill one resolution before the month of January expires.

I hope everyone had a terrific holiday season and a safe and happy New Years.

2011 was an interesting and eventful year for many of us in the Business Continuity and Emergency Management arena.  If the Mayans are correct, 2012 will prove to be even more so.

If you have any subjects or ideas you would like me to try to tackle in the next few blog entries, by all means, let’s hear them.  Otherwise, I may ask you to bear with me as I repeat a few topics, with perhaps, a new spin or refocused emphasis.

May 2012 bring you much joy, happiness and few disasters – at least none that you can’t recover from.

Happy New Year,

Joe Flach

Critical Data: Don’t Overlook the Hardcopy

I know we like to think we now work in a paperless society, but the fact is, we do not.  There are still plenty of industries and processes that rely on hardcopy documentation for historical records and in support of daily operations.  Business Continuity and Disaster Recovery programs often overlook these vital records as they focus on technology and electronic medium – I caution you not to fall into this same trap.

In know this to be true, especially in airlines, medical and educational organizations as well as in some financial services and other industries. 

For example:

Airlines are required to maintain and have access to all mechanical and maintenance records for each and every aircraft that they fly.  In many instances maintenance initiatives issued by various agencies are printed and given to the mechanics and engineers who then make handwritten notations and sign off on the printed form.  These printed forms, with their notations, become the official record of the maintenance activity in compliance with the initiative.  Should this physical, hardcopy record be destroyed or lost, the plane (or an entire fleet of planes) will have to be grounded until the maintenance check is performed once again and a new record created.  Some airlines maintain these records in a single location and do not scan or digitally record the information (keeping costs down, you know).  Should the facility housing these documents go up in smoke, it could take months or longer to recreate the audit trail for those planes – which, by law, must be grounded until proof that all the maintenance initiatives have been completed.

Many medical offices maintain a slew of forms and doctor reports in handwritten form.  Just notice all the filing cabinets up and down the halls in your doctor’s office.  These records are seldom scanned or stored electronically and are susceptible to numerous risks and threats.  The same is true for school records and other information gathered in handwritten forms.

Financial services firms and brokerages still house plenty of hardcopy documents in the form of payment instructions and customer documentation that could cause plenty of financial exposure and compliance irregularities if lost or destroyed.

For those of you who think that we operate in a paperless society, just take a look around and count the number of filing cabinets still in use.  What do you suppose is kept in all this space?  And, what would be the cost or impact to the organization if they were permanently destroyed?

Now, I am not saying this is true in every environment.  Certainly there are many, many offices and industries that truly have no exposure to hardcopy documentation and information.  I am just suggesting that your risk analyses, impact analyses and recovery requirements analyses do not simply overlook this potentially critical information base and include consideration of this potentially risky business practice.

Backing up or electronically scanning and storing hardcopy documentation, especially historical documentation, may be something your organization needs to look into.  There are plenty of vendors that can help you achieve this end.

Disaster Recovery Tests: Please DO Feed the Animals

This past weekend, my 7 yr old son and I visited our own little disaster site in the hopes of doing a little cleanup work – his bedroom!  My challenge was to make it fun enough for him to participate in the effort with as little whining and crying as possible.  It occurred to me that this was very similar to the challenge I, and others, have when trying to get folks to participate on a business continuity and/or disaster recovery test.

Let’s face it folks – we can really be a pain in the backside to these people who have better and “funner things to do” – as my son put it this weekend.

I know with the budget crunches going on and the all out efforts to cut costs it is hard to get too creative with this stuff, but I still think it is worth the effort and expense to reward your test/exercise participants with snacks, meals, refreshments and the like, if not also with some kind of other tchochke item.  In the past, I have seen testers give out tee shirts, coffee mugs, and other stuff as reward for participating on tests.  One creative planner, used to have snacks tied to a theme; like ice cream cones over the summer; or hot dogs for a test scheduled during the World Series; etc.

I know this sounds corny and I see many of you rolling your eyes (yeah, this blog technology is scary – I am watching you), but these little gestures go a long way with winning good favor with those we rely on to get tests scheduled and completed.  They also can soften the impact of failures you will undoubtedly experience along the way.

Well, by singing songs, counting stuff we put away, making a game out of throwing stuff in the trash and a promise of a Dairy Queen Blizzard after the job was done – the disaster area that was my son’s bedroom finally got clean.  Now all we need to do is administer CPR to his mother who fainted when she saw what we had accomplished!

Disaster Response – Enforcing Time Limits

Do you have a policy in your business continuity, disaster recovery, emergency response and/or crisis management program that establishes a limit on the number of hours responders can work before requiring a mandatory break?  Are you in position to enforce this policy?  Do you enforce it during recovery tests?

I know that during time of crisis people rise to the occasion and can sometimes exhibit superman (or woman) like powers and appear to go strong for many, many hours – but the fact of the matter is, the longer they are active, the less effective they are likely to be and the more errors or poor decisions they are prone to make.

I strongly suggest that your programs – all of them, technology recovery teams as well – have a stipulated policy that no one individual can work for more than 12 straight hours without taking a break.  And, I highly recommend that you have individuals on your team responsible for ensuring that this policy is followed. 

I think a 12 hour on, 12 hour off schedule should work fine, requiring only two subject matter experts for each role in the program.  I would prefer three 8 hour shifts – this can still be accomplished with just two individuals – but 12 on / 12 off makes it easier to ensure your primary team member is on during the most important 12 hours of the day or night.

I know it can be difficult making the second shift team members stay away from the response during the 1st twelve hours following the disaster, but you need to let them know how important it is that they show up 12 hours into the crisis, rested, refreshed and ready to operate. 

I also recognize that the 12/12 shift does require some turnover time from one shift to the next, but we need to make sure that that turnover does not draw out too long.  It will be tough to get the first shift team members to remove themselves from all the activity after 12 hours, but it is for the benefit of the individual and for the benefit of the organization that they should be required to remove themselves from the event and get some rest.  I think it is also important to have them physically removed from the crisis, as much as the situation allows, and put up in a location where they can rest undisturbed and away from all the activity.

I know this is not easy.  It is not easy for me to follow my own rule.  But it really is for the benefit of all that this policy be established and enforced.  I remember the old days, during mainframe recovery tests, where teams of us would go almost 48 hours non-stop in the recovery process.  And, still today, there are technology, network, database and other recovery teams that have few, or even a single, subject matter expert that will work on an issue until it is resolved no matter how long it takes.  I think it is up to us, as planning professionals to identify these employee-related, single points of failure in our solutions, communicate the problem to management and seek options for remedying this exposure.

If you have technology recovery tests scheduled for more than 12 hours – you need to let it be known that no one individual will be allowed to participate in the test exercise for more than 12 hours – and, you need to make sure that that rule is enforced.

There are actually companies that provide employee health and well being services who can help you enforce this rule and help provide mental health counseling for employees impacted by and/or participating in crisis situations.  You may want to check them out for advice on how to implement this particular component of your program.

This blog was written in less than 12 hours – just so you know.

Today’s Disaster – Wild Animals on the Loose!

Okay, here’s a new one – a city in lockdown mode because there are wild animals on the loose roaming the city streets!

I can’t help but chuckle imagining the broadcast message that one would send out to their employees telling them the office is closed due to a city lockdown caused by wild animals.

I really have no more to say about this one, other than I just had to share this story with you.  I will have to challenge myself a little harder to come up with a legitimate blog post – but, you can read the story and adjust your plans accordingly for this risk.

Risk Free, Satisfaction Guaranteed Program Review

Safe Harbor Consulting (SHC), a management consulting firm specializing in business continuity, disaster recovery, emergency response and crisis management, is offering a risk free, satisfaction guaranteed Program Review.  SHC will review your program documentation, interview employees with key responsibilities in your solutions and review other program material in an effort to discover opportunities to strengthen your programs, improve your strategies and/or expand your solutions.

If, at the completion of the review and following the delivery of the SHC Findings Report, you are not satisfied that we have identified valid, substantial opportunities to advance your program and/or better position your organization’s response and recovery posture, you will not be invoiced for SHC services.

“I have found that having outside experts review program material prior to conducting a Tabletop Exercise or Physical Program Test is an excellent technique for ensuring your program material is in tip-top condition prior to sharing it with internal management and employees”, says Joe Flach, CEO and Lead Consultant at SHC.  “If the material we review is in excellent condition and, other than a few cosmetic fixes has no real identifiable issues, problems or concerns, than our review will indicate as much and we will not charge you for our efforts.  Only if we discover legitimate opportunities to improve the program or program material, and only if the customer agrees that we have achieved this, will we prepare an invoice for our agreed upon fees.”

To take advantage of this Risk Free, Satisfaction Guaranteed Program Review offer, please contact Safe Harbor Consulting at (253) 509-0233 or email them at safeharborconsulting@yahoo.com.  To learn more about Safe Harbor Consulting you can visit them at www.safeharborconsulting.biz.

National Failure Day

I found this news story to be rather intriguing and, although a little bit of a stretch to suggest it is business continuity or disaster recovery related, I have been known to stretch things out of proportion a time or two in the past.

Finland is celebrating National Failure Day today (Thursday, October 13) to help stimulate growth in the economy and combat their culture of being risk adverse and not prone to trying new business ventures due to a fear of failure.

People that have that core characteristic, fear of failure, probably should not pursue a career in business continuity / disaster recovery planning and certainly not be in charge of managing the testing process for these programs.  But, we do try to address that fear in our programs.  That is one reason why we started avoiding the word “test” in our methodology, because it implies pass/fail and who wants to fail?

I try to get people to understand that your recovery and continuity programs do have areas in which you will fail – the testing process is to discover and fix those prior to the real event.  Hopefully, through testing, we can uncover the most damaging failure points in a controlled, testing environment rather than discover them when all Helsinki is breaking loose (see how I kept the Finland theme going there?)

But, alas, this fear of failing tests results in people jury-rigging the test and preparing months in advance; taking special back-ups; installing equipment and software; etc.  As long as the disaster gives us a month’s warning that it is on the way, we have proven we can recover.  But hey, we didn’t fail the test – Yippee.

I commend Finland for their courage to face their fear and try to cultivate a willingness to take chances in order to stimulate a down economy.  Who knows, maybe one day, people from Finland will make good business continuity, disaster recovery planners.

I wonder if I can get my organization to celebrate a Company Failure Day during our next test?