Archive for Disaster Recovery

PS-Prep: Why Get Certified?

For those of you who don’t know, PS–Prep is a voluntary private sector preparedness accreditation and certification program established by the US Department of Homeland Security as a direct result of a law passed by Congress following the Recommendations of the 9/11 Commission.

Basically, PS-Prep provides a means for private sector organizations that have business continuity, disaster recovery and emergency preparedness programs compliant with any one of three widely accepted planning standards to be certified by trained and approved Certifying Bodies (CB).

Although backed by Public Law 110-53, the need to be certified is not a law.  This is strictly a voluntary program.

So, the question is – Why get Certified?

This question is a topic of much debate amongst business continuity professionals, certifying bodies and the public authorities trying to promote PS-Prep.  I don’t think anyone is arguing against the benefits or principals behind PS-Prep, but rather, are skeptical that PS-Prep will provide any real added incentive to corporations to plan.  There is some discussion on the appropriateness of PS-Prep being a government initiative versus managed by a private sector forum, and there is some debate on whether or not PS-Prep has aligned itself with the right, or all of the right established standards, but these are arguments of the details and do not provide answer to the question, Why get certified?

I think many of the proponents of PS-Prep are answering the wrong question.  Much of the argument I hear supporting PS-Prep really simply answers the question, why do business continuity planning?  Why plan is a much different question than why get certified.

Although I have met up with violent opposition to my belief, I think the most compelling reason today supporting the benefit of being certified is to provide a defensible position for after-the-disaster litigation showing your organization had taken due care to protect your organization up to DHS supported standards.

Remembering that the answers; because it is a good business practice; it is necessary to stay in business; it protects your employees and corporate assets – are all answers to the question “why plan” and not “why get certified” – I think providing a certificate showing you planned to DHS standards as a defense in court helps support the PS-Prep initiative.

Another potential answer to “why certify” is to leverage a marketable position communicating that your organization has taken steps to protect its organization and assets consistent with the findings in the 9/11 Commission’s Report.   Should PS-Prep become a more recognizable label, including a banner or logo stating PS-Prep accredited in advertising and marketing material could have some benefit.

What DHS would love to see happen is for large, private companies to embrace PS-Prep and make it a requirement that their suppliers, vendors and partners be PS-Prep certified.  Should that start to occur, the answer to “why get certified” will be market-driven and accelerate the program tremendously.

One other impetuous that might help get PS-Prep going is to have insurance companies that offer loss of business insurance to discount these premiums for firms that are PS-Prep certified.

I hate sounding like a skeptic, but until you can show real marketable, return on investment reasons for certifying these programs, I just don’t see companies jumping on the PS-Prep band wagon.

But the debate is not over and PS-Prep is just starting to hit the headlines.  So, it should be interesting to see how this plays out over the next few months and years.  Regardless of PS-Prep acceptance however, business continuity planners should (and I believe most of the good ones do) continue to create programs consistent with and in compliance of the standards identified in the PS-Prep program.

Virtual Emergency Meeting Locations

I have been working with a few companies lately in reviewing their business continuity plans and strategies for individual business units.  Many of these plans include listing an off-site meeting location or department command center for managers to gather following a building evacuation and prior to opening an alternate site facility.  In many cases, this location is the head manager’s home or a local coffee shop or other public gathering place.

Whereas, I like the concept of gathering the managers for information sharing and decision making purposes, I like even more the use of a “virtual meeting place” through the use of conference, bridge calls.

I have been recommending that these individual departments utilize their existing conference bridge capabilities to initially get the decision makers together to assess the impacts on their employees and discuss their options for responding to and recovering from the incident.  Furthermore, I have suggested that, when a situation occurs where they are alerted of an incident preventing access to the primary facility, they establish a default meeting time via the conference bridge.  For example, the department plan could be, “Once alerted of a situation in one of our facilities housing department personnel or business functions, until such time as you are contacted otherwise, call into the bridge conference number every hour on the hour.”  I think this is a good default plan should other communication techniques or alerts not be viable at the time.  You call into the conference bridge on the top of every hour and see who else may be on the call and do the best you can to manage the situation.  Once other arrangements or schedules are made for this particular event, then you adjust from there.

This suggested strategy has been well received from all the management teams I have talked to and most of them have implemented this strategy in their plans.

Just thought I would share some free advice here in my blog.  If you like the suggestion and are thinking about using it or you have a better idea, I welcome you to share your comments.  Thanks.

Continuity Insights Management Conference 2012

Having a great time at the Continuity Insights Management Conference 2012 in Scottsdale, AZ.  This conference provides a terrific atmosphere for skilled and experienced practitioners to get together and share their experiences, successes and challenges.  There are also a number of new practitioners eager to listen and learn from those that have blazed the paths ahead of them.

Bob Nakao and team do a terrific job planning and delivering this conference and my hat goes off to all of those behind the scenes individuals that make allow us to see only the duck gracefully glide across the pond without even noticing the manic flapping of webbed feet beneath the surface.

I was awarded a prime spot on Monday to deliver my session on “Revisiting the BC/DR Methodology” to a packed room.  No one left and no one feel asleep so I consider it a success.  I am now planning to play my role as a panel member on an Ask the Experts session about Exercise Tools and Techniques.

Safe Harbor Consulting is hosting a hospitality suite and we are having fun hosting many new friends and, hopefully, future clients, in a relaxed and comfortable atmosphere between sessions and at day’s end.

Once again, thanks Bob and thanks CI for putting on such a fine show and on inviting Safe Harbor Consulting to participate in such an active way.

Work-from-home Solutions in Your Business Continuity Program

I am often asked my opinion about using a work-from-home solution as part of a Business Continuity strategy.  So, in this blog, I will give my opinion.

I am all for leveraging an already existing work-from-home capability in your business continuity program but I am against using business continuity reasons as the justification for; and, using business continuity budget dollars as the source for building out a work-from-home capability.

If there are other, legitimate business reasons for providing a work-from-home capability for a portion of your work force, than, by all means, take advantage of that during business interruption events in your production facilities.  But, there are just too many negative aspects and too many better solutions to spend your business continuity dollars on than providing a work-from-home capability.

Work-from-home solutions are a one-to-one strategy – capability provided that works only for one employee.  Should that employee not be available to participate during the contingency period, those resources are useless.  And, if you enforce an eight hour work period, which I think business continuity programs should, these resources are only useful during the time that one employee can work.  It is not reasonable to think that you could have other employees go to one employee’s home to utilize this capability.

Also, employees come and go.  Should that employee, who has work-from-home resources provided, decide to leave the company, or even just transfer to another position in the company, those resources either need to be redeployed or are no longer valid for business continuity purposes?

I think, if you are going to spend business continuity dollars on outside of the production facility working environments, they are better spent on centralized, work-area business recovery solutions.  Your typical alternate site work area solutions allow…

  • Resources to be used by a variety of different personnel; over shifts that can be utilized 24 hours a day.
  • The solution to be leveraged across a number of geographically distanced production facilities.
  • A solution that survives employee turnover.
  • A solution that can be leveraged during non-emergency times as training facilities or for other purposes.
  • A centralized solution to gather employees and better manage them through the crisis.

So no, I would not build out a work-from-home capability solely to support the business continuity program, but, if there are other legitimate business reasons, supported by outside the BCP budgets, than, yes, you should evaluate the benefits of utilizing this capability in your BCP strategy.

Also, however, I like to issue this word of caution:  Many business environments provide a work-from-home capability to allow employees to work outside of the office on special occasions for a variety of reasons.  These capabilities are used throughout the year with no real problems.  But, they are not used by everyone at the same time.  Often, the work-from-home capability supports a few users at any given time – for off hour access, or other rare occasions.  Accessing the production resources from home, over time, for all the employees having this capability does not necessarily prove that you can handle all employees using their work-from-home capability at the same time.  Companies that plan to rely on this capability during times of crisis must stress test that capability with high volume usage to ensure that the infrastructure can handle the capacity.

Hopefully, at a high level, I have adequately stated my position on this topic.  But, if you would like, we can talk about it more – give me a call … on my home phone.

Conducting a Test – Yes, a “Test”

Yeah, I know, I know … we don’t have “tests” we have “exercises”, because tests imply pass/fail and exercises imply getting stronger.  Yeah, I used to sing that same silly song, too.

I now understand that there are times when you need to test; times when you need to drill; and, times when you need to exercise.  You do need to test your solutions to make sure that they do, in fact, work.  You do want to know if you can “pass the test”.  You may refer to these as validation tests.

I’ll take it a step farther and would even like to see us test the people.  I think it would be great to gather our key business continuity players, managers and employees into a room and give them a regular, school-like, no. 2 leaded pencil, don’t start until I tell you to, and put down your pencils when instructed, actual tests.  Why not?

I would like to ask key players and managers questions that they should know about our Emergency Preparedness, Business Continuity and Disaster Recovery Programs.  These questions might include:

  • If the fire alarm went off right now:
  • What would you do?
  • Where would you find evacuation routes posted?
  • Where would you congregate once outside of the building?
  • Who are your floor wardens?
  • If you received a bomb threat on the phone, what would you do?
  • If you got a call at 2:00 am that the building had burnt down…
  • What would you do?
  • Who would you call?
  • Where would you go to work?
  • Where would you find your Business Continuity Plan?
  • If the Data Center experienced a disaster…
  • What applications that you use would be recovered?
  • In what timeframe?
  • What would be the status of the data?
  • What applications would not be recovered?
  • What business processes would you be expected to continue?
  • What business processes would be temporarily suspended?
  • If you do not know the answers to any of the questions on this test, where would you go to find them?
  • If we experienced a disaster and you weren’t available to participate in the recovery, who has been trained to play your role?  Have you trained them to be successful in this effort?

I could go on, but I want to try to make this an interactive blog and challenge you to post the types of questions you would want to include on this pencil to paper test.  You can do so by posting a comment to this blog.

You can exercise your solutions all you want.  You can physically recover hardware, applications, data, networks, etc., time after time – but there are some things that people need to know when the alarms go off or your ability to execute your plans will be severely hampered.

How do you think your organization would do if given this kind of test?  My bet is most companies would not fare too well.  There are historical cases where adequate recovery capabilities were in place but the people were not educated well enough to implement these capabilities at time of event.  How better to determine our level of preparedness than to give them a test?

You can exercise and get as strong as you want, but if you can’t pass the test … you will fail.

Continuity Insights 2012 Management Conference

The Continuity Insights 2012 Management Conference is scheduled for April 16 – 18 in Scottsdale, Arizona.  And, Safe Harbor Consulting will be there and well represented on the agenda.

We have been slated a terrific spot on the agenda with Joe Flach presenting his break-out session, “Revisiting the BC/DR Planning Methodology” (Session B4) on Monday, April 16 from 11:00 am – 12:00 noon.  Then, on Tuesday, April 17 from 9:45 – 11:00 am, Mr. Flach will be a panel member in a break-out session on “Exercise Facilitation Techniques” (Session G4).

You can register through the Continuity Insights website and enjoy early registration discounts.

Safe Harbor Consulting will also be hosting a Hospitality Suite in the “Talking Stick Resort”, where the conference is being held, and we look forward to meeting and entertaining you there.  We will post the room number for the hospitality suite on the Conference Bulletin Board at the conference.

We are looking forward to some fun in the sun; interesting and educating sessions; and good times with good friends.  Let us know if you are planning to attend the conference so we can be sure to connect in Scottsdale.

Removing the Fluff from the Stuff

I understand and appreciate the need to document your Business Continuity, mission, objectives, process, justification and analytical results.  I think all of this information is valuable in advancing and promoting your programs; educating management and participants on your program; and, providing a foundation for internal and external audits.  And, I think it is important to mandate that certain, key players in your program read and understand all of this information.

But …  Oh, come on, you knew there was a “but” coming.  But, I don’t think that stuff belongs in the same document that includes your “at-time-of-interruption” action plans.

Too often, I see organizations create one tomb of everything Business Continuity and call that their Business Continuity Plan.  In my opinion, you should have the book that explains everything business continuity related and then a stand-alone “Action Plan” that is you instruction manual of what to do when the lights go out.  In fact, you should probably create a whole bunch of “Action Plan”s – one for each unique function in your continuity, recovery and emergency response program.

I think reading and understanding “the book” prior to an emergency is important and should be mandated.  However, much of the stuff in “the book” just gets in the way at time of event.

“The book” should: set the premise for your program; identify objectives, assumptions and givens; explain, holistically how teams work together; provide background on how and why strategies and solutions were selected; explain the entire planning and implementation process – all stuff that is important to know, prior to an interruption, but just becomes a burden to flip through when trying to figure out just what to do now that an event is occurring.

The “Action Plan”s should be concise and detailed, step-by-step, vetted through testing, tactical tasks to be undertaken once a disaster is declared.  These plans need to be easy to follow (relatively) easy to reference and to enough detail that a backup team member, with less experience in the process, can follow and successfully implement.

I have seen organizations get very creative in producing wallet sized plans, tri-folds and/or thin plans that get right to the point and, in some cases, provide a way to record which steps were executed along the way.

There are a number of software products that also help maintain these action plans and allow you to track the implementation process at time of disaster.  These can be very powerful emergency management tools if properly implemented and monitored.  (They can also become just as cumbersome and useless as the War and Peace-like plan if implemented incorrectly.)

So look at your documentation and ask yourself if you have successfully separated the wheat from the chaff?  Do your team members have a useful tool to help them execute their recovery responsibility at time of disaster without having to flip through pages of “fluff” to find out exactly what they are supposed to do?  And, when you test your program, do you make sure these plans are being referenced and updated to ensure the documentation matches the reality?

More on this topic to come in future blogs, but – I use that “but” word a lot, don’t I – I think that is enough for today.  Thanks for visiting our blog.

Deadly Volcanoes

Last night I stumbled upon an interesting episode of “Nova” on PBS – “Deadliest Volcanoes”

Now I am not suggesting everyone update their emergency preparedness and business continuity plans to prepare for a volcano eruption, but it did present a pretty scary scenario of just how devastating a volcano can be.  We even have recent history of how volcano ash clouds can be very disruptive to the air travel industry with the recent eruptions in Iceland – (Eyjafjallajökull in 2010) and Alaska (Mt. Redoubt in 2009)

There were stories included about potential eruptions all over the world, including some relatively highly populated regions, including Naples, Italy; Japan; Yellowstone and others.  The Yellowstone situation is actually pretty interesting, because it is not what one would normally think about when considering volcanic eruptions.  The Yellowstone “super volcano” does not include the cone shaped mountain spout that most of us associate with volcanoes. 

Then they started talking about the volcano that practically sits in my backyard!  I am awed by the sight of Mount Rainier each and every clear day that she appears on my horizon.  I have lived here for only 4 years, but natives of the area tell me she is always an amazing sight that you will never get used to.  I knew Mt. Rainier was an active volcano – similar to her sister mountain, Mt. St. Helens, which erupted relatively recently in 1980 with significant damages being incurred – but, never really thought about the risk too much.  Well, this episode has given me a little greater appreciation of what could be in our future.  Interestingly enough, this segment suggested that the eruption itself, as devastating as it may be, would probably be the least of our worries.  No, there is a phenomena known as a Lahar, which is a catastrophic mud and rock slide that flows down the volcano into the valleys below.  A Lahar caused by an eruption in Mt. Rainier has the potential to reach all the way to Seattle destroying much of what lies in its path.  The nearby town of Orting, WA, even has a Lahar warning system installed in their community. 

I found this episode to be very educational and informative.  You may want to watch it, too.  Unfortunately for me, my seven year old son was listening to the show from another room and is now terrified by that beautiful mountain that we often hike near and around.  I hope he is not so scared that he won’t want to take another hike out there with me – it is truly inspiring. 

Check out this show if you have time – and, check out the risks that might be near your places of business.

Goodbye 2011; Hello 2012

My how time flies.

An entire month went by without a single blog being posted to this site.  What was I thinking!?

The good news is:

  • Business at Safe Harbor Consulting has picked up and I have been busy spending time on other, income generating tasks.
  • I took time off in December to spend with my family and friends and let this blog take second priority for a while.
  • There have been no real big, disaster events demanding my attention and prompting me to write a blog.

The bad news is: 

  • I have been experiencing a little writer’s block and have been challenged to come up with new topics to write about.
  • I have received little feedback from blog readers on my postings and have wondered about the effectiveness of this tool.
  • Business at Safe Harbor Consulting has picked up and I have been busy spending time on other tasks.  (Sometimes, news is both good and bad.)

But alas, it is time for the New Year’s resolutions to kick in and I must, at least, try to fulfill one resolution before the month of January expires.

I hope everyone had a terrific holiday season and a safe and happy New Years.

2011 was an interesting and eventful year for many of us in the Business Continuity and Emergency Management arena.  If the Mayans are correct, 2012 will prove to be even more so.

If you have any subjects or ideas you would like me to try to tackle in the next few blog entries, by all means, let’s hear them.  Otherwise, I may ask you to bear with me as I repeat a few topics, with perhaps, a new spin or refocused emphasis.

May 2012 bring you much joy, happiness and few disasters – at least none that you can’t recover from.

Happy New Year,

Joe Flach

Critical Data: Don’t Overlook the Hardcopy

I know we like to think we now work in a paperless society, but the fact is, we do not.  There are still plenty of industries and processes that rely on hardcopy documentation for historical records and in support of daily operations.  Business Continuity and Disaster Recovery programs often overlook these vital records as they focus on technology and electronic medium – I caution you not to fall into this same trap.

In know this to be true, especially in airlines, medical and educational organizations as well as in some financial services and other industries. 

For example:

Airlines are required to maintain and have access to all mechanical and maintenance records for each and every aircraft that they fly.  In many instances maintenance initiatives issued by various agencies are printed and given to the mechanics and engineers who then make handwritten notations and sign off on the printed form.  These printed forms, with their notations, become the official record of the maintenance activity in compliance with the initiative.  Should this physical, hardcopy record be destroyed or lost, the plane (or an entire fleet of planes) will have to be grounded until the maintenance check is performed once again and a new record created.  Some airlines maintain these records in a single location and do not scan or digitally record the information (keeping costs down, you know).  Should the facility housing these documents go up in smoke, it could take months or longer to recreate the audit trail for those planes – which, by law, must be grounded until proof that all the maintenance initiatives have been completed.

Many medical offices maintain a slew of forms and doctor reports in handwritten form.  Just notice all the filing cabinets up and down the halls in your doctor’s office.  These records are seldom scanned or stored electronically and are susceptible to numerous risks and threats.  The same is true for school records and other information gathered in handwritten forms.

Financial services firms and brokerages still house plenty of hardcopy documents in the form of payment instructions and customer documentation that could cause plenty of financial exposure and compliance irregularities if lost or destroyed.

For those of you who think that we operate in a paperless society, just take a look around and count the number of filing cabinets still in use.  What do you suppose is kept in all this space?  And, what would be the cost or impact to the organization if they were permanently destroyed?

Now, I am not saying this is true in every environment.  Certainly there are many, many offices and industries that truly have no exposure to hardcopy documentation and information.  I am just suggesting that your risk analyses, impact analyses and recovery requirements analyses do not simply overlook this potentially critical information base and include consideration of this potentially risky business practice.

Backing up or electronically scanning and storing hardcopy documentation, especially historical documentation, may be something your organization needs to look into.  There are plenty of vendors that can help you achieve this end.