Tag Archive for disaster recovery

Are RTO’s Stagnant? Should They Be?

In many business continuity programs, there are known and established Recovery Time Objectives (RTO) for business processes and for IT applications.  More times than not, these RTOs are static and the response and recovery programs are built around these numbers as they came out of a Business Impact Analyses (or were merely assigned based on an educated guess).

I just wonder if it is reasonable to assume that recovery priorities remain the same throughout time.  And I am not necessarily questioning whether they remain the same over time – but are they the same at different points in time throughout the business year or business cycle?

For example, is it reasonable to assume that our recovery priorities, or RTOs, might be different if the disaster occurred at month end or at year end as opposed to some other time in the year?  Might our recovery priorities be different if we are in the middle of launching a new campaign or product or service?

And, could the disaster itself influence our recovery priorities?  Could RTOs be different if we experience a data center only disaster versus a disaster that also impacted our workforce?  Could our recovery priorities be different for a single-site disaster versus a multiple site disaster?  Could we have different RTOs if we knew that the downtime was going to be hours or days versus weeks or months?

Now, I hate to over-complicate things in the planning process.  I am always warning folks to avoid paralysis by analysis in the planning process, but I think these are legitimate questions to pose to mature and solid programs that are looking to continue to improve and strengthen their recovery posture.

This is also why I think it is important for your recovery program to include a well thought out and implemented Crisis Management component that gets the right decision makers together and empowers and enables them to make changes to the recovery process as the situation, at that time, dictates.  So, maybe we maintain our single RTO, but we have the infrastructure in place that can accommodate changes in our recovery priority if and when needed.

Just something for the experienced planners to thing about and challenge their teams to consider in the maintenance and improvement process.

Having Plans vs Being Prepared – Avoid the Oops

I have recently posted a couple of blogs discussing the difference between Planning and having Plans.  In this blog, I want to explore the difference between Having a Plan and Being Prepared.

I have been in a number of environments where I thought the organization had great business continuity or disaster recovery plans – but, I did not believe that they were prepared to recover from a business interruption event.

Most plans rely on a number of “enablers” that have to be in place in order for the plan to be successfully executed.

First and foremost, the physical environment that the plan relies on has to be in place.  I have gone into a number of situations where the Executive Teams were convinced that their planning team had put great plans in place and I had to be the one to tell them that the plans were based on infrastructure not yet put into place.  “Yes, your plan is to recover applications in an alternate recovery site, which is a terrific plan … but you have not invested in or built out that site yet.”  Oops.

Secondly, the plan must be socialized and known by those who must manage to the plan.  I have seen some great plans sitting on shelves, known by only those few who wrote the plan – but all the people that would have to oversee and manage the execution of the plan had never read or been educated on the plan.  Oops.

And, third, in order to really be prepared, you must test, exercise and drill the plan.  It is through tests that you validate the correctness of the plan; through exercises that you discover ways to improve the plan; and through drills that you condition people on how to respond when executing the plan.  I have been in many environments where the plan may be understood by everyone, but never physically put into action to see if it will actually achieve the intended results.  Oops.

So there is much more to being prepared than to simply having a good plan.

Having just passed the anniversary of the D-Day invasion, perhaps that will serve as an example of what I am talking about.  There were relatively few people that actually “planned” the invasion.  And only a few more that were educated on the plan.  But, 10’s of thousands of others that had to “prepare” for it, in order for the plan to work.

It only takes a few people to create a good plan … but, it takes an entire organization to be prepared.

Don’t let your good plans fail because of an oops.

More Thoughts on Planning and Plans

Mike Tyson is quoted as saying, “Everyone has a plan, ‘till they get punched in the mouth.”

How well do your plans stand up to the punch in the mouth?

Field Marshal Helmuth von Moltke put it this way, in a more familiar quote, “No plan survives contact with the enemy.”

In our case, the enemy is the disaster or business interruption event we are planning for.

And, Arthur C. Clarke, had this observation, “All human plans [are] subject to ruthless revision by Nature, or Fate, or whatever one preferred to call the powers behind the Universe.”

The point is; whatever you had in mind when developing your business continuity, emergency response or disaster recovery plans, the event you will have to respond to will be nothing like what you envisioned.  Now, I know many of you are thinking, “That is why we do not plan for particular scenarios, we plan for the impacts of scenarios!”  But, I still say, you cannot plan without certain assumptions and certain biases about how the response will take place or how the crisis will unfold – and, I suggest, it won’t happen that way.

This is why I always like to look for evidence in a plan that you have provided the framework for decision makers to get together, make changes to the plan as needed, and, have the means to communicate these decisions to those who need to know this information.

I happen to believe in what Lester Robert Bittel had to say about planning, “Good plans shape good decisions.”  But, it is important to understand that not all decisions are made ahead of the event and the good plan must lay the foundation for at-time-of-disaster decisions to be made to adjust the plan based on how the enemy is responding.

Now, I happen to make a good living from helping organizations create, document and test crisis management, emergency response, business continuity and disaster recovery plans.  So, I would not dare under-emphasize the importance of planning – but, like some of the quotes I will share below – I think the value gained is in the planning process and not so much in the plans.

Dwight D. Eisenhower said it this way, in a quote that is often repeated, “In preparing for battle I have always found that plans are useless, but planning is indispensable.”

Dr. Gramme Edwards paraphrases it this way, “It’s not the plan that’s important, it’s the planning.”

Indeed!  It is in the planning process where we build out solutions, implement recovery capabilities and exercise our abilities to respond.  This is the real value and the enablers that will allow us to survive the business interruption event.  The written plan, with step-by-step instructions for how we operate, sometimes for weeks after the event – will hardly ever be referenced and certainly, not referenced after the first 24 hours.  I do believe that those decisions we made before the event that provide action steps within the first few hours of an event can be valuable – but once decision makers get together and have the luxury of a little time to figure out where we currently stand – decisions made before the event occurred will have less value.

The capabilities we have in place because of the planning process will be the key to our survival.  How we utilize those capabilities will require flexibility based on the event itself.

Winston Churchill said, “Those who plan do better than those who do not plan even thou they rarely stick to their plan.”

I think that is a much better way of saying what I mean!

I do run up against “pride of authorship” when I evaluate written plans – and I understand and completely empathize with that.  I am guilty of the same.

But, Publilius Syrus says, “It’s a bad plan that admits of no modification.”

I do believe in the power of planning.  And, I agree that planning is essential.

Although attributed to many different people, I think Tariq Siddique says it best and simplest, when he states, “If you are failing to plan, you are planning to fail.”  (This quote is often attributed to Benjamin Franklin, who may have said the same thing or something very similar.)

And, I couldn’t agree more with Sun Tzu in his The Art of War, when he suggests, “Plan for what is difficult while it is easy.”

This is why we must plan before the disaster.  Not only because we do not have the luxury of time to plan afterwards, but because the planning process is easier lacking the chaos and confusion that will accompany the disaster.

But, remember, it is the planning that is important and the resulting capabilities put in place during the planning process.  The plans themselves, may not be what is needed to get you through the particular crisis you are responding to.

Hillel J. Einhorn states, “In complex situations, we may rely too heavily on planning and forecasting and underestimate the importance of random factors in the environment. That reliance can also lead to delusions of control.”

I think our plans need to allow for the flexibility to respond to these random factors.  And, yes, I do think some of us have “delusions of control” when it comes to assessing our state of readiness.

I want to end with two more thoughts on planning.  I have witnessed so many programs lacking progress because of their desire to create the perfect plan.

George Patton is quoted as saying, “A good plan today is better than a perfect plan tomorrow.”

I agree.

And, lastly, when exercising our plans and our recovery capabilities, I so often find planners who like to assign pass/fail grades to the tasks.  I like to rely on what Thomas Edison had to say about failures, “I have not failed.  I’ve just found 10,000 ways that won’t work.”

There, I think I have reached my quota of quotes.  If you made it all the way to the end of this blog – I applaud you.  Thanks.

If you have a favorite quote to share with us, please do so by adding a comment.

Disaster Recovery Planning vs. Disaster Recovery Plans

So often, when we are engaged to review existing business continuity and disaster recovery plans, we find volumes of “plans” with very important planning information but very little in the way of action plans for at-time-of-recovery activity.

By this, I mean, many “plans” include information discovered in the BIA and Risk Analyses.  There are tables and reports on what the impacts are for being down, what the requirements are in a recovery center, how many desks are needed in a recovery site, special equipment requirements, special forms, vital records listings and locations, what the critical applications are, RTO’s, RPO’s, vendor listings, employee listings, and on and on and on.

All of this information is CRITICAL INFORMATION for designing a recovery solution, but is of no real value at time of an incident.

At time of disaster, I need to know how to engage the plans and how to employ the capabilities that are provided –based on all that information listed above.

In my opinion, this information should be segregated.  When a business interruption event occurs, I do not care what the findings were in the BIA or RA – all I want to know is what is in place now, how do we get to it and what do we do when we get there.

I review many plans that pass the weight test but are so full of “noise” and so loaded with information that they become too bulky and are not usable as an action plan for what we do.

Sometimes it can be as simple as separating the two parts of the plan – many times, the “action plan” component is missing altogether.  This is sometimes especially true when a database software tool is used.  The database reports look so good and fill up so many pages, people think that that is the plan.  No, that is a collection of information needed to ensure we put the proper capability in place, but is not the action plan for how we employ that capability.

Practical, pragmatic, easy-to-use action plans are hard to come by, but, what I am most interested in finding when asked to review an organization’s level of response preparedness.

Do not confuse a compilation of information gathered in the planning process as being your disaster recovery plan.

Business Continuity Planning – Beyond the Doomsday Scenario

At the Continuity Insights Management Conference 2012 that I recently attended in Scottsdale, AZ, there was a lot of conversation around PS-Prep which bled into the discussion of “Why get certified” or, the more generic question of, “Why perform business continuity planning?”  An oft repeated answer to this question, echoed by business continuity planners around the world is, “Because without a plan you will not survive as a company.”

I think this is a disingenuous answer without any history to support it.  Where exactly is the evidence of this fact?  What historical data can you share with me, or the CEO you are trying to convince, that this is the case?  I am confident that you can dig up cases of small companies that did not survive a disaster, but where is that story about the big guy who did not survive the disaster?

The one and only case study I can think of off the top of my head is Enron, but that was a disaster of a different kind.

Look at BP and the horrific Gulf Coast disaster – they survived.  Did they have a plan in place for this?  Maybe … if so, most professionals would argue against its effectiveness.  Were they certified?  No.

Look at Cantor Fitzgerald, the one company most widely spoke about concerning the extent of their losses during the events of 9/11.  Survived.  With much loss and many significant challenges, but they are still in business.

We found this article that lists 8 Infamous Business Disasters – those companies all survived – albeit some under a new name and different business model, but they did survive.  Now, not all of these cases are the kinds of disasters we plan for, but I can’t find that one poster child event that proves the statement, “Without a business continuity plan, you will not stay in business.”

Now look, I am a business continuity planner.  I make a living out of helping companies put these programs in place.  I want … no, I NEED … CEO’s and Boards of Directors to embrace the need for these plans and to invest in professionals like me to help put them in place.  But, I think we need a better sales pitch than the shallow threat of; this is needed to survive a disaster.

I don’t think we need C-level executives to buy into this all or nothing proposition with business continuity planning.  No, I think that the message should be:  Business continuity plans will allow us to mitigate our losses should a disaster occur. The goal is to ensure the investment we make in our plans and solutions is justified by the potential losses that could occur considering the probability that an event happens.

The losses that could occur is measured by performing a Business Impact Analysis and the probability that an event happens is measured by a Risk Analysis.

We plan because it is a reasonable business practice to protect our assets and our stakeholders against losses that could impact the market value of our company not just if, but when, a business interruption event occurs.  If you want the answer to, “Why get certified”, check out this earlier blog we posted.

We need to sell business continuity planning using business terms that executives can understand and stop with the doomsday scenario selling technique.  At least, that’s the way I see it.

In the meantime, if you can share those stories with me that support the position companies will not survive without plans, I would love to read them.  Thanks.

Recovery Options Evaluation Criteria

When assessing recovery options for both technology and workarea, we recommend creating an options comparison chart with the following evaluation criteria:

  • Costs  (1x, Recurring, ATOD)
  • Accessibility
  • Testability
  • Scalability
  • Flexibility
  • Compatibility
  • Lead Time to Implement
  • Proximity
  • Shared Risks
  • Solution Complexity
  • Burden of Obsolescence
  • Ability to Meet Recovery Objectives

These criteria can be evaluated and scored with a relative value, such as High, Neutral, Low or a tailored value for each criteria, such as Most Complex, Neutral, Least Complex, etc.

Simply as an example, the final “scorecard” might look something like this:

Evaluation Criteria

Internal Recovery Solution

Hosted Recovery Solution

Recovery Service Vendor

Costs
    One Time Costs

$ 5.0 mil

$500k

Minimal

    Recurring Costs

Minimal

$5k/mo

$5k/mo

    At-Time-Of-Use Costs

Minimal

Minimal

$15k + $2k/day

Accessibility

Very Accessible

Mostly Accessible

Limited Accessibility

Testability

Easy to Test

Moderate

Must be Scheduled

Scalability

Limited

Moderate

Very Scalable

Flexibility

Limited

Moderate

Very Flexible

Compatibility

Very Compatible

Very Compatible

Somewhat Compatible

Lead Time to Implement

Long

Moderate

Short

Proximity

Nearby

Options

Options

Shared Risks

Some

Few

Few

Solution Complexity

Moderate

Moderate

Complex

Burden of Obsolescence

On Our Company

On Our Company

On Recovery Vendor

Ability to Meet Recovery     Objectives

Fully

Fully

Mostly

Security

Secured

Somewhat Secured

Somewhat Secured

Availability at time of     Need

Total

Total

At Risk

Facility Management     Requirement

Internal

External

External

 

Each value may bear farther explanation and justification, but the summary is nice to display in a side-by-side comparison format.

If you want a more detailed explanation of any of the Evaluation Criteria or want to suggest others to include in a full analysis, please feel free to share your thoughts by entering a comment.

 

PS-Prep: Why Get Certified?

For those of you who don’t know, PS–Prep is a voluntary private sector preparedness accreditation and certification program established by the US Department of Homeland Security as a direct result of a law passed by Congress following the Recommendations of the 9/11 Commission.

Basically, PS-Prep provides a means for private sector organizations that have business continuity, disaster recovery and emergency preparedness programs compliant with any one of three widely accepted planning standards to be certified by trained and approved Certifying Bodies (CB).

Although backed by Public Law 110-53, the need to be certified is not a law.  This is strictly a voluntary program.

So, the question is – Why get Certified?

This question is a topic of much debate amongst business continuity professionals, certifying bodies and the public authorities trying to promote PS-Prep.  I don’t think anyone is arguing against the benefits or principals behind PS-Prep, but rather, are skeptical that PS-Prep will provide any real added incentive to corporations to plan.  There is some discussion on the appropriateness of PS-Prep being a government initiative versus managed by a private sector forum, and there is some debate on whether or not PS-Prep has aligned itself with the right, or all of the right established standards, but these are arguments of the details and do not provide answer to the question, Why get certified?

I think many of the proponents of PS-Prep are answering the wrong question.  Much of the argument I hear supporting PS-Prep really simply answers the question, why do business continuity planning?  Why plan is a much different question than why get certified.

Although I have met up with violent opposition to my belief, I think the most compelling reason today supporting the benefit of being certified is to provide a defensible position for after-the-disaster litigation showing your organization had taken due care to protect your organization up to DHS supported standards.

Remembering that the answers; because it is a good business practice; it is necessary to stay in business; it protects your employees and corporate assets – are all answers to the question “why plan” and not “why get certified” – I think providing a certificate showing you planned to DHS standards as a defense in court helps support the PS-Prep initiative.

Another potential answer to “why certify” is to leverage a marketable position communicating that your organization has taken steps to protect its organization and assets consistent with the findings in the 9/11 Commission’s Report.   Should PS-Prep become a more recognizable label, including a banner or logo stating PS-Prep accredited in advertising and marketing material could have some benefit.

What DHS would love to see happen is for large, private companies to embrace PS-Prep and make it a requirement that their suppliers, vendors and partners be PS-Prep certified.  Should that start to occur, the answer to “why get certified” will be market-driven and accelerate the program tremendously.

One other impetuous that might help get PS-Prep going is to have insurance companies that offer loss of business insurance to discount these premiums for firms that are PS-Prep certified.

I hate sounding like a skeptic, but until you can show real marketable, return on investment reasons for certifying these programs, I just don’t see companies jumping on the PS-Prep band wagon.

But the debate is not over and PS-Prep is just starting to hit the headlines.  So, it should be interesting to see how this plays out over the next few months and years.  Regardless of PS-Prep acceptance however, business continuity planners should (and I believe most of the good ones do) continue to create programs consistent with and in compliance of the standards identified in the PS-Prep program.

Virtual Emergency Meeting Locations

I have been working with a few companies lately in reviewing their business continuity plans and strategies for individual business units.  Many of these plans include listing an off-site meeting location or department command center for managers to gather following a building evacuation and prior to opening an alternate site facility.  In many cases, this location is the head manager’s home or a local coffee shop or other public gathering place.

Whereas, I like the concept of gathering the managers for information sharing and decision making purposes, I like even more the use of a “virtual meeting place” through the use of conference, bridge calls.

I have been recommending that these individual departments utilize their existing conference bridge capabilities to initially get the decision makers together to assess the impacts on their employees and discuss their options for responding to and recovering from the incident.  Furthermore, I have suggested that, when a situation occurs where they are alerted of an incident preventing access to the primary facility, they establish a default meeting time via the conference bridge.  For example, the department plan could be, “Once alerted of a situation in one of our facilities housing department personnel or business functions, until such time as you are contacted otherwise, call into the bridge conference number every hour on the hour.”  I think this is a good default plan should other communication techniques or alerts not be viable at the time.  You call into the conference bridge on the top of every hour and see who else may be on the call and do the best you can to manage the situation.  Once other arrangements or schedules are made for this particular event, then you adjust from there.

This suggested strategy has been well received from all the management teams I have talked to and most of them have implemented this strategy in their plans.

Just thought I would share some free advice here in my blog.  If you like the suggestion and are thinking about using it or you have a better idea, I welcome you to share your comments.  Thanks.

Continuity Insights Management Conference 2012

Having a great time at the Continuity Insights Management Conference 2012 in Scottsdale, AZ.  This conference provides a terrific atmosphere for skilled and experienced practitioners to get together and share their experiences, successes and challenges.  There are also a number of new practitioners eager to listen and learn from those that have blazed the paths ahead of them.

Bob Nakao and team do a terrific job planning and delivering this conference and my hat goes off to all of those behind the scenes individuals that make allow us to see only the duck gracefully glide across the pond without even noticing the manic flapping of webbed feet beneath the surface.

I was awarded a prime spot on Monday to deliver my session on “Revisiting the BC/DR Methodology” to a packed room.  No one left and no one feel asleep so I consider it a success.  I am now planning to play my role as a panel member on an Ask the Experts session about Exercise Tools and Techniques.

Safe Harbor Consulting is hosting a hospitality suite and we are having fun hosting many new friends and, hopefully, future clients, in a relaxed and comfortable atmosphere between sessions and at day’s end.

Once again, thanks Bob and thanks CI for putting on such a fine show and on inviting Safe Harbor Consulting to participate in such an active way.

Work-from-home Solutions in Your Business Continuity Program

I am often asked my opinion about using a work-from-home solution as part of a Business Continuity strategy.  So, in this blog, I will give my opinion.

I am all for leveraging an already existing work-from-home capability in your business continuity program but I am against using business continuity reasons as the justification for; and, using business continuity budget dollars as the source for building out a work-from-home capability.

If there are other, legitimate business reasons for providing a work-from-home capability for a portion of your work force, than, by all means, take advantage of that during business interruption events in your production facilities.  But, there are just too many negative aspects and too many better solutions to spend your business continuity dollars on than providing a work-from-home capability.

Work-from-home solutions are a one-to-one strategy – capability provided that works only for one employee.  Should that employee not be available to participate during the contingency period, those resources are useless.  And, if you enforce an eight hour work period, which I think business continuity programs should, these resources are only useful during the time that one employee can work.  It is not reasonable to think that you could have other employees go to one employee’s home to utilize this capability.

Also, employees come and go.  Should that employee, who has work-from-home resources provided, decide to leave the company, or even just transfer to another position in the company, those resources either need to be redeployed or are no longer valid for business continuity purposes?

I think, if you are going to spend business continuity dollars on outside of the production facility working environments, they are better spent on centralized, work-area business recovery solutions.  Your typical alternate site work area solutions allow…

  • Resources to be used by a variety of different personnel; over shifts that can be utilized 24 hours a day.
  • The solution to be leveraged across a number of geographically distanced production facilities.
  • A solution that survives employee turnover.
  • A solution that can be leveraged during non-emergency times as training facilities or for other purposes.
  • A centralized solution to gather employees and better manage them through the crisis.

So no, I would not build out a work-from-home capability solely to support the business continuity program, but, if there are other legitimate business reasons, supported by outside the BCP budgets, than, yes, you should evaluate the benefits of utilizing this capability in your BCP strategy.

Also, however, I like to issue this word of caution:  Many business environments provide a work-from-home capability to allow employees to work outside of the office on special occasions for a variety of reasons.  These capabilities are used throughout the year with no real problems.  But, they are not used by everyone at the same time.  Often, the work-from-home capability supports a few users at any given time – for off hour access, or other rare occasions.  Accessing the production resources from home, over time, for all the employees having this capability does not necessarily prove that you can handle all employees using their work-from-home capability at the same time.  Companies that plan to rely on this capability during times of crisis must stress test that capability with high volume usage to ensure that the infrastructure can handle the capacity.

Hopefully, at a high level, I have adequately stated my position on this topic.  But, if you would like, we can talk about it more – give me a call … on my home phone.