Tag Archive for risk analysis

The Adjusted Recovery Confidence Factor – Repeat Blog

Over the past few weeks I have actually had a few people ask me to send them the link to an earlier blog I posted about an Adjusted Recovery Confidence Factor.  Since there actually seems to be some interest in this idea – and, since I am really busy working on client deliverables – I have decided to take a blog short cut today and simply redirect you to an article we posted a few months back.

The Adjusted Recovery Confidence Factor

We had much less blog site traffic when this was originally posted so maybe its not a bad idea to put it out there again.  Any thoughts?  We would love to receive more comments on our postings.

Thanks.  And now, back to work.

Business Continuity Planning – Beyond the Doomsday Scenario

At the Continuity Insights Management Conference 2012 that I recently attended in Scottsdale, AZ, there was a lot of conversation around PS-Prep which bled into the discussion of “Why get certified” or, the more generic question of, “Why perform business continuity planning?”  An oft repeated answer to this question, echoed by business continuity planners around the world is, “Because without a plan you will not survive as a company.”

I think this is a disingenuous answer without any history to support it.  Where exactly is the evidence of this fact?  What historical data can you share with me, or the CEO you are trying to convince, that this is the case?  I am confident that you can dig up cases of small companies that did not survive a disaster, but where is that story about the big guy who did not survive the disaster?

The one and only case study I can think of off the top of my head is Enron, but that was a disaster of a different kind.

Look at BP and the horrific Gulf Coast disaster – they survived.  Did they have a plan in place for this?  Maybe … if so, most professionals would argue against its effectiveness.  Were they certified?  No.

Look at Cantor Fitzgerald, the one company most widely spoke about concerning the extent of their losses during the events of 9/11.  Survived.  With much loss and many significant challenges, but they are still in business.

We found this article that lists 8 Infamous Business Disasters – those companies all survived – albeit some under a new name and different business model, but they did survive.  Now, not all of these cases are the kinds of disasters we plan for, but I can’t find that one poster child event that proves the statement, “Without a business continuity plan, you will not stay in business.”

Now look, I am a business continuity planner.  I make a living out of helping companies put these programs in place.  I want … no, I NEED … CEO’s and Boards of Directors to embrace the need for these plans and to invest in professionals like me to help put them in place.  But, I think we need a better sales pitch than the shallow threat of; this is needed to survive a disaster.

I don’t think we need C-level executives to buy into this all or nothing proposition with business continuity planning.  No, I think that the message should be:  Business continuity plans will allow us to mitigate our losses should a disaster occur. The goal is to ensure the investment we make in our plans and solutions is justified by the potential losses that could occur considering the probability that an event happens.

The losses that could occur is measured by performing a Business Impact Analysis and the probability that an event happens is measured by a Risk Analysis.

We plan because it is a reasonable business practice to protect our assets and our stakeholders against losses that could impact the market value of our company not just if, but when, a business interruption event occurs.  If you want the answer to, “Why get certified”, check out this earlier blog we posted.

We need to sell business continuity planning using business terms that executives can understand and stop with the doomsday scenario selling technique.  At least, that’s the way I see it.

In the meantime, if you can share those stories with me that support the position companies will not survive without plans, I would love to read them.  Thanks.

Critical Data: Don’t Overlook the Hardcopy

I know we like to think we now work in a paperless society, but the fact is, we do not.  There are still plenty of industries and processes that rely on hardcopy documentation for historical records and in support of daily operations.  Business Continuity and Disaster Recovery programs often overlook these vital records as they focus on technology and electronic medium – I caution you not to fall into this same trap.

In know this to be true, especially in airlines, medical and educational organizations as well as in some financial services and other industries. 

For example:

Airlines are required to maintain and have access to all mechanical and maintenance records for each and every aircraft that they fly.  In many instances maintenance initiatives issued by various agencies are printed and given to the mechanics and engineers who then make handwritten notations and sign off on the printed form.  These printed forms, with their notations, become the official record of the maintenance activity in compliance with the initiative.  Should this physical, hardcopy record be destroyed or lost, the plane (or an entire fleet of planes) will have to be grounded until the maintenance check is performed once again and a new record created.  Some airlines maintain these records in a single location and do not scan or digitally record the information (keeping costs down, you know).  Should the facility housing these documents go up in smoke, it could take months or longer to recreate the audit trail for those planes – which, by law, must be grounded until proof that all the maintenance initiatives have been completed.

Many medical offices maintain a slew of forms and doctor reports in handwritten form.  Just notice all the filing cabinets up and down the halls in your doctor’s office.  These records are seldom scanned or stored electronically and are susceptible to numerous risks and threats.  The same is true for school records and other information gathered in handwritten forms.

Financial services firms and brokerages still house plenty of hardcopy documents in the form of payment instructions and customer documentation that could cause plenty of financial exposure and compliance irregularities if lost or destroyed.

For those of you who think that we operate in a paperless society, just take a look around and count the number of filing cabinets still in use.  What do you suppose is kept in all this space?  And, what would be the cost or impact to the organization if they were permanently destroyed?

Now, I am not saying this is true in every environment.  Certainly there are many, many offices and industries that truly have no exposure to hardcopy documentation and information.  I am just suggesting that your risk analyses, impact analyses and recovery requirements analyses do not simply overlook this potentially critical information base and include consideration of this potentially risky business practice.

Backing up or electronically scanning and storing hardcopy documentation, especially historical documentation, may be something your organization needs to look into.  There are plenty of vendors that can help you achieve this end.

Disaster Preparedness: A Risky Combination

At the risk of having a bunch of folks attack me for being an alarmist and pointing out how uninformed I am about the real vs. perceived dangers from nuclear power plants, I am going to go ahead and post this blog any way.

I am working on a couple of unrelated projects with tabletops and risk analyses.  One company is planning a tabletop exercise around an incident at a nuclear generating plant near one of their campuses and another is concerned about potential risks from earthquakes.  Just as a hoot, I thought I would combine the two risks and do an Internet search on “nuclear power plants near earthquake fault lines”.  And, I thought I would pass along what I found and let you decide if this is worth losing any sleep over.

Now, I am going to post a few links to stories I found that suggest there might be a concern with nuclear sites near fault lines.  I do recognize many of these websites as being those that others have told me are “whackos with an agenda” – but, then again, perhaps some of those folks who are just as adamant that this is all a bunch of hyped up fear mongering, have a bit of an agenda themselves?  I’m sure it’s probably somewhere in between; maybe not as bad as some of these sites might suggest, but, maybe a bit more risky than some nuclear specialists are willing to admit.  Anyway, I am not an authority on either side of this argument, just passing on some information I found.  I will let you be the judge.

STORIES ON NUCLEAR PLANTS NEAR EARTHQUAKE FAULT LINES

Like I said, I just did this out of my own curiosity.  The one thing I did learn from my quick research is, I am not the first one to ask this question.  Those who wish to accuse me of unnecessarily alarming others, I want to assure you I have not passed these findings on to any clients indicating they have a risk to be concerned about.  I am simply posting links here in the blog for other professionals to take a look at and come to their own conclusions.

I do welcome, however, comments from anyone who wishes to refute the reports, chastise me for passing the links along, or to, heaven forbid, thank us for thinking about looking into this possible threat.

Today’s Disaster – Wild Animals on the Loose!

Okay, here’s a new one – a city in lockdown mode because there are wild animals on the loose roaming the city streets!

I can’t help but chuckle imagining the broadcast message that one would send out to their employees telling them the office is closed due to a city lockdown caused by wild animals.

I really have no more to say about this one, other than I just had to share this story with you.  I will have to challenge myself a little harder to come up with a legitimate blog post – but, you can read the story and adjust your plans accordingly for this risk.

The Blackberry Outage

The current Blackberry outage going on throughout Europe, and now the US, provides an opportunity to discuss two important Business Continuity Planning issues: 

  1. Don’t rely on a single communications device
  2. Ensure you have processes for addressing the backlog

I remember immediately after the events on 9/11 people were touting how well their Blackberries continued to function during the crisis while all other communications tools were failing.  Shortly after, it seems, everyone was running out and buying a Blackberry.  I was not suggesting people not invest in Blackberries, but I was warning people that just because this particular tool was working in this crisis does not mean it will be the one tool working in the next crisis.  One reason the Blackberry worked so well in 2001 was because so few people were using this device, the infrastructure that supported it was not being overburdened during the time of crisis.  Blackberries rely on a different technology and different infrastructure that was not damaged during 9/11 – I was warning anyone who cared to listen (probably no one) that this might not be true during the next crisis.  My point was not that Blackberries won’t always work, but that you should not rely on a single tool or technology for all of your communications channels.  Lo and behold, we now find out that Blackberries are susceptible to network wide outages similar to other communication tools.

In the referenced article, Research in Motion is saying that they have fixed the underlying problem causing the outage but that the backlog of emails and text messages is delaying getting the service fully functional once again.  This is a reminder to make sure that business areas consider the impact of the backlog during times of outage and have procedures in place to address the backlog once their systems are back online.

I have even seen instances when the inability to handle the backlog that would develop was the primary justification for establishing an RTO for some applications.

Procedures for handling the backlog (and, reentering lost transactions where the RPO is not, point of failure) need to be included in each department’s business continuity plan.  For some financial based applications, this may include having to post date transactions to ensure they have the right effective date with them.  For some applications that automatically generate the transaction date and time, this may require some additional programming or rebooting servers with different time stamps to ensure the proper entry date.

For all applications and business processes that are not immediately failed over, there is the potential for a backlog to develop.  How you handle that backlog must be considered in the recovery and continuity plans.

Business Continuity Planning: Vendor Risks

One of the risks that a lot of companies may benefit from looking at a little closer is that of “vendor risks”.  Vendors can be suppliers or outsourced entities that perform a critical service on behalf of our organization.  We need to ensure that our critical vendors know how to respond in the event of our disaster and we need to know that the vendor can continue to provide materials or support in the event of their disaster.

I know many organizations include “Service Level Agreement” (SLA) clauses in vendor contracts, but I suggest that we may want to go further than that and, every now and then, ask to be shown evidence that they could meet those levels of performance at time of disaster.  How many of you audit or review your vendor’s Business Continuity / Disaster Recovery Plans?  How many participate in their vendor’s Business Continuity / Disaster Recovery tests or exercises?

Many organizations try to mitigate or eliminate vendor risk by engaging multiple vendors to provide a similar product or service.  Just be aware that, sometimes, even though you diversify your vendors, you may not have diversified the infrastructure they depend on.  Lessons learned from the events of 9/11 showed proof of the issues that can arise here.  Many companies felt confident that they were using multiple communication vendors only to discover that they all relied on the same underground infrastructure and same “points of presence” (POPs).  One central office failure; one cable conduit compromised and all vendors were out of service – the diversity did not provide the stability they thought they were getting using multiple vendors.  Even though you may think you have eliminated a potential “Single Point of Failure” (SPoF) by using multiple vendors, make sure you do not still have SPoF in the physical infrastructure they rely on.

Another example of this I recently encountered was working with an airport authority in dealing with a potential flooding risk caused by a suspect dam near the airport.  Many of the airlines at the airport had fuel provided by two or more fuel suppliers, but the delivery of the fuel was all through the same single source pipeline that was in the flood zone.  Even though the pipeline itself was underground and, potentially, not susceptible to damage by the flood, the fuel line switching station was above ground and in the flood zone.  The pipeline needed this switching station operable to move the fuel.

I also was once hired to look into the reliability of an off shore outsourced call center.  This facility, located in India, was a state-of-the-art facility in a pretty resilient compound.  The outsourced company felt so secured in their “hardening” of the facility that they did not see the need in investing in contingency operations.  The problem was, however, that the infrastructure that fed power, phone service and other utilities into the compound was very suspect.  Additionally, the employees did not live in the compound and a disaster in the area could easily prevent them from getting to the complex.  My client decided that they needed a contingency should their primary vendor suffer a business interruption event and took the necessary steps to cover this risk.

And, remember to make sure your vendors know what changes in their delivery or performance must be made at time of your disaster.  One simple example – do your mail carries (US Post Office, UPS, FedEx, others) know where to reroute your mail or where to do pickups from when a particular facility is compromised? 

Also make sure if you have vendor personnel on site that you educate them in the evacuation, notification and escalation process.  Are you responsible for accounting for vendor personnel during a disaster, or do you call the vendor and have them account for or alert their employees at time of crisis?  Do not forget those vendors that may not perform a critical service but are on site – such as, cafeteria staff; custodial staff; plant suppliers; landscapers; etc.  Make sure they are notified of an office closure and are included in the process for accounting for who may have been injured or killed in the disaster.

Sometimes it is easy to overlook our vendors in the planning process.  Make sure your program and department managers have adequately accounted for them.

Atmospheric River 1000 (ARk)

Really, it’s not like I just sit around thinking up the next disaster that might occur; or that I spend all day searching the Internet for catastrophic events to scare the stuffing out of people.  But, given what I do for a living, they just kind of have a way of finding me.

About a year ago, someone told me about the potential of a huge rain storm on the West Coast of the United States known as an ARk Storm.  Apparently there is this weather pattern known as an Atmospheric River (the “AR”) that has a catastrophic occurrence about every 1000 years or so (thus the “k”) that scientists are studying and suggesting could occur again.  The last ARk Storm to hit the West Coast happened in the late 1800s, (so, in my book that means we have another 900 years to wait) but meteorologists are just starting to understand the potential impacts another ARk Storm could carry.

Up here in the Pacific Northwest we are very used to the Pineapple Express weather pattern, which, I now understand, is an example of an Atmospheric River – just not the build your Ark kind of event the ARk Storm is supposed to be.

Certainly, throughout the US we have had our share of floods, mudslides and other heavy rain events that have resulted in Emergency Response, Crisis Management and Business Continuity plans being engaged.  Who cares what neat, scientific name we give the events that caused them?  But, I did find some interesting articles about this ARk Storm potential that I thought you might want to check out, including a story that suggests its just a bunch of Internet hype.

And no, I am not suggesting we now need an ARk Storm Response Plan!  I just like to share potential risk information with people who might want to know.  You know, to scare the stuffing out of you!

Business Continuity Planning: Have We Fallen Victims to our own Methodology?

I understand the importance of all the phases of the typical Business Continuity Planning Methodology.  I know the value of and why we conduct Business Impact Analyses (BIA) and Risk Analyses.  I understand the benefits and process for defining Recovery Time Objectives and Recovery Point Objectives.  I appreciate the need for defining Recovery Requirements and know the value of identifying different Recovery Solutions and conducting Cost/Benefit Analysis to evaluate and select the best alternatives.

I get it, really, I do.  I have been following this recipe for years (don’t ask how many) and have made a living at convincing clients they need all of this stuff.  And, I believe that they do … eventually.  I also believe, however, that we sometimes fall victims to our own methodology and sometimes lose sight of what it is our clients need, at this point in time.

I have witnessed myself, senior management teams getting frustrated because teams of consultants had been working for months on “The Analysis Phase” of business continuity planning and all they were wanting to learn was who was going to call them at two in the morning when a disaster occurs.

Sometimes I think we get so caught up in the business continuity planning aspect of things that we forget to first implement a baseline emergency response plan that addresses the crisis management components of the program.  After all, we need crisis management with or without a comprehensive business continuity capability.

Don’t get me wrong – we need to implement the BCP Methodology and all of its bells and whistles.  But I think we sometimes get so caught up in planning the menu, determining the best foods to eat, evaluating the nutrition content, balancing the diet and so on and so forth, while our patient starves to death waiting for some food.

Baby steps.

I think we serve our clients (internal or external) best by first documenting the imperfect programs in place today, even if the strategy is to figure it out at time of disaster.  If we can at least put together a baseline plan that includes a communication process, notification and escalation procedure and crisis management framework that gets the right people together to “figure things out” – we can at least ensure the patient is eating something while we design and implement the perfect meal plan.

Does any of this make sense?  I simply wish to suggest, that we do not blindly follow an academic approach to the planning process without first understanding what the patient needs.  Stop the bleeding before designing the perfect health care program.  To do that, we need to find the bleeding.  Rather than trying to explain the methodology – first ask, “What are you looking for your Business Continuity Program to do for you?”  You might be surprised by the answer.

Disasters, Disasters, Disasters

One of the challenges that Business Continuity and Disaster Recovery Planners have had to face over the years is in dealing with a largely apathetic business community.  Many of the management personnel we try hard to work with just do not buy into the belief that a disaster is likely to occur – or, at least – not during their time in the position, so why invest time and resources to plan for an unlikely event?

In this day and age, that is dangerous thinking.

I have written a few blogs over the past month about real events that have recently impacted the business community – the threats are real; the impacts are meaningful.  Safe Harbor Consulting alone has received numerous calls from companies that have been impacted by these events – even if just impacted by having to prepare for potential losses – realizing the need to update, expand and improve their emergency response and business continuity plans and posture.

It used to be that I would search for disaster related stories on the internet to try to validate the work we do, but now all you have to do is look at the top news stories for the day.

Today, for example, some of the top news stories on Yahoo include:

A Google news search, in addition to similar headlines, includes:

And these are just the top news stories for a typical day.  Each of these events have the potential of causing some sort of business interruption or impacting the workforce in some way for companies in the vicinity of the event.

These stories range from the scary (earthquake) to the sublime (satellite falling to earth), but they all have crisis management, emergency response and potential business continuity concerns.

We can no longer pretend that the threats are not out there.  And, we as professional planners can no longer use the excuse that management just does not appreciate the need for planning – it is our job to make them understand the need for planning!  So, let’s get out there and do our jobs.

I almost hate to see what tomorrow’s headlines will bring!