Tag Archive for recovery requirements

Establishing RTOs

I think there is a common mistake that we, as business continuity planners, make when working with our business partners to determine RTOs for processes and applications that support them.

I think we do a good job in using the findings from our Business Impact Analyses (BIA) to help identify the Most Critical, Critical and Essential business processes (or whatever labels you happen to use) to ensure that these processes are what we recover first, but, I think when we work with these areas to define Recovery Time Objectives (RTO) we do not properly establish the post-disaster performance objectives.  I think that most of us allow our business partners to establish their RTOs based on the assumption that they will be operating at or close to business as usual.

Sure, we instruct them to try to establish the minimum requirements and consider work arounds and the such … but, to achieve what end?  How many of us first ask senior management if there will be any changes to our management objectives following a serious business interruption event?  Will revenue or income targets be adjusted?  How much additional costs and expenses can we incur?  Will response or service targets be adjusted?  Margin targets adjusted?  ROI?  ROE?  Or, any other management metrics adjusted because we are in crisis mode of operations?

Although this goes against my overall philosophy of trying to simplify things, I think it would be beneficial to establish three modes of operation when establishing RTOs with our business partners.

  1. Survival Mode
  2. Sustain Mode
  3. Business as Usual Mode

The goal of Survival Mode operations is simply to keep the company solvent.  Forget trying to be profitable; forget growth targets; forget avoiding all penalties, fines and service interruptions – what, minimally, does the company need to do to not jeopardize the solvency of the firm?

The goal of Sustain Mode operations is to satisfy the commitments we have today with our current customer base.  What do we need to do to keep our current customer base satisfied and meet the regulatory and contractual obligations we already have in place.

And the goal of Business as Usual is … well, just what the words say.

I think if we could get senior management to define the management objectives for each mode of operation and how long the company can operate in each mode, the RTOs we establish will be much more realistic.

I work in many environments testing their RTO capabilities where, when short time-frames are missed, they report this as a failed exercise but, the business areas ultimately say, we could have lived with the delays.  I think our RTOs, in general, are much tighter than they need be if we think about Survival first, then Sustain and then BAU.

I know, I know, I know … for those of you cursing me out; yes, there are some real crucial business processes that legitimately have very short RTOs (or require immediate failover with no downtime), but I think that pool of requirements is much smaller than many of our programs suggest.

So, yes, I think we do a good job focusing on Most Critical job processes, but I don’t think we establish the right mindset in gathering the requirements to support them after a disaster.

I welcome all comments to the contrary or, heavens forbid, in support of this concept.

Business Continuity Planning: Recovery Requirements

Remember the movie, “The Jerk” with Steve Martin?  Great movie.  Anyway, there is a scene in this movie where Navin Johnson (Steve Martin’s character) loses all his money and walks out of the house saying something like, “I don’t need anything.  I need this ashtray.  But, that’s all I need.  I don’t need anything but this ashtray … oh, and this lamp.  Just this ashtray and this lamp.  I need this. …”.  You get the picture, right?

There have been times when working with business managers in identifying recovery requirements that I have felt like I was in the middle of that scene.  This somehow seems to be especially true when working with trading floor operations.  Often when I first sit down with trading floor managers and ask them, “What do you minimally need to conduct business”, they answer, quite simply, “A phone.  You give me a phone and I can conduct my business from anywhere.”

Oh really?

You mean you have the phone numbers for all your clients, brokers and the trading floors?

Well no.  I need those numbers programmed into the phone.

So it’s not just any phone you need?

No – but, if I had a phone with those numbers programmed in, I can do my job.

So you don’t need to know the state of your client positions?

Well sure, I need that.  But if I had a phone, programmed with numbers and a copy of the start of day report, I can do my job.

And you don’t need access to any market data feeds?

Well, not all of them.  If I had a phone, programmed with numbers and a copy of the start of day report and access to Bloomberg, I could do my job.

And you will remember the trades you transacted to call into the back office.

Well no.  I need my blotter system or trade tickets.  If I had a phone …

Aaaaaahhhhhhhhh.

It got so bad that I would walk into these meetings with a roll of quarters in my pockets.  When they said all they needed was a phone, I would take out the roll of quarters and say, “Okay, take these.  There is  a pay phone out on the corner of the street in front of this building – go do your business for the rest of the day and let’s see how that works.”  I know that analogy kind of dates me.  Some of you reading this probably don’t even know what a pay phone is – but it often got my point across.

Once I got their egos to agree that it was more than just their expertise that made them a successful trader, and that they did depend on technology and tools more than they liked to admit, the next challenge was in identifying what was needed under what circumstances.

The difficulty with trading floors and many other business functions these days is the interchangeability of certain tools.  You get into discussions where as long as I have Application A, I can do without B or C; but, if I don’t have A, I need both B and C.  This can get very complicated.  Forget about the complication of also recovering the behind the scene applications and tools that the business managers don’t know about – that complexity is up to the technology teams to figure out.

Getting the right applications recovered in the right timeframes takes coordination between many different departments that share applications and databases.  I know some organizations try to identify “application owners” in the business community, but, this too, can get complicated.  Easiest example is, who owns email?  In a trading floor environment, there are many market data feeds that are shared between different trading desks.  Defining the ultimate owner can be challenging and cause troubles.

Once you identify the toolset (or toolset options) required to support critical operations you can start researching options for getting them operational in the requisite timeframe.

Be prepared, however, for some business managers to answer your question on what they need with another question, “Well, what can I have?” or, “How much will it cost me to have …?”

I don’t think it is too out of line for some managers to take the position that it is not a matter of what I minimally need in place, it is the matter of what I (or the company) is willing to pay to have in place.  If it is a reasonable expense to provide full functionality in a recovery site, why go through the exercise of asking me what I can get by with.  Give me solution options, and I will tell you what I can afford to invest in.  This often results in a push me-pull you working atmosphere that not all continuity planners are ready for, but one in which I think we should be prepared to handle.

Just a few more things for us to think about.  I sure am glad this job is not an easy one, otherwise anybody could do it.