Aaron Gasperi is a Director in Velocity’s Innovation group. He writes about consumer insights, product development and the practice of innovation at Velocity.

On a Thursday evening in early 2006, I got a strange call from my father. He could hardly contain himself. “Aaron, you’ll never believe what just happened,” he said.  

I was returning home from work, so I was juggling a package, my phone, and my briefcase. 

“Because of you, a huge number of people will get paid tomorrow.”  

I had learned at an early age not to be too surprised at anything he said: my father worked on the largest of the large; stock exchanges, international banks, jet fighters. He was semi-retired and was consulting for a payroll processor, but I had no idea what he was talking about.  

“You know that USB drive you sent me for my birthday?” I did. A few months prior while desperately searching for a gift for the man that had everything technology, I learned he didn’t have a USB drive. I had sprung for best drive I could find and afford. 

“I’m consulting at this company because their systems have been failing.” I wasn’t surprised, this wasn’t necessarily anything new. “Can you believe they don’t run backups?”

I dropped my phone.

A company responsible for a substantial portion of the country’s payroll wasn’t running backups. 

He went on to say, “I backed up my team’s work from the week on the USB drive. Even the file servers have been unreliable. On a whim I also grabbed the code that was running in memory.”
Not quite understanding the gravity, I asked, “How is that even possible?” He said, “It’s old code, son. It’s not big, and they don’t store it anywhere outside of those machines.”

He went on, “I got a panicked call from the CIO that payroll processing crashed. I was driving home, so I turned around and drove as fast as I could right back to the company with that USB drive.  Once I got there, I re-loaded everything, and now it’s running and millions of people will get paid tomorrow.”  

I thought: Just how long can a company go without investing in technology before it’s being negligent?

I’m sure today their investors require insurance and accounting audits, including demonstration of a data backup plan which they surely have now. But demands on IT were different several years back. CIOs were measured on how well IT initiatives stayed within budget, rather than the transformative value of their efforts for growing the business or reducing risk, as is the case today; nor was technology as pervasive in everyday business as it is now.  

Look no further than the latest industry analyst meeting or meeting with a CIO: business demands and expectations have increased on a formerly technology-only role. But why are the CFO and CEO allowed to plead technology ignorance? 

I have no doubt that the CIO would have been fired if that many people didn’t get paid. But it was much worse than that. The payroll processor’s customers and their employees would have been impacted as well. 

Just as it is the CIO’s job to understand business strategy and business needs, it needs to be the CEO and CFO’s job to understand technology funding and the implications of underfunding mission critical tasks. 

Technology is a part of business. At the executive level, not understanding is no longer acceptable.

While most situations are less dramatic than this one, consider the financial implications of not pushing for backups and resiliency:

  1. The impact on stock price if a major failure stops business. (if you work for a public company)

  2. Long term erosion of customer trust in your product or service.

  3. Failure could have career implications for you.

So? What to do? Talk to an expert

I called Andy, a fellow Velocity colleague who talks with many companies about disaster recovery (DR), and asked him about some of the pitfalls in today’s approach to DR. He gave me some places to look:

  1. Everybody says they have DR and backups nailed. Ask, “What, exactly, does this mean?” In most DR plans, time to return to operations varies system by system (RPO/RTO). One system might be up and running in an hour, while others might be a week or longer!

  2. Have you tried to restore a system? Many companies do data backups but you don’t know for sure it works until you test and flip the switch. Velocity DR services manager, Jim Knight, has published a webcast that describes some of the pitfalls and one company’s struggles to recover on their own after a disaster.

  3. From a business perspective, how fast SHOULD your systems come back online? These days most companies have completed a DR plan, but few know the details of the plan or have updated it to meet today’s needs. DR has evolved substantially since a plan was initially put in place, and often, the time before operations resume (RTO) can be substantially improved if part of the plan.

Over time, Wall Street has become less tolerant of technology problems. Failure is no longer a tenable option for the majority of companies; and, one of the reasons why companies such as Carolina Bio engage partners like Velocity to help them mitigate technology risk. Besides, you can’t always rely on a USB drive to save the day.