Are your backup systems secure?

This post first appeared at InfoSec Institute 6/2011.

All seemed well with backup operations at my company, until I got a visit from an operations center engineer.  The lock already hanging open, he was holding one of the “secure” transports that our off-site tape storage vendor uses to move backup tapes.  But this time, the tapes inside were not ours—someone else’s data was in our hands.  I couldn’t help but think: how many times have our tapes been sent into our competitors’ hands? Did they send them back immediately as I did—or did they land in less scrupulous places? In this article, I’m going to show you how systemic thinking helps identify security vulnerabilities like this one in your company’s data backup systems, particularly related to issues around backup tapes.

You will learn:

• Factors to consider before you move to a tapeless data backup system
• Encryption challenges and opportunities-related to keeping your off-site data secure
• Approaches for backup tape reduction and legacy storage technology elimination

Backup and documentation are together the most-often neglected facets of IT operational management. Why? Backups usually don’t contribute directly to revenue generation. But this is a fatal mistake: ignoring backup and documentation can cost a corporation millions of dollars, even in typical litigation or disaster recovery scenarios.

By addressing backup challenges systemically in the context of the entire organization, you will see that running effective, efficient and secure backup operations is an opportunity to effectively contribute to the success of a company, even beyond protecting from financial risk.  This perspective will help you to:

• coherently manage data security
• control data and data storage growth
• reduce data storage costs
• ensure business continuity
• recover from disasters
• provide effective reporting and data visualization of systems
• automate processes to reduce data exposure, fine-tuning backup coverage and identifying security vulnerabilities

Simply by reading this article, you’re already ahead of the game.  Most people aren’t aware of the organizational implications of an unsecured backup system – typically, it is thought of as an “IT problem.”  But it’s never that simple and needs to be a shared responsibility.  Recognizing the impact to the organization as a whole is Step One – now let’s dig into the nitty gritty of putting your back-up systems on solid, secure ground.  By viewing IT as an intricate part of a system rather than a separate discipline and universe, you have begun to view things from a whole-systems perspective, in which all components are part and parcel of one interconnected whole.

Back-Ups and Documentation: An Overview

Organizations – whether public, private, academic, healthcare or otherwise – run on data.  Without that data, the wheels stop turning: cash registers can’t ring up customers, trading floors can’t trade, and hospitals can’t keep patients alive.  Serious stuff.  Back-up systems ensure that when something goes wrong, there’s another way to bring the data where it’s needed to keep things moving, in a timely manner.

How do they do it?  Simply put, backup systems make copies of your data and place that data on completely different storage media.  Unauthorized access to those systems and media can potentially destroy a company in days or even hours.  Backup systems come in many different shapes and forms from the different backup software options available to the storage media the data is stored on to the methods of copying and transferring the data itself.  Let’s take a look at the most common forms of data storage found in most large scale enterprise computing environments, to see which might be the best fit for your organization.

Tape v. Disk for Backup — Factors to consider before you move to a tapeless data backup system 

There are those who say tape is going the way of the dinosaurs.  You might instinctively think that by removing tape backup from the equation you eliminate a security risk of lost tapes.  In fact, it never fails that when I’m presenting at a conference, someone asks ‘why don’t you just get rid of tapes altogether and move everything to disk or to the cloud?’  The short answer is, it’s rarely a good idea to keep everything on-line with nothing backed up to removable storage.  In fact, you dramatically increase your risk (of going out of business, of loss of life, of operational delays, or of core service delivery) by exclusively keeping your data on-line.  From a whole systems perspective, multi-tiered redundancy of data storage systems is an important principal of best-practice architecture for disaster recovery and business continuity plans.  The bottom line: don’t put all your eggs in the on-line basket.  Keep copies on tape, so that when one fails, you can turn to the other.  Some of the most forward-thinking and advanced companies out there rely on tape as a component of their back up strategy.  Need an example?  Read on.

February 2011: Tens of thousands of Google email users panicked when they discovered that all of their emails and contacts had disappeared.  The tone of panic in social media was over-the-top.  Although Google maintains multiple copies of data in multiple data centers, disasters still happen.  Google had released a software update to their system and an unforeseen software bug resulted in a cascading failure where several copies of the data was effected and ultimately was lost.  But, the news that saved Google’s reputation was that this monolith of the tech world takes a multi-tiered approach to data protection and in addition to replicating their data many times in many different data centers, they also backup their data to tape and lots of it. Google is the largest consumer of tape in the world today, using some 50,000 LTO (Linear Tape-Open) tape cartridges every quarter. Google was ultimately able to recover all the lost data from backup tapes for all the 40,000 GMail users whose data had been inadvertently lost.

Need more? Here are some quick facts about tape-based data back-up:

1. In many cases, tape is much faster than disk for backup and restoration of data
2. Tape has a much lower cost per gigabyte
3. Tape has a much higher density of data per tape v. per disk drive
4. Tape uses immensely less power and cooling than spinning disk
5. Tape isn’t subject to cascading failures
6. Tape has a far greater life span than disk
7. Tape libraries have infinite storage capacity without taking up any more data center space

One last fun fact: there isn’t any network on earth that provides faster data transfer rates than a U-Haul full of tapes at highway speeds.

Truth of the matter is, data protection is never simple, and you will most likely choose a hybrid architecture of both tape and online back-up solutions.  Every business has different requirements and there are no optimal one-size-fits-all solutions.  You need to employ the right people and do the proper analysis that will align holistically-designed solutions with your business requirements.

Encrypting your data — Encryption challenges and opportunities-related to keeping your off-site data secure

OK, it’s backed up and protected.  Keep it that way!

Now that I’ve suggested that you’ll most likely use tape somewhere in your data protection strategy, let’s move on to ensuring that the data remains protected over time.

It would seem obvious that companies should encrypt their backup tapes, but the fact of the matter is, most don’t.  Whether you use tapes for all your backups or just some of your backups, mismanagement of tape media can be the largest security hole in your organization.  If you do nothing else after reading this article, look up your organization’s encryption strategy—and make sure there is one.  Now let’s identify what approaches to encryption are out there, and what might be a good fit for you.

There are four general approaches to encryption for backup data:

1. Software encryption is a system in which the backup application does the encryption on the data as it’s backed up.
   • Advantages: Encryption and key management is contained within the backup system itself.
   • Disadvantages: Backups may slow down depending on the resources available, there can be heavy processor load on the systems doing the encryption, and this option will not compress or de-duplicate data.
2. Appliance based encryption is a system in which in-band encryption takes place over the wire by a dedicated appliance.  This can take place via a network switch, or in the I/O blades of a backup library.
   • Advantages: Faster encryption and data transfer capability is delivered without increased load on the servers, and often the data can be compressed prior to encryption in-line resulting in fewer tapes.
   • Disadvantages: Key management is a separate system and may be difficult to maintain over many years.  Many of the companies that created these products are either out of business or have EOL (end of life) the product that spells out that there is no continued support if you run into problems.
3. Drive-based encryption is a system in which newer tape drives now come with the capability to encrypt data as it is written
   • Advantages: Drives compress the data before encryption and this system tends to be faster without the drag on production system resources.
   • Disadvantage: Key management may become an issue for long term storage.
4. Encrypt-Always is a system in which the data on disk (in its native, active form) is always encrypted.
   • Advantage: You don’t need to worry about encryption from a backup point of view, so long as data is not decrypted on its way to tape.
   • Disadvantage: The data will not compress or de-duplicate to any kind of storage. Also, system security can be difficult to manage (key management), data can become difficult to access and utilize.  It is also noteworthy that this system may slow down overall performance, and the industry is still young for this technology.

Why people aren’t encrypting their backups.

With these options available to encrypt your data, why don’t more people do it? The short answer is, encrypting is cumbersome to execute and manage over time.  The only way encryption tends to work is with highly disciplined IT teams.  Key management represents the part of the process where the risk and vulnerability to your backup systems will occur: if keys are lost in personnel or technology transitions and there are not back-up systems in place to recover the keys, significant data loss can result.  While it would be great to have all 12PB (petabytes) of your backup data that is residing on some 130,000 backup tapes encrypted, the downside is that if you lose the encryption key(s), you would then have 130,000 tapes you could never read from and you would most likely be looking for a new job.  Fortunately, regardless of the method you choose, a solution to the problem of key management also solves a few other problems when dealing with long term data storage.

More long-term storage challenges

Yes, there’s more.  With state and federal regulations such as the SEC, NASD, NYSE, HIPAA, Sarbanes-Oxley Act, Gramm-Leach-Bliley, and FRCP, most companies are required to keep data for many years.  If you are one of these companies and need to keep data for three, seven, twelve or twenty or more years you are going to have other challenges when it comes to managing the systems and restoring the data.

During the life of any given company, a lot of things happen when there are changes in management, changes in technology and corporate changes such as mergers and acquisitions.  Odds are your company will experience one or more, if not all, of the following:

• You will switch/add backup media types within the same family (e.g. LTO-1 ⟶ LTO-2 ⟶  LTO-3, etc.).
• You will switch/add backup media types across media formats (e.g. DLT7000 ⟶ AIT-3 ⟶ LTO-5).
• You will upgrade your backup software from version to version, each version coming with its own unique upgrade capacity and capability and sometimes, changing the way it backs up or writes data.
• You will switch backup software vendors each of which have their own way of backing up data and will usually not be able to read the backups of another vendor (e.g. NetBackup ⟶ NetWorker ⟶ TSM , etc.).

The end result of these scenarios is the inability to restore vital business data andincreased cost to the business either from fines, lawsuits, or operational redundancies of maintaining extra tape drives, tape libraries, backup servers, networks, staff, etc.  Additionally, companies lose the expertise of running legacy backup systems no one has seen in years and the hardware falls out of support.  Replacement parts become scarce or impossible to acquire.

Gain control over your tapes — Approaches for backup tape reduction and legacy storage technology elimination

Historically, tape is something that needs to stick around for a long time–but times change.  The good news is, everyone faces change management challenges, so there are many technologies and methodologies available for arming you for those inevitable transitions.

Here are a few ways you can manage your backup systems, reduce the number of backup tapes and save yourself from data loss:

Data Lifecycle Management — Viewing backup data/systems as a holistic system from “birth-to-death” is essential to planning an effective long-term solution.

A solution to the problem of legacy systems and data storage also solves problems associated with encryption key management.  While it’s fairly simple to do and set up the policies, it does, however, take a disciplined team to employ the practice and should be automated through and through whenever possible.  One way of viewing a data lifecycle management process is illustrated in the following media migration feedback loop.

Employing a process such as the media migration feedback loop ensures the following:

• All data is on media that is no more than three years old.
• All data is on a higher density and current media technology.
• This approach reduces your off-site storage costs by reducing the slot count and number of tapes while using higher density media.
• This approach increases your ability to backup and restore data faster with newer tape technology.
• You ensure you have the latest encryption and key management system reducing the likelihood of lost keys or defunct key management systems.
• This cycle-based system tests and verifies data integrity concurrently.

Control data retention at the application layer.

Many organizations think that they have to keep their backup tapes for compliance reasons.  This isn’t necessarily so.  Barring requirements for a large data archive system, a much more strategic method towards data retention can be approached at the application layer.  Let me paint a picture of a typical private sector scenario:

Let’s say your State requires companies to maintain personnel records for seven years past their last date of employment.  Does this mean that you need to backup your HR database and keep the tapes (storage layer) for seven years each time you run a backup?  Or does it mean that the HR database itself (application layer) needs to retain a record of each employee for seven years past their departure date?  If the retention of employee records is managed inside the HR database itself for the required period of time, then you can continue to backup your HR database and keep the tapes for a shorter period of time, say a year or less.

Cost-Savings Case Scenario

Here’s a scenario in which costs can be reduced from approximately $30,000 to $1,500 – and that is for only one small system, in one department:

• Controlling retention at the storage layer: The weekly backup for the HR database is 200GB and you keep it off-site for seven years.  This will result in more than 364 tapes at a cost of $30,000
  in tapes after seven years not including the tape storage costs.
• Controlling retention at the application layer:  Give the data retention control to the HR database itself and keep the backup tapes for only six months, then you’d have roughly twenty tapes off-site at a time at a cost closer to
  $1,500.

Let’s take it one step further, reducing costs across the corporation:

Take these numbers and apply it across a typical mid-sized company for all databases, file servers and systems and the cost savings easily turn into millions of dollars.

For example:

Cost-Reduction Area/Activity	Savings
Savings in the HR database	$28,500
Number of similar major databases	x      20
Subtotal of savings	$570,000
Deeper dive into a typical mid-sized company:	> $75,000
Number of data-intensive systems:	x      50
Savings	$3,750,000

You see that by moving the mechanism at which you control data retention reduces the number of tapes, or disk space in the occurrence of a VTL (virtual tape library).  Additionally, moving data retention controls from the storage layer to the application layer provides reductions not only in the amount of storage media you will require, but also in the cost of supporting resources (i.e. people, drives, data center space) for media migration and management. And, as always when dealing with data management and retention of regulated data, consult with your corporate attorneys when putting together your data lifecycle management plan to ensure you remain compliant.

Conclusion

Whether you are the CSO, CIO or storage administrator for your company, systemic understanding of your information systems is essential for ensuring the security and integrity of your corporate data.  When approaching the vectors of security exposure to your backup data systems, don’t be too quick to follow the blind rantings of madmen that represent competing vendor solutions that are only interested in selling you hardware and software by throwing technology at the problem.  Seek the expertise of information and technology strategists that understand the systemic nature of your systems, and more importantly, your business.

As we’ve seen, tape backup systems continue to be used by some of the most forward-thinking and advanced technology companies [like Google!] and have been recognized as the last line of defense against data loss by the best in this business.  A wholistic approach to a coherent tape strategy also allows you to introduce a well-designed encryption system into your data protection strategy while ensuring the ability to manage the whole system (i.e. encryption keys, key management, media formats, backup software versions and applications, etc.)   This type of whole-system strategy will provide you with the ability to reduce the number of tapes you have in order to lower your cost of operations and ensure that you retain the ability to manage and recover mission critical data by employing a media migration strategy.  It’s the same sort of thing we’ve been doing for years with our NAS and SAN storage, why shouldn’t you do it with your backup tape storage too?

Action Steps – Do This Now:

• Find and read your companies policies and procedures regarding:
  • data protection (backup requirements)
  • data retention (where is retention controlled, storage or application layer?)
  • encryption of removable media (backup tapes)
  • Do all of the above policies and procedures exist?  When were they last updated?
• Check your data back-up systems.  Is the data on the backup tapes currently encrypted?  If so, has it been verified?  If not, why?
• How many backup tapes do you have and how many different formats are there?
• Make data protection a shared responsibility between IT, legal, information security and the records management and compliance departments with regular meetings, quarterly or semi-annually, regarding the requirements around data protection.
• Begin to apply a whole-system lens to all systems: how will changes to one part of the system impact other departments, other stakeholders to the data and the ability to deliver your companies products and services to your customers?
• Have an initial, high level external audit conducted by a professional with a whole systems perspective.

© 2011 Brian J. Greenberg