Your Backups Might Not Save You. Here's Why

What Is the Give IT To Me Straight Podcast?

Shane Naugher: Welcome to Give IT To Me Straight, the podcast from Dazzee IT. Each week, we break down technology, cyber security, AI, and business IT into practical insights you can actually use. Now, here's your host, Shane Naugher.

What Are We Covering Today?

Shane Naugher: Hey guys, and welcome back to another edition of Give IT To Me Straight, where we dive into the technical questions and topics that we commonly hear from our clients in the hopes that it saves you some time and effort when it comes to your environment. Today, I want to get into the topic of disaster recovery and what that really means. And more broadly, instead of just disaster recovery, really what we want to get to is business continuity, which goes a step further than just recovering from a disaster.

Does Having Backups Mean You Have Disaster Recovery?

Shane Naugher: When we start talking about the topic of disaster recovery, before we even get to business continuity, the most common thought that comes into play is backups. One of the most common issues we see and have to walk through with clients is just because you have backups running does not mean you have disaster recovery. And along with that, we see a lot of common misconceptions when it comes to how backups work, what's being backed up, and what the right expectations are from those.

Do You Know Where All Your Critical Data Lives?

Shane Naugher: So let's start with a base level understanding of backup. When it comes to backups, most of the time your IT department or your IT provider is going to give you a strategy. And when it comes to the overall data that you have inherent to your organization, one of the most important things is to understand where your critical data is, what's writing to that data, how frequently is it writing to that data, and who is contributing to it. One of the biggest pitfalls that we see organizations fall into when it comes to backups is not understanding where all this data lives. Ultimately, what we find is a lot of times critical data is being missed because it's not being centrally stored. It's being stored on individual computers and we're not capturing every application that we should. When it comes to backups, start from scratch and say, okay, what are the critical applications that we use on a daily basis, and where does it store that data? A lot of times that's going to be in a database. A lot of times it could be in individual files, and a lot of times it may even be in the cloud. You have to get a formal inventory of where your data is, what applications are writing to it, and how frequently does it change. That's important because understanding where that data is tells you where you need to set up your backup jobs.

Are Individual Workstations Being Left Out of Your Backup Strategy?

Shane Naugher: Once you have that in place, you want to look at whether everyone is writing to those same locations. One of the biggest things that we see most organizations miss is they back up their server data, but individuals have important and critical files that they use on their local machines, maybe in their My Documents, maybe in their SharePoint, but it's not discussed in the overall corporate backup strategy. They hear the fact that they have a backup and just assume that it covers all critical data. So you need to look beyond just the centralized server applications, the databases, and the things that are stored on the servers themselves, and really go a level deeper. Are there critical files on individual workstations and folders?

How Can You Centralize Data to Strengthen Your Backups?

Shane Naugher: Once you have identified that, one of the things we recommend is, if you can, push all of the critical data, even if it's being used on a local machine, to the server so that you have a central repository that can be backed up. A lot of people will work with My Documents on their local computer. You can actually map the My Documents folder to redirect to your server, so your end users really have no difference in experience. They're still using the My Documents folder, it's just pushing all of that data to sync to the server itself. So when you do that central server backup, you are incorporating a more robust backup approach and strategy. Now, a lot of times that may not be possible. You may have remote people that are in and out of the office, and getting data to a central repository just may not be feasible. If that is the case, then what you want to look at is a multi-pronged approach when it comes to backups, making sure that you have your server set up with a backup process and strategy, and then the individual workstations themselves on another backup process and strategy.

How Does Backup Frequency Affect What You Can Actually Recover?

Shane Naugher: Collectively, you want to look at how you store that data. So we've identified where our data lives, we've identified what applications are writing to it, we know what the individual files on individual computers are, and we've established a process for that. Now we need to talk about what does that look like in terms of schedule. How often do you back those up? How frequently are you capturing full backups? And what does recovery actually look like? People make the assumption that just because there's a backup, it always has the most current data. The reality is most backup strategies may only back up that data once a day. Which means that if you backed up last night at 2:00 a.m. and you've been writing data all day long and you have to restore to that backup, you in effect lose any of those changes up until 2:00 a.m. that morning. So you have to factor that in. And then that comes into the discussion of, okay, well, why don't we just back up more frequently? And you can, but the trade-offs are you have to schedule more storage, you have to schedule the data transfer, which can impact network performance and individual server and computer performance. So a lot of times you don't want to run those backups during normal business hours.

Shane Naugher: We do see organizations that are highly dependent upon their data, and it's very critical data, that have to go to incremental backups throughout the day. You can do that to where it is not a full and complete backup, but is a backup of the changed data relative to the last full backup. So when we do a full comprehensive backup, we're backing up everything at once. All files, including installation files, all data files, everything at once. And then you can do incremental, which is what's changed since that last full backup.

What Are the Different Types of Backups?

Shane Naugher: When it comes to types of backup, you can do a file level backup or an image level backup. Typically what we see is, on a file level backup, those are going to be more aligned to a workstation or an individual user's files. They're storing a dozen or even up to several hundred files locally that they want to back up. Typically we're going to do that on a file level because we don't necessarily care as much if their individual operating system is backed up, because you can always reinstall it and then use the files to restore. Whereas a server, you want to capture everything. All of the installation files, the operating system, all of the log files. You want to capture that as an intact image so that you have everything that you can restore to. So different types of backups, different storage requirements, different capabilities when it comes to how that makes you more robust in your data recovery.

Where Should You Store Your Backups?

Shane Naugher: Let's say we've gotten all of that ironed out. We know what the schedule is, we know what types of backups, we know how frequently we're going to back up, and whether we're going to do full or just changed data in those backup strategies. The next thing we have to look at is where are we going to store those backups, because there are trade-offs there. Backing up to a local device on your network is most of the time going to give you the quickest restoration. So in the event that you lose that data, restoring it from your local network is going to be faster than trying to download it from a cloud server. In terms of the most rapid response to get the data back, having a local appliance is going to give you that fastest recovery. However, you still have all of your eggs in one basket. If it's sitting in your local environment and a tornado comes through or the building burns down, your backup goes with it. So it's important to realize that is a limitation when it comes to how you're backing that up.

Shane Naugher: The flip side is we can back up to the cloud and put that offsite, and that gives you the most robust recovery so that in the event something happens to your physical location, you do have it offsite. You'll hear a lot of people talk about offsite backups. Now, if you have multiple locations, you can back up between those locations if they're geographically dispersed enough that the likelihood of a tornado hitting both locations is limited. What we typically recommend for our clients is a hybrid approach where we keep an onsite appliance that backs up locally, which gives us that quickest restoration of data, but we're also replicating that data to the cloud. So we've got a local copy and one in the cloud. That way, in the event that we lose the local location completely, we do at least have it in the cloud and it's protected from that perspective. But if we don't have to go that route, we can restore the quickest from the local copy.

What Does True Disaster Recovery Actually Involve?

Shane Naugher: We've been talking about backup, and that is really about getting the data. When it comes to disaster recovery, we really need to go a step further and look at what does it take to go from a failure of the data point to operational again. That goes into strategic planning and making sure that everyone is on the same page, because when we look at the majority of the folks that we start conversations with around this, they have a completely different expectation from reality when it comes to how long it's going to take to get back up and running. Most people assume that if they've got a backup, they can be back up within a matter of hours, which may be the case. But a lot of times when we have to go to a backup, it's because the hardware itself has failed. So you've got to factor in that if the hardware has failed, we've got to get new hardware in place and get it up and running before we can restore the data back to a server.

Shane Naugher: Going back to having a local appliance to back up our data to, a lot of times we'll build these where they're robust enough that they can run a couple of the most critical servers on that appliance in a pinch. We build them that way so that there is that critical coverage, knowing that it may not give you the same performance and it may not give you all of the servers in your environment, but your most critical ones that allow you to work as quickly as possible can be run on that backup appliance itself.

Shane Naugher: In the event that you don't have that robust appliance-based solution and you have to get in new physical server hardware, one of the things we see the most misconception around is the time it's going to take to get that up and running. Even in an emergency when you have a server fail, unless you have an advanced hardware warranty that gives you a four-hour replacement, you're probably talking about days if not weeks to get new server hardware in place so that you can actually begin the restoration process and restore your data to a working piece of hardware.

Shane Naugher: So when you start talking about disaster recovery and looking at what's called recovery time objectives, that's mapping out how long is it going to take us to go from failure to operational status. You've got to factor in if you have to replace hardware. Now you could also look at a cloud server environment where if you fail on your local hardware, can you replicate and spin up a cloud-based server? There is that option, but there's ongoing expense associated with that. Provisioning that, keeping that server in a position that you could restore to and getting that set up. And then you've got to factor in setting up VPN connections, so there is some technical conversation that needs to be had. When it comes to disaster recovery, you've got to get to a point where you're looking at what does it look like in the failure and what are our steps going to be to get it back up and operational. So again, going back to the start of the conversation, disaster recovery, most people assume it's just backups. It really is what are the backups? How are they being backed up? Where are they stored? What does our process look like in the event of a failure? And how do we get to a point that we can manage the expectations? Because if you go into it with the wrong expectations, I can guarantee you you are going to be wildly disappointed in what you thought was your strategy to approach this.

What Is Business Continuity Planning and Why Does It Matter?

Shane Naugher: Which takes us to what we like to get organizations to in true business continuity planning, which is basically going another step forward and actually mapping this out. You've probably heard about an incident response plan. If you've filled out cyber liability insurance applications or even your general liability policies, a lot of times they'll ask, "Do you have an incident response plan?" And that's multipronged. It could be referring to a cyber response or a data loss response. Both of them are going to incorporate a lot of the same components. You need to map out, okay, in the event that our building disappears, what is step one? How are we going to operate? What are the most critical systems? What people are going to need access by when and how are we going to provision that for them. You really need to step through that methodically and actually have a documented process so that when it comes to that actual emergency, you know what to do in that time of excitement, because I can tell you based on what we've seen historically, when those events happen, the logical thinking becomes very, very hard to do and a lot of times mistakes are made. So it's important to map that out ahead of time, have that plan put together, know what the steps are, and then make sure that you document that and make it readily available so that people understand where it's at, how to get to it, and how to use it.

What Should an Incident Response Platform Include?

Shane Naugher: Which takes us to where we ultimately like to get our clients to, and that is leveraging what we call an incident response platform. This is basically a living document that not only tells you what the steps technically are to get your data back up and operational, but it also maps out what are the logical things that need to happen. You need to get your attorney involved when you have one of these issues, especially if it's a cyber incident. If you have cyber liability, who are the contacts and what is the process? One important caveat I'll also throw into the mix here is that a lot of times people assume if it's a cyber event that causes data loss, they need to call their cyber liability carrier first. And depending on your situation, that may or may not be true. I would say that your first call probably needs to be to your individual business attorney that knows your operations first, so that they can help guide you through this, because your insurance carrier's interest is going to be protecting a claim payout. They're going to be looking at how to minimize that payout from a liability perspective. So you want to make sure that you have your attorney looking out for your interest first and helping guide you through that process as well. But the key thing is you have to map this out ahead of time. You have to know who to call, when to call, what the contact methods are. Do you need to notify your clients? Do you need to notify your employees, your vendors? All of that needs to be mapped out in advance. And that's where an incident response platform can actually walk you through that process. What we recommend is, if you were to have a disaster occur, whether that's losing the building or a cyber incident, you go into this cloud-based platform that can be accessed anywhere, you launch that incident response plan, and it walks you through what to do but also documents what happened so that as you get to the legal aspect and the insurance aspect, you have all of your ducks in a row.

Are Your Backups Actually Being Tested and Verified?

Shane Naugher: The last point that I will make when it comes to business continuity planning is a critical component that we see missed in almost every environment we go into with a new client. They may have backups. They may even know what's being backed up and when. However, the majority of the time when we dig into the details in a new client environment, we find that there has not been any ongoing testing and verification of that backed up data. It is super critical that you monitor when backups have been run. One of the things that we know from experience is that threat actors, one of the first things they're going to do when they get inside an organization's network is they're going to disable the backups and potentially even remove any of the backups. They can do this silently a lot of times by just disabling the backup job.

How Do You Know If Your Backups Are Actually Running?

Shane Naugher: If you're only keeping a week's worth of backups, they only have to wait until those backups time out. We see that a lot where the backup job is disabled and there's not an error generated from that backup job saying it didn't run, because the job itself has been cancelled. So there's not anything that brings attention to it, and those backups time out. Therefore, when you try to go back and get to that data, it's simply not there or not usable. So making sure you have a process that alerts you if a backup was supposed to run, it was scheduled to run at 2 a.m., and you didn't get an error on it and you didn't get a success. That should be a cause for concern. Monitoring did it run, was it supposed to run, did it have any errors, or was it successful. All of those need to be mapped out.

Are You Testing Whether Your Backed-Up Data Is Actually Usable?

Shane Naugher: At least quarterly, if not monthly, you should be looking at some type of test restore of the data. We see this all the time where backups run and they get a success saying the data has been moved, even to the cloud, but when you go to try to restore that data, it's actually unusable. So you want to at least do some small test to make sure the data that you're backing up is going to end up in a usable format down the road.

Where Do You Go From Here?

Shane Naugher: I know that's a lot of information when it comes to backups and business continuity. Hopefully this seeds some thoughts for you to at least go back and say, "Do we have all of this in place?" If you don't have the comfort that you have all of this in place and would like to talk more about it, we would be happy to walk you through strategies to use with your existing IT department, your existing provider, or if you want to explore what Dazzee IT could do for you, happy to discuss that as well. Going back to that incident response platform, we're one of the few in the area that has that capability and can help provide that platform for you. It does save you on your cyber liability policies because you can document what's in place and what those steps are. So if you want to talk, certainly let us know. If you just want to ask questions so you can go back to your IT department, certainly let us know that as well. Thanks for tuning in, and I look forward to sharing with you on the next edition of Give IT To Me Straight.