Networking Guide 8 - Fault Tolerance and Disaster Recovery

[mindreader]

Tape Rotation Schedule

Rotating backup tapes is the most practical way to manage a tape backup scheme, since the costs of using a new tape each day are prohibitive. Although tapes are the cheapest form of storage compared with other media, that does not mean that tapes are cheap. A single DLT cartridge can cost $40. If you use a tape for every day of the month, plus one for every month, you will be spending more than $1600 per month, not counting taxes, for just one server. To go back to any day in the year on a single tape would require a tape per day. When you use a tape for every day of the year, the cost is more than $14,000. And the cost can increase even more, because most companies have more then one server and advanced backup programs can remotely back up users’ workstations as well.

The solution is tape rotation. Do not use a different tape each day. Instead, reuse tapes from previous months and weeks. We will look at some simple rotations, such as weekly, along with some rather complicated schemes.

Weekly Rotation

In a weekly rotation, you use a different backup tape or tapes for each day of the week. Weekly rotations are the simplest to understand and set up. You first assign a tape to each weekday and label the tape with the name of the day. You have five tapes, and you overwrite each tape as the day of the week comes again. The furthest you can go back to do a restore is one business week. On Friday, before the backup, you can go back to any day for one week, but no further.


Monthly Rotation

Rotating tapes on a monthly basis allows you to restore data for an entire month. Managing this type of backup scheme is more complicated because you must keep track of many more tapes. A straightforward solution is to assign 31 tapes and do a full backup each day. This becomes unwieldy if a full backup takes many tapes. For example, a thousand-user corporation’s e-mail, file, and print servers can take multiple high-capacity DLT tapes per session.

Most of your restore requests will be reported shortly after the file is accidentally deleted or corrupted. Take your typical user who accidentally deletes his home directory. Using a GUI interface, this is as easy as rightclicking a folder and then left-clicking Delete. The user will immediately call network support and plead for quick rescue. In this case, you only have to go back to the previous day’s tape. To plan for this scenario, have daily backups that go back a week. Supplement this with a weekly backup for an entire month.

In this configuration, you would use no more than nine tapes. You will use one tape for each day of the week, Monday through Thursday (four tapes) and one tape for each Friday of the month (four or five tapes, depending on how many Fridays there are in a month). A maximum of nine tapes will give you daily backups for a week and weekly backups for a month. Label the tapes Monday through Thursday, and Friday Week 1 through Friday Week 5.


Yearly Rotation

You can build a yearly backup on top of the monthly system. You’ll need 12 tapes, one for each month, labeled with the names of the months. Rename the last weekly, full backup of each month to the corresponding month. You go from nine tapes to 21 tapes and gain the capability of going back a year to restore data. Only one day out of each month is available after you go back further than your current month.


Grandfather-Father-Son Rotation

A standard rotation scheme for tapes is the Grandfather-Father-Son (GFS) method. With this method, daily backups are differential, incremental, or full. Full backups are done once a week. The daily backups are known as the Son. The last full backup of the week is known as the Father. Because the daily tapes are reused after a week, they age only five days. The weekly tapes stay around for a month and are reused during the next month. The last full backup of the month is known as the monthly backup, or the Grandfather. The Grandfather tapes become the oldest, and you retain them for a year before reusing them.

Long-Term Configurations

In addition to daily, weekly, and yearly backups, some companies, for archival purposes, do an end-of-year backup, which is then kept offsite in long-term storage. They do this to keep a record of the year’s financial and transactional data so that they can refer to it in case of tax problems. (The IRS may require businesses to keep transactional data for seven years.)

Some companies do two end-of-year backups—one before closing out the fiscal year, and another after closing out. They do this in case they mess up the closing and need to start over. When the closing out is finished, they back up the closed-out system and place the tape in long-term storage.

[mindreader]

Virus Protection

A virus is a program that causes malicious change in your computer and makes copies of itself. Sophisticated viruses encrypt and hide themselves to thwart detection. There are tens of thousands of viruses that your computer can catch. Known viruses are referred to as being “in the wild.” Research laboratories and universities study viruses for commercial and academic purposes. These viruses are known as being “in the zoo,” or not out in the wild. Every month, the number of viruses in the wild increases.

Viruses can be little more than hindrances, or they can shut down an entire corporation. The types vary, but the approach to handling them does not. You need to install virus protection software on all computer equipment. This is similar to vaccinating your entire family, not just the children who are going to summer camp. Workstations, personal computers, servers, and firewalls all must have virus protection, even if they never connect to your network. They can still get viruses from floppy disks or Internet downloads (via modem).

Types of Viruses

Several types of viruses exist, but the two most popular are macro and boot sector. Each type differs slightly in the way it works and how it infects your system. Many viruses attack popular applications such as Microsoft Word, Excel, and PowerPoint, which are easy to use and for which it is easy to create a virus. Because writing a unique virus is considered a challenge to a bored programmer, viruses are becoming more and more complex and harder to eradicate.

Macro Viruses

A macro is a script of commonly enacted commands that are used to automatically perform operations without a user’s intervention. Macro viruses use the Visual Basic macro scripting language to perform malicious or mischievous functions in Microsoft Office products. Macro viruses are among the most harmless (but also the most annoying). Since macros are easy to write, macro viruses are among the most common viruses and are frequently found in Microsoft Word and PowerPoint. They affect the file you are working on. For example, you might be unable to save the file even though the Save function is working, or you might be unable to open a new document— you can only open a template. These viruses will not crash your system, but they are annoying. Cap and Cap A are examples of macro viruses.


Boot Sector Viruses

Boot sector viruses get into the master boot record. This is track one, sector one on your hard disk, and no applications are supposed to reside there. The computer at boot up checks this section to find a pointer for the operating system. If you have a multioperating system boot between Windows 95/98, Windows NT, and Unix, this is where the pointers are stored. A boot sector virus will overwrite the boot sector, thereby making it look as if there is no pointer to your operating system. When you power up the computer, you will see a Missing Operating System or Hard Disk Not Found error message. Monkey B, Stealth, and Stealth Boot are examples of boot sector viruses.

Note These are only a few of the types of viruses out there. For a more complete list, see your antivirus software manufacturer’s website, or go to Symantec’s website at www.symantec.com/.

[mindreader]

Updating Antivirus Components

A typical antivirus program consists of two components:

The definition files
The engine
The definition files list the various viruses, their type and footprints, and specify how to remove the specific virus. More than 100 new viruses are found in the wild each month. An antivirus program would be useless if it did not keep up with all the new viruses. The engine accesses the definition files, or database, runs the virus scans, cleans the files, and notifies the appropriate people and accounts. Eventually viruses become so sophisticated that a new engine and new technology are needed to combat them effectively.

Note Heuristic scanning is a technology that allows an antivirus program to search for a virus even if there is no definition for that specific virus. The engine looks for suspicious activity that might indicate a virus. Be careful if you have this feature turned on. A heuristic scan might detect more than viruses.



For an antivirus program to be effective, you must upgrade, update, and scan in a specific order:

Upgrade the antivirus engine.
Update the definition files.
Create an antivirus emergency boot disk.
Configure and run a full on-demand scan.
Schedule monthly full on-demand scans.
Configure and activate on-access scans.
Update the definition files monthly.
Make a new antivirus emergency boot disk monthly.
Get the latest update when fighting a virus outbreak.
Repeat all steps when you get a new engine.

If you think this is a lot of work, you are right. However, not doing it can be a lot more work and a lot more trouble.

Upgrading an Antivirus Engine

An antivirus engine is the core program that runs the scanning process; virus definitions are keyed to an engine version number. For example, a 3.x engine will not work with 4.x definition files. When the manufacturer releases a new engine, consider both the cost to upgrade and the added benefits.

Warning Before installing new or upgraded software, back up your entire computer system, including all data.


Updating Definition Files

Every week you need to update your list of known viruses—called the virus definition files. You can do this manually or automatically through the manufacturer’s website. You can use a staging server within your company to download and then distribute the updates, or you can set up each computer to download updates.

[mindreader]

Scanning for Viruses

An antivirus scan is the process in which an antivirus program examines the computer suspected of having a virus and eradicates any viruses it finds. There are two types of antivirus scans:

• On-demand
• On-access

An on-demand scan searches a file, a directory, a drive, or an entire computer. An on-access scan checks only the files you are currently accessing. To maximize protection, you should use a combination of both types.

On-Demand Scans

An on-demand scan is a virus scan initiated by either a network administrator or a user. You can manually or automatically initiate an on-demand scan. Typically, you’d schedule a monthly on-demand scan, but you’ll also want to do an on-demand scan in the following situations:

• After you first install the antivirus software
• When you upgrade the antivirus software engine
• When you suspect a virus outbreak
  

Note Before you initiate an on-demand scan, be sure that you have the latest virus definitions.


When you encounter a virus, scan all potentially affected hard disks and any floppy disks that could be suspicious. Establish a cleaning station, and quarantine the infected area. The support staff will have a difficult time if a user continues to use the computer while it is infected. Ask all users in the infected area to stop using their computers. Suggest a short break. If it is lunchtime, all the better. Have one person remove all floppies from all disk drives. Perform a scan and clean at the cleaning station. For computers that are operational, update their virus definitions. For computers that are not operational or are operational but infected, boot to an antivirus emergency boot disk. Run a full scan and clean the entire system on all computers in the office space. With luck, you will be done before your users return from lunch.

On-Access Scans

An on-access scan runs in the background when you open a file or use a program. For example, an on-access scan can run when you do any of the following:

• Insert a floppy disk
  
• Download a file with FTP
  
• Receive e-mail messages and attachments
  
• View a web page
  

The scan slows the processing speed of other programs, but it is worth the inconvenience.

A relatively new form of malicious attack makes its way to your computer through ActiveX and Java programs (applets). These are miniature programs that run on a web server or that you download to your local machine. Most ActiveX and Java applets are safe, but some contain viruses or snoop programs. The snoop programs allow a hacker to look at everything on your hard drive from a remote location without your knowing. Be sure that you properly configure your on-access component of antivirus software to check and clean for all these types of attacks.

Warning Many programs will not install unless you disable the on-access portion of your antivirus software. This is dangerous if the program has a virus. Your safest bet is to do an on-demand scan of the software before installation. Disable on-access scanning during installation, and then reactivate it when the installation is complete.


Emergency Scans

In an emergency scan, only the operating system and the antivirus program are running. An emergency scan is called for after a virus has invaded your system and taken control of a machine. In this situation, insert your antivirus emergency boot disk and boot the infected computer from it. Then scan and clean the entire computer.

Tip If you don’t have your boot disk, go to another computer and create one.

[mindreader]

Software Patches

Patches, fixes, service packs, and updates are all the same thing—free software revisions. These are intermediary solutions until a new version of the product is released. A patch may solve a particular problem, as does a security patch, or change the way your system works, as does an update. You can apply a so-called hot patch without rebooting your computer; in other cases, applying a patch requires that the server go down.

Is It Necessary?

Because patches are designed to fix problems, it would seem that you would want to download the most current patches and apply them immediately. That is not always the best thing to do. Patches can sometimes cause problems with existing, older software. Different philosophies exist regarding the application of the newest patches. The first philosophy is to keep your systems only as up-to-date as necessary to keep them running. This is the “if it ain’t broke, don’t fix it” approach. After all, the point of a patch is to fix your software. Why fix it if it isn’t broken? The other philosophy is to keep the software as up-to-date as possible because of the additional features that a patch will sometimes provide.

You must choose the approach that is best for your situation. If you have little time to devote to chasing down and fixing problems, go with the first philosophy. If you always need the latest and greatest features, even at the expense of stability, go with the second.


Where to Get Patches

Patches are available from several locations:


The manufacturer’s website

The manufacturer’s CD or DVD

The manufacturer’s support subscriptions on CD or DVD

The manufacturer’s bulletin (less frequently an option)

You’ll notice in every case that the source of the patch, regardless of the medium being used to distribute it, is the manufacturer. You cannot be sure that patches available through online magazines, other companies, and shareware websites are safe. Also, patches for the operating system are sometimes included when you purchase a new computer.

How to Apply Patches

Just as you always need to plan for an upgrade, you need to plan for a patch. Never blindly install patches (or any other new software) without examining the potential impact on the network. Although patches are designed to fix known problems, they may create new ones. It is best to try patches on a test network or system before installing them on all systems on the network.

Follow these steps to apply a patch:
Research the enhancements and changes that the patch provides. Go to the manufacturer’s website, or take a look at the official documentation.
Download the patch and related documentation to an isolated test network (or computer if you don’t have an entire test network).
Decompress any documentation files and read them. (Yes, the manual is something you read before installation, not after things crash.)
Note the changes, and define a way to test the new features.
Install the patch on a test workstation/server.
Select the installation method that allows you to save previous configurations so that you can uninstall if necessary.
Record any options and your selections, such as retaining or replacing drivers.
Reboot the computer.
If the operating system does not load or work properly, start over with a clean test machine. Select to keep your original drivers. (NIC drivers are commonly updated and may not work.)
Try out the new features. Test all patches to see if they work as advertised.
Run the test workstation/server for two weeks. Reboot it and try different tasks during this time.
If all goes well, do a limited rollout of the update to your support staff’s personal computers and applicable servers, and have them test the patch.
After the IS support staff determines the product is safe, do a limited rollout to some users’ workstations and applicable servers.
Roll out the patch to all production servers and all workstations via an automated procedure.
Ensure proper revision control. Make sure that all equipment has the same approved patch.

Note Remember that these are general steps. Refer to the documentation that comes with the patch (most likely a README.TXT file) for specific instructions on installing a specific patch.


You can see that this process can take a long time, even with multiple test machines and people helping you. The process can be speeded up a little, but do not skip any steps. If at any point you cannot get a system to work, even with changing the install options, stop the installation and refer to the support documentation for the patch to see if you are doing something wrong. Do not roll out a patch until it has been proven stable in all test environments.

Warning If you use your operating system or application CDs to make changes to the operating system or an application after applying the patch, you may overwrite the updates made by the patch. You will need to reapply the patch after accessing these files.

[mindreader]

Summary

In this guide, you learned about fault tolerance and disaster recovery. Fault tolerance is the ability of a system to resist failures and faults and to recover from them by itself. Disaster recovery is the ability of a system to recover from some kind of disaster where data is lost. Disaster recovery methods are used to replace data that has been lost due to some unforeseen circumstance.

You learned that the most critical component of a fault tolerance plan is the amount and type of disk fault tolerance implemented. Types of disk fault tolerance include RAID in its various forms. Additionally, you learned about the different methods of backup and how they apply to disaster recovery.
You also learned how important virus protection is to a network, and that if antivirus measures are in place, fault tolerance and disaster recovery are not needed. Viruses are small programs that can interrupt the normal function of a computer. Viruses can spread on a network like wildfire and must be eliminated before they have a chance to spread and cause damage.

Finally, you learned that, in order to keep systems running with the least amount of problems, a software patch must occasionally be applied. A software patch is a piece of software that temporarily fixes small problems within an existing program until the next major release of the software.