Domain System State Backups

When it comes to backing up the controllers (DCs) in an Active Directory (AD) forest, each DC is a backup of the other DCs within any forest domain. So long as just one DC is operational for a domain, the domain is in service. New domain controllers can be added and will quickly synchronize with the others. If one DC fails, the others can take its place without interuption. To ensure restorability an enterprise administrator need only back up the "System State" of one DC in each domain of the forest. Taking a system state backup of one DC in each domain will allow the administrator to recreate the entire domain (including user accounts and passwords) in the event all domain controllers happened to become damaged, disabled or otherwise corrupted.

NOTE: This analysis assumes that all DCs in an AD forest are used simply as DCs, providing only services which are consistent with DCs. Any add-on third party services such as SSH or any additional applications are not included in a System State Backup.

What Is A System State Backup (SSB)

The system state is not a complete server backup but a smaller subset representing the server's operational state at the time of backup. This includes the following list of items assuming they are installed on the server in question.

• System Registry
• COM + Database
• Certificate Services
• Active Directory
• SYSVOL
• IIS Metabase

You can restore system state to the same server, or another server with identical hardware.

Install Windows Backup Server Feature Onto Domain Controllers

Using the Server Manager utility (Administrative Tools -> Control Panel) install the "Windows Backup Server" feature. This is the native Microsoft software which will perform the backup process. Remember, this is not a server roll which can also be installed using the same wizard.

Logs For Windows Backup Server Activity

Like other Windows features, the Windows Backup Server feature writes events in the server's Application event log to record begining and ending of backups as well as errors and failures. However, other log files are created for each run of the Windows Backup Server software. These are text files recorded in the C:\Windows\Logs\WindowsServerBackup\ folder. It is important to examine both log sources when debugging backup issues as both sources record different kinds of information.

Create Network Share For Storage Of Backups

For the time being we will be maintaining the SSBs for each of our forest domains on a special network share provided by our core environment SAMBA server smb-files.cs.uwaterloo.ca. This will be neseccary for two reasons.

• We wish the backups to be stored on a CSCF managed system which is separate from the source servers both logically (all our domain controllers are virtual) and physically.
• The account on the domain controllers which will drive the backup process much be recognizable (trusted) by the network server maintaining the backups.
  As will be outlined below, only one set of credentials may be suplied to the Windows Server Backup feature for scheduling or initiating a backup.

To this end, the following share configuration has been added the the smb.conf file on the core SAMBA server.

[adbackups]
   writable = yes
   printable = no
   csc policy = disable
   path = /usr/home/adbackups
   browsable = no
   read only = no
;
   ea support = yes
   strict allocate = yes
;
   create mode = 0600
   directory mode = 0700
   guest ok = no
   valid users = CS-GENERAL\ctucker, ctucker
   preserve case = yes
   case sensitive = no
   hide dot files = yes
   hosts allow = 10.0.0.0/255.0.0.0 

NOTES:

• The share is not browsable so it will not appear for normal client share list queries.
• The "hosts allow" restriction will ensure that connections are only permitted from SCS's private VLANs which is where all CSCF forest domain controllers are located.
• Currently my personal CS-GENERAL account (ctucker) has read/write access to this share and no one else. This will have to be changed to a requisite service account at some point in the near future.
• The home directory is on the local SAMBA server and not on any of the shared mounted from the core region nettap.
• It is also recommend to activate "strict allocate" and "ea support" for ext3 extended attributes when it comes to transferring an NT files system backup to a UNIX files system.
  http://serverfault.com/questions/172349/wbadmin-system-state-backup-fails

Assign User Account To Drive Backup Process

It is always recommended that scheduled or daemon tasks be driven by service accounts posessing only sufficient authority to perform the prescribed task. To this end the user account CSCF\cscf_server_backups has been created in the CSCF forest. It is a member of the Builtin domain group Backup Operators (in each forest domain) which is authorized to perform backups on all domain servers including domain controllers. This account's credentials will be supplied to the Windows Server Backup console when configuring the scheduled backups of the domain controllers.

On its own CSCF\cscf_server_backups will not be authorized to access the above network share \\smb-files.cs.uwaterloo.ca\adbackups to store backups. This can be corrected by mapping the account CSCF\cscf_server_backups to a SAMBA server account with authorization to read and write on this share. In the [global] section of a SAMBA server's smb.conf file the option username map can point to the location of a text file where Active Directory accounts can be mapped to accounts known to the local SAMBA server. In this case the following mapping was created which effective gives CSCF\cscf_server_backups read/write access to the adbackups network share.

...../username_map.txt:
ctucker = CSCF\cscf_server_backups

Disable Symantec Application and Device Control (ADC)

This workaround may be required if the domain controller utilizes Symantec Endpoint Protection version 12. System State backups have been known to fail with the following errors.

• Windows Server Backup console reports the following:
  "The process cannot access the file because it is being used by another process"
• Event 517 in the Application logs:
  The backup operation that started at 'time_stamp' has failed with following error code '0x80070020' (The process cannot access the file because it is being used by another process.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
• In C:\Windows\Logs\WindowsServerBackup\error_file_for_specific_backup_time:
  Error in deletion of [C:\System Volume Information\EfaData\SYMEFA.DB] while pruning the target VHD: Error [0x80070020] The process cannot access the file because it is being used by another process.

According to Symantec support, this problem is caused by the Application and Device Control (ADC) module installed with Symantec Endpoint Protection. Currently, uninstalling this module is the best way to prevent the above error from occuring. Symantec is aware of this bug and hope to have a fix for a future update of Endpoint Protection. To uninstall the ADC perform the following steps.

• From Start or Start Button select Control Panel
• Select Programs and Features [Uninstall Program on Vista, Win 7, and 2008 operating systems or Add/Remove Programs for older systems.
• Right click on Symantec Endpoint Protection
• Select Change: The Symantec installation wizard will open
• Select Next
• Select Modify and then Next
• Click on the Plus + sign next to Proactive Threat Protection
• Click on the arrow icon next to Application and Device Control
• From the drop down menu select Entire feature will be unavailable
• Select Next and follow through with the wizard.
• Reboot the System

Make Registry Correction For LSI_SAS System Control

This fix may be required if the domain controller resides as a guest system on a VM Ware server. System State backups have been known to fail with the following errors.

• Event 517 in the Application logs:
  "The backup operation that started at 'time_stamp' has failed with following error code '0x80780049' (None of the items included in backup were backed up.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved."
• In C:\Windows\Logs\WindowsServerBackup\error_file_for_specific_backup_time
  "Error in backup of C:\windows\\systemroot\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect."

The above mentioned string C:\windows\\systemroot\ (note the double \\) does not exist as a real Windows system path. If the backup service was looking for critical information there then it would not have found it. Hence explaining the rather cryptic error in the Application Event Logs.

Many blogs point to an error in the HKLM\SYSTEM\ControlSet\Services\LSI_SAS registry key with the value for ImagePath being incorrect. Instead of systemroot\system32\drivers\lsi_sas.sys it should read as system32\drivers\lsi_sas.sys. Making this correction should prevent this specific backup failure.

Schedule Daily System State Backup

Open the Windows Server Backup console (Constrol Panel -> Administrators Tools) S

• Select Local Backup in the left hand panel
• In the Actions menu select Backup Schedule...
• The Backup Schedule Wizard appears
  • Select Create/Modify backup and click on Next
  • Select Custom configuration and click on Next
  • Click on Add Items - select System state in the selecter followed by OK - and click on Next
  • Select Once a day and select a time - normaly in the early morning - and click on Next
  • Select Backup to a shared network folder and click on Next
  • Set location to be \\smb-files.cs.uwaterloo.ca\adbackups and click on Next
  • Register Backup Schedule:
    • Enter the username and password of the service account assigned to manage System State Backups - CSCF\cscf_server_backups
    • Click OK followed by Next
  • Click on Finish