Enduria - by Guido van Brakel

With Windows Server 2012, there is a new cool feature, called Data Deduplication. Data deduplication involves finding and removing duplication within data without compromising its fidelity or integrity. The goal is to store more data in less space by segmenting files into small variable-sized chunks (32–128 KB), identifying duplicate chunks, and maintaining a single copy of each chunk. Redundant copies of the chunk are replaced by a reference to the single copy. The chunks are compressed and then organized into special container files in the System Volume Information folder. In addition, chunks are also compressed for further space optimization.

The result is an on-disk transformation of each file as shown in the picture below. After deduplication, files are no longer stored as independent streams of data, and they are replaced with stubs that point to data blocks that are stored within a common chunk store. Because these files share blocks, those blocks are only stored once, which reduces the disk space needed to store all files. During file access, the correct blocks are transparently assembled to serve the data without calling the application or the user having any knowledge of the on-disk transformation to the file. This enables administrators to apply deduplication to files without having to worry about any change in behavior to the applications or impact to users who are accessing those files.

Data Deduplication in Windows Server 2012 is designed to be installed on primary data volumes without adding additional dedicated hardware. This means that you can install and use the feature without impacting the primary workload on the server. Ideal workloads include software deployment shares, virtual machine template folders and archived data folders, where data is not changed much and is quite static. Data deduplication requires an NTFS file system and is not supported for the new ReFS file system which is introduced in Windows Server 2012.

How Do I Install Data Deduplication?

Step 1:Add the role service using the Server Manager:

1. From the Add Roles and Features Wizard, under Server Roles, select File and Storage Services (if it has not already been installed).
2. Select the File Services check box, and then select the Data Deduplication check box.
3. Click Next until the Install button is active, and then click Install.

This can also be done using Powershell:


PS C:\> Import-Module ServerManager
PS C:\> Add-WindowsFeature -name FS-Data-Deduplication
PS C:\> Import-Module Deduplication

Step 2: Enable data deduplication for a certain volume

1. From the Server Manager dashboard, right-click a data volume and choose Configure Data Deduplication. The Deduplication Settings page appears.
2. Select the Enable data deduplication check box, enter the number of days that should elapse from the date of file creation until files are deduplicated, enter the extensions of any file types that should not be deduplicated, and then click Add to browse to any folders with files that should not be deduplicated.
3. Click Apply to apply these settings and return to the Server Manager dashboard, or click the Set Deduplication Schedule button to continue to set up a schedule for deduplication.

Again this can also be done using Powershell:

Here we enable it for the on volume E:

PS C:\> Enable-DedupVolume E:

We can also set the minimum number of days that must pass before a file is deduplicated by using the following command

PS C:\> Set-Dedupvolume E: -MinimumFileAgeDays 20

If you set MinimumFileAgeDays to 0, deduplication will process all files, regardless of their age. This is suitable for a test environment, where you want to exercise maximum deduplication. In a production environment, however, it is preferable to wait for a number of days (the default is 5 days), because files tend to change a lot for a brief period of time before the change rate slows. This allows for the most efficient use of your server resources.

3.Configure Data Deduplication

In Windows Server 2012, data deduplication jobs can be run on demand (manually) or scheduled. There are three types of jobs that you can perform on a volume: Optimization, Data Scrubbing, and Garbage Collection. The Data Deduplication feature comes with built-in jobs that will automatically launch and optimize the specified volume(s) on a regular basis. Optimization jobs deduplicate data and compress file chunks on a volume per the policy settings. After the initial optimization is complete, optimization jobs run on the files that are included in the policies, according to the job schedules that you have configured or the default job schedules that ship with the product. You can trigger an optimization job on demand in Windows PowerShell by using the Start-DedupJob cmdlet. For example:

PS C:\> Start-DedupJob –Volume E: –Type Optimization

You can query the progress of the job on the volume by using the Get-DedupJob cmdlet:

Get-DedupJob

The Get-DedupJob command show current jobs that are running or are queued to run.

How Do I Check the Integrity of a Volume?

Data Deduplication has built-in data integrity features such as checksum validation and metadata consistency checking. It also has built-in redundancy for critical metadata and the most popular data chunks. As data is accessed or jobs process data, these features may encounter corruption, and they will record the corruption in a log file. Scrubbing jobs use these features to analyze the chunk store corruption logs, and when possible, to make repairs.

Possible repair operations include using three sources of redundant data:

1. Deduplication keeps backup copies of popular chunks when they are referenced over 100 times in an area called the hotspot. If the working copy is corrupted, deduplication will use the backup.
2. When using Storage Spaces in a mirrored configuration, deduplication can use the mirror image of the redundant chunk to serve the I/O and fix the corruption.
3. If a file is processed with a chunk that is corrupted, the corrupted chunk is eliminated, and the new incoming chunk is used to fix the corruption.

Scrubbing jobs output a summary report in the Windows event log located here:
Event Viewer\Applications and Services Logs\Microsoft\Windows\Deduplication\Scrubbing
Data Deduplication defaults create a data integrity scrubbing job on a weekly basis, but you can also trigger one on demand by using the following command:

PS C:\> Start-DedupJob E: –Type Scrubbing

This initiates a job that attempts to repair all corruptions that were logged in to the deduplication internal corruption log, during the I/O to deduplication files. To check the data integrity of all the deduplicated data on the volume, use the -full parameter:

PS C:\> Start-DedupJob E: –Type Scrubbing -full

Also known as Deep Scrubbing, the -full parameter will scrub the entire set of deduplicated data and look for all corruptions that are causing data access failures.

How Much Disk Space Can I Expect to Reclaim?

When you install the Data Deduplication role service on a server running Windows Server 2012, DDPEVAL.EXE is also installed in the C:\Windows\System32 folder as an additional command-line tool.  DDPEVAL.EXE can be run against any local NTFS volumes or NTFS network shares to estimate the amount of disk space that can potentially be reclaimed by moving that data to a Windows Server 2012 NTFS volume with Data Deduplication enabled. 

C:\> DDPEVAL \\server\folder /V /O:logfile.txt