Scoped shadow copies

Have you ever heard of scoped shadow copies? They have been around since the release of Windows 8, but not much information is available on this topic.

A shadow copy becomes scoped when data blocks not required by the system restore process are excluded from copy-on-write operations. When you create a restore point, a scoped shadow copy is created by default for a system volume (in Windows 8, 8.1 & 10).

A scoped shadow copy is created just like the usual one, but right after this shadow copy is available to the system, “unnecessary” data blocks are removed by the Windows System Protection background tasks (the executable file is “SrTasks.exe”, it uses the “srcore.dll” library to discover and report “unnecessary” data blocks).

Technically, “unnecessary” data blocks are simply marked in the bitmap of the shadow copy (by changing corresponding bits to 1, the meaning of this bitmap is described below).

For usual shadow copies, unallocated space, clusters occupied by several system files (a page file, a hibernation file, and a swap file), and clusters occupied by shadow copies (and related metadata) are marked in this bitmap. For scoped shadow copies, this extends to allocated clusters of many user files. Since each bit in the bitmap corresponds to a 16384-byte block (and in most cases, a cluster is smaller than such a block), actual user data can survive in many cases.

When reading a 16384-byte block marked in the bitmap, one of the following can be returned as data:

• null bytes,
• data from a current volume (directly from a requested offset).

A decision is made based on the flags set within a shadow copy descriptor (as a side note: this is not documented here).

The same bitmap is also used when performing a copy-on-write operation. If a block is marked in this bitmap, a copy of this block isn’t made when it’s overwritten. Thus, this block is no longer available in a shadow copy.

As a result, many files found in a shadow copy can contain invalid data (as documented here and here). Here is an example of an SQLite3 file (Chrome history) found in a virtual Windows 10 machine:

Some statistics captured using a kernel debugger (for a system volume of a virtual Windows 10 machine):

• Total number of bits in the bitmap: 3898704.
• Number of set bits before scoping: 2582485 (66%).
• Number of set bits after scoping: 3198408 (82%).
• More than 9 GiB of data was excluded by scoping!

And here is a quote from Microsoft:

I’ve confirmed with product team, this is an expected behavior. Scope snapshot is a special volume snapshot for volsnap performance. Mainly used by Windows critical updates. Scope means the volsnap only creates Copy on Write (Shadow) volume for the files that are involved in the updates instead of all the files on the volume.

Scope snapshot prevents volume snap revert operations, thus it will cause Hyper-V host backup failure since Hyper-V needs to perform mount and revert VM vhd operations on Hyper-V host. It can cause the hyper-v host backup failed even you didn’t include the hyper-v guest. That is the reason that we blocked it inside of VM during the backup.

(Source.)

So, scoped shadow copies are inconsistent when dealing with user files. Invalid data is expected in this type of shadow copies.