SNAPSHOTS FOR FILE RECOVERY IN RESEARCH COMPUTING NAS FILESYSTEMS

Table of Contents

What is Snapshot?

Background

Research Computing Snapshot Schedule and Retention Policy

Getting an Older Version of a File Back from Snapshot

Example

Additional Helpful Notes

Additional Help

What is Snapshot?

Snapshot is a technology that provides efficient time-based archives of data so that each user can quickly restore a file accidentally deleted or changed to its previous state on his/her own. Older versions of files are available based on the time the snapshot was taken – not based on each time the file was changed. It is not a file versioning system: It does not keep every version of a saved file. Instead, it knows what the file contained at a particular time. Every user with an account on UNC’s research clusters automatically has home directory files in the snapshot system.

Snapshot supports only directories and files in Research Computing’s NAS Filesystems: home: /nas/longleaf/home/ and dept (/nas/depts) spaces. No other locations are protected with Snapshot (e.g. /work/, /pine/scr/, /21dayscratch/, /proj/, ...).

Background

Snapshot gives us a fast self-service automated file recovery system. Behind the scenes, Snapshot stores the addresses on the disk of changed sections of files, not a copy of the entire changed file. This saves space and makes for a faster back up process. It reconstitutes a requested file by piecing together the list of file addresses on the disk for the requested timestamp. As a result, restoring a file from its collection of snapshots is almost instantaneous (taking only seconds to produce).

Snapshot by itself does not provide full disaster recovery services because it does not keep copies of data off-site.

Research Computing Snapshot Schedule and Retention Policy

Research Computing takes hour-based snapshots at 11am and 4pm and keeps the most current 11am and 4pm snapshots. Snapshot has the ability to update hourly, nightly and/or weekly. Hour-based, or hourly, snapshots are those scheduled for a particular time every day (not every hour of the day). Hourly snapshots are useful for mistakes made that you recognize immediately, i.e. for when you accidentally delete a file, update the wrong file, or overwrite a file copy. You can quickly correct your mistake by copying files out of one of the two hourly snapshots. There will always be only two hourly snapshots: When today's 11am snapshot is created, yesterday's 11am shapshot is deleted but yesterday's 4pm snapshot remains until today at 4pm. At 4pm the 4pm snapshot from yesterday is removed and a new 4pm one created. If you make changes to a file at noon and then accidentally delete it at 3, snapshot will be able to restore the one it had as of 11am (not noon).

Research Computing takes day-based snapshots daily at midnight and keeps the last 24 days, space permitting. Nightly snapshots happen at midnight Monday through Saturday. If used, the weekly snapshot is taken at midnight on Sundays. Nightly and weekly snapshots are useful when time has passed before you realize a file has been compromised and you need the version of the file from a day or more ago.

One caveat: As already mentioned, disk space consumption is required for snapshots based on the rate-of-change within the filesystem. If a high rate-of-change or odd use of the filesystem causes the filesystem to run low on disk space, there is a mechanism in place to delete the oldest snapshot in order to free up space in the filesystem. This should rarely (if ever) happen.

Getting an Older Version of a File Back from Snapshot

As mentioned previously, each snapshot uses disk space in the filesystem. This means that snapshots can be accessed from the filesystem itself. It also means you can restore what you need without having to submit a request for a restore to the system administrators (self-service). The block method Snapshot uses means that while each snapshot does impact remaining space available due to space quotas, snapshot files for a directory are so small (typically < 10k) that the impact is minimal.

Users access a snapshot by listing or changing directories to the “.snapshot” sub-directory in the directory where the file was stored. To restore files, go to the .snapshot sub-directory that has the file you need and copy that file out of snapshot to either the same location it existed previously or to a new location (so as not to overwrite the existing file in the current filesystem space). Do not try to edit it in its snapshot location.

Example

Let’s look at an example where today is 4/10/2023, 2pm

On the research clusters (e.g. longleaf, dogwood) your home directory and starting point upon login is the /nas/longleaf/home/<onyen>/directory. You have a large quota of space and are encouraged to save any applications, scripts or smaller input/output files in this location. Please note the following example will work for all directories and files in Research Computing’s NAS Filesystems: home and dept space. No other spaces are protected with snapshot (e.g. work/, pine/, proj/, ...).

[joesonyen@longleaf-login1 ~]$ pwd 

/nas/longleaf/home/joesonyen

Here's a list of all of the snapshots times available for today (4/10/2023, 2pm):

[joesonyen@longleaf-login1 ~]$ ls -l .snapshot

total 416
drwxr-x--- 88 joesonyen users 16384 Mar 17 17:08 transition_snapshot_schedule_6.2023-03-18_0000
drwxr-x--- 88 joesonyen users 16384 Mar 17 17:08 transition_snapshot_schedule_6.2023-03-19_0000
drwxr-x--- 88 joesonyen users 16384 Mar 17 17:08 transition_snapshot_schedule_6.2023-03-20_0000
drwxr-x--- 88 joesonyen users 16384 Mar 17 17:08 transition_snapshot_schedule_6.2023-03-21_0000
drwxr-x--- 88 joesonyen users 16384 Mar 17 17:08 transition_snapshot_schedule_6.2023-03-22_0000
drwxr-x--- 88 joesonyen users 16384 Mar 17 17:08 transition_snapshot_schedule_6.2023-03-23_0000
drwxr-x--- 88 joesonyen users 16384 Mar 17 17:08 transition_snapshot_schedule_6.2023-03-24_0000
drwxr-x--- 88 joesonyen users 16384 Mar 24 17:04 transition_snapshot_schedule_6.2023-03-25_0000
drwxr-x--- 88 joesonyen users 16384 Mar 24 17:04 transition_snapshot_schedule_6.2023-03-26_0000
drwxr-x--- 88 joesonyen users 16384 Mar 24 17:04 transition_snapshot_schedule_6.2023-03-27_0000
drwxr-x--- 88 joesonyen users 16384 Mar 27 17:54 transition_snapshot_schedule_6.2023-03-28_0000
drwxr-x--- 88 joesonyen users 16384 Mar 28 10:30 transition_snapshot_schedule_6.2023-03-29_0000
drwxr-x--- 88 joesonyen users 16384 Mar 28 10:30 transition_snapshot_schedule_6.2023-03-30_0000
drwxr-x--- 88 joesonyen users 16384 Mar 30 11:14 transition_snapshot_schedule_6.2023-03-31_0000
drwxr-x--- 88 joesonyen users 16384 Mar 30 11:14 transition_snapshot_schedule_6.2023-04-01_0000
drwxr-x--- 88 joesonyen users 16384 Mar 30 11:14 transition_snapshot_schedule_6.2023-04-02_0000
drwxr-x--- 88 joesonyen users 16384 Mar 30 11:14 transition_snapshot_schedule_6.2023-04-03_0000
drwxr-x--- 88 joesonyen users 16384 Apr  3 17:04 transition_snapshot_schedule_6.2023-04-04_0000
drwxr-x--- 88 joesonyen users 16384 Apr  3 17:04 transition_snapshot_schedule_6.2023-04-05_0000
drwxr-x--- 89 joesonyen users 16384 Apr  5 16:06 transition_snapshot_schedule_6.2023-04-06_0000
drwxr-x--- 89 joesonyen users 16384 Apr  5 16:06 transition_snapshot_schedule_6.2023-04-07_0000
drwxr-x--- 89 joesonyen users 16384 Apr  5 16:06 transition_snapshot_schedule_6.2023-04-08_0000
drwxr-x--- 89 joesonyen users 16384 Apr  5 16:06 transition_snapshot_schedule_6.2023-04-09_0000
drwxr-x--- 89 joesonyen users 16384 Apr  5 16:06 transition_snapshot_schedule_6.2023-04-10_0000
drwxr-x--- 89 joesonyen users 16384 Apr  5 16:06 transition_snapshot_schedule_7.2023-04-09_1600
drwxr-x--- 89 joesonyen users 16384 Apr 10 10:57 transition_snapshot_schedule_7.2023-04-10_1100

Let's look at the most recent snapshot (ending in 2023-04-10_1100) for a directory called testJob and recover the most recent version of a file called sbatch.sample into a directory called ~/anotherDir/. There are two ways to do that, both give the same result:

  1. Start with the snapshot timestamp you want ( .snapshot/transition_snapshot_schedule_7.**2023-04-10_1100**/ ) and then navigate in that directory for the _testJob_ subdirectory in it:
[joesonyen@longleaf-login1 ~]$ ls -l .snapshot/transition_snapshot_schedule_7.2023-04-10_1100/testJob/
      
total 16
drwxr-xr-x 2 joesonyen users 4096 Dec 21  2018 a
drwxr-xr-x 2 joesonyen users 4096 Dec 21  2018 b
-rw-r--r-- 1 joesonyen users   61 Dec 21  2018 sbatch.sample
-rw-r--r-- 1 joesonyen users   99 Dec 21  2018 slurm-4153754.out
-rwxr--r-- 1 joesonyen users  109 Dec 21  2018 testSlurm.sh
      
[joesonyen@longleaf-login1 ~]$ cp -p .snapshot/transition_snapshot_schedule_7.2023-04-10_1100/testJob/sbatch.sample ~/anotherDir/

Notice the “-p” flag with the cp command above to preserve file attributes: permissions, timestamps, etc. Without this flag, the copy is seen as a new write and thus the timestamp is the current time and the permissions are based upon your umask and primary UNIX GID group.

  1. Navigate to the directory where you know the file is/was and look at the snapshot timestamp there:
[joesonyen@longleaf-login1 ~]$ cd testJob
[joesonyen@longleaf-login1 ~]$ ls -l ./snapshot/transition_snapshot_schedule_7.2023-04-10_1100/
      
total 16
drwxr-xr-x 2 joesonyen users 4096 Dec 21  2018 a
drwxr-xr-x 2 joesonyen users 4096 Dec 21  2018 b
-rw-r--r-- 1 joesonyen users   61 Dec 21  2018 sbatch.sample
-rw-r--r-- 1 joesonyen users   99 Dec 21  2018 slurm-4153754.out
-rwxr--r-- 1 joesonyen users  109 Dec 21  2018 testSlurm.sh
     
[joesonyen@longleaf-login1 ~]$ cp -p ./snapshot/transition_snapshot_schedule_7.2023-04-10_1100/sbatch.sample ~/anotherDir/

Notice the “-p” flag with the cp command above to preserve file attributes: permissions, timestamps, etc. Without this flag, the copy is seen as a new write and thus the timestamp is the current time and the permissions are based upon your umask and primary UNIX GID group.

Additional Helpful Notes

When listing snapshot, use the –u flag with the ls command to see the access times and when the snapshot was created. Otherwise, you would go by the snapshot name.

[joesonyen@longleaf-login1 ~]$ ls -lt .snapshot
      
total 208
drwxr-xr-x 2 joesonyen users 8192 Apr 10 09:48 transition_snapshot_schedule_7.2023-04-10_1100
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_6.2023-04-04_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_6.2023-04-05_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_6.2023-04-06_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_6.2023-04-07_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_6.2023-04-08_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_6.2023-04-09_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_6.2023-04-10_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 16:31 transition_snapshot_schedule_7.2023-04-09_1600
drwxr-xr-x 2 joesonyen users 8192 Mar 30 11:14 transition_snapshot_schedule_6.2023-03-31_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 30 11:14 transition_snapshot_schedule_6.2023-04-01_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 30 11:14 transition_snapshot_schedule_6.2023-04-02_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 30 11:14 transition_snapshot_schedule_6.2023-04-03_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-18_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-19_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-20_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-21_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-22_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-23_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-24_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-25_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-26_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-27_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-28_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-29_0000
drwxr-xr-x 2 joesonyen users 8192 Mar  6 16:21 transition_snapshot_schedule_6.2023-03-30_0000

Compared with:

[joesonyen@longleaf-login5 ref]$ ls -ltu .snapshot

total 208
drwxr-xr-x 2 joesonyen users 8192 Apr 10 11:00 transition_snapshot_schedule_7.2023-04-10_1100
drwxr-xr-x 2 joesonyen users 8192 Apr 10 00:00 transition_snapshot_schedule_6.2023-04-10_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  9 16:00 transition_snapshot_schedule_7.2023-04-09_1600
drwxr-xr-x 2 joesonyen users 8192 Apr  9 00:00 transition_snapshot_schedule_6.2023-04-09_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  8 00:00 transition_snapshot_schedule_6.2023-04-08_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  7 00:00 transition_snapshot_schedule_6.2023-04-07_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  6 00:00 transition_snapshot_schedule_6.2023-04-06_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  5 00:00 transition_snapshot_schedule_6.2023-04-05_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  4 00:00 transition_snapshot_schedule_6.2023-04-04_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  3 00:00 transition_snapshot_schedule_6.2023-04-03_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  2 00:00 transition_snapshot_schedule_6.2023-04-02_0000
drwxr-xr-x 2 joesonyen users 8192 Apr  1 00:00 transition_snapshot_schedule_6.2023-04-01_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 31 00:00 transition_snapshot_schedule_6.2023-03-31_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 30 00:00 transition_snapshot_schedule_6.2023-03-30_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 29 00:00 transition_snapshot_schedule_6.2023-03-29_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 28 00:00 transition_snapshot_schedule_6.2023-03-28_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 27 00:00 transition_snapshot_schedule_6.2023-03-27_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 26 00:00 transition_snapshot_schedule_6.2023-03-26_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 25 00:00 transition_snapshot_schedule_6.2023-03-25_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 24 00:00 transition_snapshot_schedule_6.2023-03-24_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 23 00:00 transition_snapshot_schedule_6.2023-03-23_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 22 00:00 transition_snapshot_schedule_6.2023-03-22_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 21 00:00 transition_snapshot_schedule_6.2023-03-21_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 20 00:00 transition_snapshot_schedule_6.2023-03-20_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 19 00:00 transition_snapshot_schedule_6.2023-03-19_0000
drwxr-xr-x 2 joesonyen users 8192 Mar 18 00:00 transition_snapshot_schedule_6.2023-03-18_0000
  • To preserve file attributes, use cp –p or rsync –a when copying from snapshots to the active filesystem.
  • If copying lots of files or directories, use rsync instead of cp.
  • Snapshot permissions are the same as filesystem permissions; if you don’t have access…you don’t have access.

Additional Help

Be sure to check the Research Computing page for information about other resources available to you.

 

Last Update 11/21/2024 1:31:45 AM