DISK QUOTAS FOR RESEARCH COMPUTING HOME DIRECTORIES

**Table of Contents **

What is a disk quota?

What is the difference between a soft and hard disk quota?

What is the default quota set for home directories in Research Computing?

How do I know if I am over my limit?

How do I know what value is set for my quota limits?

What do I do if I am over my quota limit?

Examples of quota usage

Additional Help

What is a disk quota?

A disk quota is a configuration setting on file servers which monitors your use of file capacity or file counts in order to impose a limit on your ability to write data to a file system when that particular quota setting threshold is surpassed.

What is the difference between a soft and hard disk quota?

A soft quota is one which, when exceeded, will not stop writes to the file system. It merely issues a warning so that you can take action prior to hitting the hard quota threshold. Once a folder reaches its soft limit, a 7-day grace period clock is started. If the folder does not get back below the soft limit by the end of the grace period, then the usage that folder has at that point becomes the hard limit. No additional space is available in that folder until it is reduced back below the soft limit.

A hard quota is one which, when exceeded, will stop writes to the file system and issue a “Disk quota exceeded” and/or “out of space” error message.

What is the default quota set for home directories in Research Computing?

Soft: 50GB or 51200M

Hard: 70GB or 76800M

How do I know if I am over my limit?

There are three ways to check your disk consumption.

$ quota

Disk quotas for user smith (uid 20279):

     Filesystem  space   quota   limit   grace   files   quota   limit   grace

--

/nas/longleaf

                   461M  51200M  76800M            163   4295m   4295m

The output shows the user ‘smith’ has used 461Mb of their 51,200MB or 50Gb quota limit.

  • This next method is very useful in identifying your largest files which are candidates to be moved away from your home directory space and to some other location (i.e. Mass Storage).
$ du –sh $HOME/*

$ du –h /nas/longleaf/home/onyen/

How do I know what value is set for my quota limits?

$ quota

## What do I do if I am over my quota limit?

You have several options available to you if you are over your disk threshold limit for your home directory:

  • Delete unneeded files
  • Tar files into a scratch directory [/work, /pine, /21dayscratch, etc…] and move the archives into your home directory or group project space

To tar a directory named “mydir” with gzip compression, and then move the resulting tarball (i.e., myfiles.tgz) to your home directory:

$ tar –cvzf myfiles.tgz mydir; mv myfiles.tgz ~

Note: There are other compression algorithms but gzip is going to be the most efficient when dealing with massive files. Alternatively, if you have smaller files, bzip2 will provide better compression for binaries; text and audio / video files all compress similarly, so just use gzip for those as well.

The following is an example using bzip2, i.e. for smaller files or if you have the time to wait for the better compression:

$ tar –cvjf myfiles.tar.bz2 mydir; mv myfiles.tar.bz2 ~

Examples of quota usage

The following are some helpful examples of using the “quota” command.

  1. “quota” is the default quota command equivalent to “/usr/bin/quota -s” and it displays the quota usage and limits for the current user for all NFS-mounted filesystems.

The following is an example of where I am not over quota:

$ quota

Disk quotas for user smith (uid 123456):

     Filesystem  space   quota   limit   grace   files   quota   limit   grace

--

/nas/longleaf

                   461M  51200M  76800M            163   4295m   4295m

This example shows:

  • The filesystem to which the quota applies; in this case it is “/nas/longleaf” which is where user home directories are located.
  • Space shows the amount of space currently used.
  • I have a soft quota limit set to 51200MB or 50GB; this means I can write up to 50GB of data without issue, but once the soft quota is reached, warning messages are displayed
  • I have a hard limit set to 76800MB or 70GB; this means that once 70GB is reach, an I/O error will occur for out of disk space, i.e. “disk quota exceeded” or “out of disk space”
  • Ignore the second set of headings for “quota” and “limit” (which display a value of 4295m) as there are no file count quota limits in place at this time (4295m is just the default value for those fields).
  1. The following is an example of being over quota. In this example, du reveals if it is a soft or hard quota limit.
$ quota

Disk quotas for user biggie (uid 2212121):

     Filesystem  space   quota   limit   grace   files   quota   limit   grace

/nas/longleaf

                 76800M*  51200M  76800M            163   4295m   4295m

The quota output above shows that I have reached the limit of 76800M (70Gb) of space in my home directory. I should clean up some disk space, or I risk jobs failing due to not being able to write to disk.

Furthermore, if I attempt to write additional data without clean up, I may run into a file I/O error such as:

bash-3.2$ dd if=/dev/zero of=test3dd bs=1024 count=1000000

dd: writing `test3dd': Disk quota exceeded

496118+0 records in

496117+0 records out

508023808 bytes (508 MB) copied, 4.03292 seconds, 126 MB/s

The ”dd” command above should have written a 1GB file, but it was stopped because my hard quota was exceeded. In this case, it would be necessary to clean up my files in order to write to my home directory.

Additional links:

Useful tar examples / tricks: http://www.thegeekstuff.com/2010/04/unix-tar-command-examples/

Please review the help doc on Mass Storage

  • Contact Research Computing to discuss options to purchase or access additional space for your project

Additional Help

Be sure to check the Research Computing home page for information about other resources available to you.

 

Last Update 11/21/2024 1:46:48 AM