Documentation
Quotas

Quotas

To ensure that everyone has a fair share of resources, we enforce a set of quotas in the cluster.

Terminology

  • Hard limit: The limit that cannot be exceeded. If the hard limit for a disk quota is reached, the user will receive a "disk quota exceeded" error when trying to write to the disk.
  • Soft limit: The limit that can be temporarily exceeded. We set the grace period to 7 days1. If the soft limit is reached for more than 7 days, the user will receive a "disk quota exceeded" error when trying to write to the disk.

Disk quotas

Default disk quotas

This section lists the default per-user disk quotas. These quotas are subject to change as we learn more about the usage patterns.

Global disk quotas

Global disk quotas are quotas on filesystems that are shared across all nodes.

NameFS TypeSize Limit (bytes)Inode Limit
homeceph20Giunlimited

Node-local disk quotas

Node-local disk quotas are quotas on filesystems that are local to each node.

NodeNameMount PointFS TypeSize Limit (bytes)Inode Limit
SoftHardSoftHard
delta-ubuntu2docker/var/lib/clusterxfs15Gi30Giunlimitedunlimited
delta-ubuntu2scratch/mnt/scratchxfs2Gi200Giunlimitedunlimited
tr-ubuntu3docker/var/lib/clusterxfs10Gi20Giunlimitedunlimited
tr-ubuntu3scratch/mnt/scratchxfs2Gi200Giunlimitedunlimited
derek3-ubuntu2docker/var/lib/clusterxfs10Gi20Giunlimitedunlimited
derek3-ubuntu2scratch/mnt/scratchxfs2Gi200Giunlimitedunlimited

Checking your disk quota

The method for checking your disk quota depends on the filesystem type (FS Type). The filesystem type is listed in the default disk quotas section. This section describes how to check your disk quota for each filesystem type.

ceph

To check your disk quota, run the following command:

getfattr -n ceph.quota "<path>"

For example, to check the disk quota for your home directory, run the following command:

getfattr -n ceph.quota "$HOME"

In the output below, the user ben has a quota of 20 GiB and unlimited (represented by 0) files in their home directory.

# file: home/ben
ceph.quota="max_bytes=21474836480 max_files=0"

To check your current usage, run the following commands:

getfattr -n ceph.dir.rbytes "<path>"
getfattr -n ceph.dir.rfiles "<path>"

For example, to check the current usage of your home directory, run the following commands:

getfattr -n ceph.dir.rbytes "$HOME"
getfattr -n ceph.dir.rfiles "$HOME"

In the output below, the user ben has used 16.8 GiB of space across 210,104 files in their home directory.

# file: home/ben
ceph.dir.rbytes="18056911629"

# file: home/ben
ceph.dir.rfiles="210104"

xfs

To check your disk quota, run the following command2:

quota --human-readable=g,

For example, in the output below, the user ben has a soft limit of 30 GiB and a hard limit of 50 GiB on /dev/sdc, and has used 1 GiB of space.

> quota --human-readable=g,
Disk quotas for user ben (uid 1507):
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
       /dev/sdc      1G     30G    50G              51       0       0

to find out which mountpoint corresponds to /dev/sdc, run the following command:

> df -h /dev/sdc
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        200G   34G  167G  17% /var/lib/cluster

In this case, /dev/sdc is mounted on /var/lib/cluster. This drive is used to store docker-related data, as shown in the Default disk quotas section.

Requesting a disk quota increase

If you find that you regularly exceed your quotas, you can request for a quota increase by specifying your desired quota in your profile and have your WATcloud contact3 approve it. You can edit your profile using the Profile Editor.

CPU and memory quotas

On general-use machines, per-user CPU and memory quotas are enforced to ensure fair resource sharing.

NodeCPU Quota (% of 1 core)Memory Quota (bytes)
delta-ubuntu2100%2048M
tr-ubuntu3100%2048M
derek3-ubuntu2100%2048M

Unlike disk quotas, we don't allow users to request CPU and memory quota increases. Please use SLURM to run resource-intensive jobs.

Footnotes

  1. The 7-day grace period is the default on most Linux quota implementations.

  2. quota is a command-line utility that is available on most Linux installations with quota support. You can learn more about the command here (opens in a new tab) or by running man quota.

  3. Your WATcloud contact is the person as described in the Getting Access section.