Quotas

To ensure that everyone has a fair share of resources, we enforce a set of quotas in the cluster.

Terminology

• Hard limit: The limit that cannot be exceeded. If the hard limit for a disk quota is reached, the user will receive a "disk quota exceeded" error when trying to write to the disk.
• Soft limit: The limit that can be temporarily exceeded. We set the grace period to 7 days¹. If the soft limit is reached for more than 7 days, the user will receive a "disk quota exceeded" error when trying to write to the disk.

Disk quotas

Default disk quotas

This section lists the default per-user disk quotas. These quotas are subject to change as we learn more about the usage patterns.

Global disk quotas

Global disk quotas are quotas on filesystems that are shared across all nodes.

Node-local disk quotas

Node-local disk quotas are quotas on filesystems that are local to each node.

Checking your disk quota

The method for checking your disk quota depends on the filesystem type (FS Type). The filesystem type is listed in the default disk quotas section. This section describes how to check your disk quota for each filesystem type.

ceph

To check your disk quota, run the following command:

getfattr -n ceph.quota "<path>"

For example, to check the disk quota for your home directory, run the following command:

getfattr -n ceph.quota "$HOME"

In the output below, the user ben has a quota of 20 GiB and unlimited (represented by 0) files in their home directory.

# file: home/ben
ceph.quota="max_bytes=21474836480 max_files=0"

To check your current usage, run the following commands:

getfattr -n ceph.dir.rbytes "<path>"
getfattr -n ceph.dir.rfiles "<path>"

For example, to check the current usage of your home directory, run the following commands:

getfattr -n ceph.dir.rbytes "$HOME"
getfattr -n ceph.dir.rfiles "$HOME"

In the output below, the user ben has used 16.8 GiB of space across 210,104 files in their home directory.

# file: home/ben
ceph.dir.rbytes="18056911629"

# file: home/ben
ceph.dir.rfiles="210104"

xfs

To check your disk quota, run the following command²:

quota --human-readable=g,

For example, in the output below, the user ben has a soft limit of 30 GiB and a hard limit of 50 GiB on /dev/sdc, and has used 1 GiB of space.

> quota --human-readable=g,
Disk quotas for user ben (uid 1507):
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
       /dev/sdc      1G     30G    50G              51       0       0

to find out which mountpoint corresponds to /dev/sdc, run the following command:

> df -h /dev/sdc
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        200G   34G  167G  17% /var/lib/cluster

In this case, /dev/sdc is mounted on /var/lib/cluster. This drive is used to store docker-related data, as shown in the Default disk quotas section.

Requesting a disk quota increase

If you find that you regularly exceed your quotas, you can request for a quota increase by specifying your desired quota in your profile and have your WATcloud contact³ approve it. You can edit your profile using the Profile Editor.

CPU and memory quotas

On general-use machines, per-user CPU and memory quotas are enforced to ensure fair resource sharing.

Unlike disk quotas, we don't allow users to request CPU and memory quota increases. Please use SLURM to run resource-intensive jobs.