| Lustre 1.6 Operations Manual |    
|
| C H A P T E R 9 |
|
Configuring Quotas |
This chapter describes how to configure quotas and includes the following sections:
Quotas allow a system administrator to limit the amount of disk space a user or group can use in a directory. Quotas are set by root, and can be specified for individual users and/or groups. Before a file is written to a partition where quotas are set, the quota of the creator's group is checked. If a quota exists, then the file size counts towards the group's quota. If no quota exists, then the owner's user quota is checked before the file is written. Similarly, inode usage for specific functions can be controlled if a user over-uses the allocated space.
Lustre quota enforcement differs from standard Linux quota support in several ways:
Use this procedure to enable (configure) disk quotas in Lustre.
1. If you have re-complied your Linux kernel, be sure that CONFIG_QUOTA and CONFIG_QUOTACTL are enabled. Also, verify that CONFIG_QFMT_V1 and/or CONFIG_QFMT_V2 are enabled.
Quota is enabled in all Linux 2.6 kernels supplied for Lustre.
3. Mount the Lustre file system on the client and verify that the lquota module has loaded properly by using the lsmod command.
$ lsmod [root@oss161 ~]# lsmod Module Size Used by obdfilter 220532 1 fsfilt_ldiskfs 52228 1 ost 96712 1 mgc 60384 1 ldiskfs 186896 2 fsfilt_ldiskfs lustre 401744 0 lov 289064 1 lustre lquota 107048 4 obdfilter mdc 95016 1 lustre ksocklnd 111812 1
The Lustre mount command no longer recognizes the usrquota and grpquota options. If they were previously specified, remove them from /etc/fstab.
When quota is enabled, it is enabled for all file system clients (started automatically using quota_type or manually with lfs quotaon).
| Note - Lustre with the Linux kernel 2.4 does not support quotas. |
To enable quotas automatically when the file system is started, you must set the mdt.quota_type and ost.quota_type parameters, respectively, on the MDT and OSTs. The parameters can be set to the string u (user), g (group) or ug (for both users and groups).
You can enable quotas at mkfs time (mkfs.lustre --param mdt.quota_type=ug) or with tunefs.lustre. As an example:
tunefs.lustre --param ost.quota_type=ug $ost_dev
|
Caution - If you are using mkfs.lustre --param mdt.quota_type=ug or tunefs.lustre --param ost.quota_type=ug, be sure to run the command on all OSTs and the MDT. Otherwise, abnormal results may occur. |
Lustre has two kinds of quota files:
Lustre 1.6.5 introduced the v2 file format for administrative quota files, with continued support for the old file format (v1). The mdt.quota_type parameter also handles ‘1’ and ‘2’ options, to specify the Lustre quota versions that will be used. For example:
--param mdt.quota_type=ug1 --param mdt.quota_type=u2
Lustre 1.6.6 introduced the v2 file format for operational quotas, with continued support for the old file format (v1). The ost.quota_type parameter handles ‘1’ and ‘2’ options, to specify the Lustre quota versions that will be used. For example:
--param ost.quota_type=ug2 --param ost.quota_type=u1
For more information about the v1 and v2 formats, see Quota File Formats.
Once each quota-enabled file system is remounted, it is capable of working with disk quotas. However, the file system is not yet ready to support quotas. If umount has been done regularly, run the lfs command with the quotaon option. If umount has not been done:
1. Take Lustre ''offline''. That is, verify that no write operations (append, write, truncate, create or delete) are being performed (preparing to run lfs quotacheck). Operations that do not change Lustre files (such as read or mount) are okay to run.
2. Run the lfs command with the quotacheck option:
# lfs quotacheck -ug /mnt/lustre
By default, quota is turned on after quotacheck completes. Available options are:
The lfs quotacheck command checks all objects on all OSTs and the MDS to sum up for every UID/GID. It reads all Lustre metadata and re-computes the number of blocks/inodes that each UID/GID has used. If there are many files in Lustre, it may take a long time to complete.
| Note - User and group quotas are separate. If either quota limit is reached, a process with the corresponding UID/GID cannot allocate more space on the file system. |
The lfs command includes several command options to work with quotas:
lfs quotaon [-ugf] <filesystem> lfs quotaoff [-ug] <filesystem> lfs quota [-v] [-o obd_uuid] [-u <username>|-g <groupname>] <filesystem> lfs quota -t <-u|-g> <filesystem> lfs setquota <-u|--user|-g|--group> <username|groupname> [-b <block-softlimit>] [-B <block-hardlimit>] [-i <inode-softlimit>] [-I <inode-hardlimit>] <filesystem>
In all of the examples below, the file system is /mnt lustre.
To turn on user and group quotas, run:
$ lfs quotaon -ug /mnt/lustre
To turn off user and group quotas, run:
$ lfs quotaoff -ug /mnt/lustre
To display general quota information (disk usage and limits) for the user running the command and his primary group, run:
$ lfs quota /mnt/lustre
To display general quota information for a specific user ("bob" in this example), run:
$ lfs quota -u bob /mnt/lustre
To display general quota information for a specific user ("bob" in this example) and detailed quota statistics for each MDT and OST, run:
$ lfs quota -u bob -v /mnt/lustre
To display general quota information for the group to which a specific user ("bob" in this example) belongs, run:
$ lfs quota -g bob /mnt/lustre
To display general quota information for a specific group ("eng" in this example), run:
$ lfs quota -g eng /mnt/lustre
To display block and inode grace times for user quotas, run:
$ lfs quota -t -u /mnt/lustre
To set user and group quotas for a specific user ("bob" in this example), run:
$ lfs setquota -u bob 307200 309200 10000 11000 /mnt/lustre
In this example, the quota for user "bob" is set to 300 MB (309200*1024) and the hard limit is 11,000 files. Therefore, the inode hard limit should be 11000.
| Note - For the Lustre command $ lfs setquota/quota ... the qunit for block is KB (1024) and the qunit for inode is 1. |
The quota command displays the quota allocated and consumed for each Lustre device. Using the previous setquota example, running this lfs quota command:
$ lfs quota -u bob -v /mnt/lustre
Disk quotas for user bob (uid 500): Filesystem blocks quota limit grace files quota limit grace /mnt/lustre 0 307200 309200 0 10000 11000 lustre-MDT0000_UUID 0 0 102400 0 0 5000 lustre-OST0000_UUID 0 0 102400 lustre-OST0001_UUID 0 0 102400
The Linux kernel sets a default quota size of 1 MB. (For a block, the default is 128 MB. For files, the default is 5120.) Lustre handles quota allocation in a different manner. A quota must be properly set or users may experience unnecessary failures. The file system block quota is divided up among the OSTs within the file system. Each OST requests an allocation which is increased up to the quota limit. The quota allocation is then quantized to reduce the number of quota-related request traffic. By default, Lustre supports both user and group quotas to limit disk usage and file counts.
The quota system in Lustre is completely compatible with the quota systems used on other file systems. The Lustre quota system distributes quotas from the quota master. Generally, the MDS is the quota master for both inodes and blocks. All OSTs and the MDS are quota slaves to the OSS nodes. The minimum transfer unit is 100 MB, to avoid performance impacts for quota adjustments. The file system block quota is divided up among the OSTs and the MDS within the file system. Only the MDS uses the file system inode quota.
This means that the minimum quota for block is 100 MB* (the number of OSTs + the number of MDSs), which is 100 MB* (number of OSTs + 1). The minimum quota for inode is the inode qunit. If you attempt to assign a smaller quota, users maybe not be able to create files. The default is established at file system creation time, but can be tuned via /proc values (described below). The inode quota is also allocated in a quantized manner on the MDS.
This sets a much smaller granularity. It is specified to request a new quota in units of 100 MB and 500 inodes, respectively. If we look at the setquota example again, running this lfs quota command:
# lfs quota -u bob -v /mnt/lustre
Disk quotas for user bob (uid 500): Filesystem blocks quota limit grace files quota limit grace /mnt/lustre 207432 307200 30920 1041 10000 11000 lustre-MDT0000_UUID 992 0 102400 1041 0 5000 lustre-OST0000_UUID 103204* 0 102400 lustre-OST0001_UUID 103236* 0 102400
The total quota of 30,920 is allotted to user bob, which is further distributed to two OSTs and one MDS with a 102,400 block quota.
The requested quota of 300 MB is divided across the OSTs. Each OST has an initial allocation of 100 MB blocks, with iunit limiting to 5000.
Grace period -- The period of time (in seconds) within which users are allowed to exceed their soft limit. There are four types of grace periods:
The grace periods are applied to all users. The user block soft limit is for all users who are using a blocks quota.
Soft limit -- Once you are beyond the soft limit, the quota module begins to time, but you still can write block and inode. When you are always beyond the soft limit and use up your grace time, you get the same result as the hard limit. For inodes and blocks, it is the same. Usually, the soft limit MUST be less than the hard limit; if not, the quota module never triggers the timing. If the soft limit is not needed, leave it as zero (0).
Hard limit -- When you are beyond the hard limit, you get -EQUOTA and cannot write inode/block any more. The hard limit is the absolute limit. When a grace period is set, you can exceed the soft limit within the grace period if are under the hard limits.
Lustre quota allocation is controlled by two values quota_bunit_sz and quota_iunit_sz referring to KBs and inodes respectively. These values can be accessed on the MDS as /proc/fs/lustre/mds/*/quota_* and on the OST as /proc/fs/lustre/obdfilter/*/quota_*. The /proc values are bounded by two other variables quota_btune_sz and quota_itune_sz. By default, the *tune_sz variables are set at 1/2 the *unit_sz variables, and you cannot set *tune_sz larger than *unit_sz. You must set bunit_sz first if it is increasing by more than 2x, and btune_sz first if it is decreasing by more than 2x.
Total number of inodes -- To determine the total number of inodes, use lfs df -i (and also /proc/fs/lustre/*/*/filestotal). For more information on using the lfs df -i command and the command output, see Checking File System Free Space.
Unfortunately, the statfs interface does not report the free inode count directly, but instead reports the total inode and used inode counts. The free inode count is calculated for df from (total inodes - used inodes).
It is not critical to know a file system’s total inode count. Instead, you should know (accurately), the free inode count and the used inode count for a file system. Lustre manipulates the total inode count in order to accurately report the other two values.
The values set for the MDS must match the values set on the OSTs.
The quota_bunit_sz parameter displays bytes, however lfs setquota uses KBs. The quota_bunit_sz parameter must be a multiple of 1024. A proper minimum KB size for lfs setquota can be calculated as:
Size in KBs = (quota_bunit_sz * ( number of OSTS + 1 )) / 1024
We add one (1) to the number of OSTs as the MDS also consumes KBs. As inodes are only consumed on the MDS, the minimum inode size for lfs setquota is equal to quota_iunit_sz.
| Note - Setting the quota below this limit may prevent the user from all file creation. |
Using quotas in Lustre can be complex and there are several known issues.
In Lustre, granted cache does not respect quota limits. In this situation, OSTs grant cache to Lustre client to accelerate I/O. Granting cache causes writes to be successful in OSTs, even if they exceed the quota limits, and will overwrite them.
1. A user writes files to Lustre.
2. If the Lustre client has enough granted cache, then it returns ‘success’ to users and arranges the writes to the OSTs.
3. Because Lustre clients have delivered success to users, the OSTs cannot fail these writes.
Because of granted cache, writes always overwrite quota limitations. For example, if you set a 400 GB quota on user A and use IOR to write for user A from a bundle of clients, you will write much more data than 400 GB, and cause an out-of-quota error (-EDQUOT).
Available quota limits depend on the Lustre version you are using.
Lustre 1.6.5 introduced the v2 file format for administrative quotas, with 64-bit limits that support large-limits handling. The old quota file format (v1), with 32-bit limits, is also supported. Lustre 1.6.6 introduced the v2 file format for operational quotas. A few notes regarding the current quota file formats:
Lustre 1.6.5 and later use mdt.quota_type to force a specific administrative quota version (v2 or v1).
Lustre 1.6.6 and later use ost.quota_type to force a specific operational quota version (v2 or v1).
The quota_type specifier can be used to set different combinations of administrative/operational quota file versions on a Lustre node:
If quotas do not exist or look broken, then quotacheck creates quota files of a required name and format.
If Lustre is using the v2 quota file format when only v1 quota files exist, then quotacheck converts old v1 quota files to new v2 quota files. This conversion is triggered automatically, and is transparent to users. If an old quota file does not exist or looks broken, then the new v2 quota file will be empty. In case of an error, details can be found in the kernel log of the corresponding MDS/OST. During conversion of a v1 quota file to a v2 quota file, the v2 quota file is marked as broken, to avoid it being used if a crash occurs. The quota module does not use broken quota files (keeping quota off).
In most situations, Lustre administrators do not need to set specific versioning options. Upgrading Lustre without using quota_type to force specific quota file versions results in quota files being upgraded automatically to the latest version. The option ensures backward compatibility, preventing a quota file upgrade to a version which is not supported by earlier Lustre versions.
Lustre includes statistics that monitor quota activity, such as the kinds of quota RPCs sent during a specific period, the average time to complete the RPCs, etc. These statistics are useful to measure performance of a Lustre file system.
Each quota statistic consists of a quota event and min_time, max_time and sum_time values for the event.
Quota statistics are an important measure of a Lustre file system’s performance. Interpreting these statistics correctly can help you diagnose problems with quotas, and may indicate adjustments to improve system performance.
For example, if you run this command on the OSTs:
cat /proc/fs/lustre/lquota/lustre-OST0000/stats
You will get a result similar to this:
snapshot_time 1219908615.506895 secs.usecs async_acq_req 1 samples [us] 32 32 32 async_rel_req 1 samples [us] 5 5 5 nowait_for_pending_blk_quota_req(qctxt_wait_pending_dqacq) 1 samples [us] 2 2 2 quota_ctl 4 samples [us] 80 3470 4293 adjust_qunit 1 samples [us] 70 70 70 ....
In the first line, snapshot_time indicates when the statistics were taken. The remaining lines list the quota events and their associated data.
In the second line, the async_acq_req event occurs one time. The min_time, max_time and sum_time statistics for this event are 32, 32 and 32, respectively. The unit is microseconds (s).
In the fifth line, the quota_ctl event occurs four times. The min_time, max_time and sum_time statistics for this event are 80, 3470 and 4293, respectively. The unit is microseconds (s).
Quota statistics are collected in /proc/fs/lustre/lquota/.../stats. Each MDT and OST has one statistics proc file. If you have a problem with quotas, but cannot successfully diagnose the issue, send the statistics files in the folder to Lustre Support for analysis. To prepare the files:
1. Initialize the statistics data to 0 (zero). Run:
lctl set_param lquota.${FSNAME}-MDT*.stats=0
lctl set_param lquota.${FSNAME}-OST*.stats=0
2. Perform the quota operation that causes the problem or degraded performance.
3. Collect all statistics in /proc/fs/lustre/lquota/ and send them to Lustre Support. Note the following:
/proc/fs/lustre/obdfilter/lustre-OSTXXXX/quota* and /proc/fs/lustre/mds/lustre-MDTXXXX/quota*
Proc quota entries are copied to /proc/fs/lustre/lquota.
/proc/fs/lustre/obdfilter/lustre-OSTXXXX/ and /proc/fs/lustre/mds/lustre-MDTXXXX/
| Lustre 1.6 Operations Manual | 820-3681-11 |
|
Copyright © 2010, Oracle and/or its affiliates. All rights reserved.