Setting HDFS Quotas

You can set quotas in HDFS for:
  • The number of file and directory names used
  • The amount of space used by given directories
Points to note:
  • The quotas for names and the quotas for space are independent of each other.
  • File and directory creation fails if the creation would cause the quota to be exceeded.
  • The Reports Manager must index a file or directory before you can set a quota for it.
  • Allocation fails if the quota would prevent a full block from being written; keep this in mind if you are using a large block size.
  • If you are using replication, remember that each replica of a block counts against the quota.

About file count limits

  • The file count quota is a limit on the number of file and directory names in the directory configured.
  • A directory counts against its own quota, so a quota of 1 forces the directory to remain empty.
  • File counts are based on the intended replication factor for the files; changing the replication factor for a file will credit or debit quotas.

About disk space limits

  • The space quota is a hard limit on the number of bytes used by files in the tree rooted at the directory being configured.
  • Each replica of a block counts against the quota.
  • The disk space quota calculation takes replication into account, so it uses the replicated size of each file, not the user-facing size.
  • The disk space quota calculation includes open files (files presently being written), as well as files already written.
  • Block allocations for files being written will fail if the quota would not allow a full block to be written.

Setting HDFS Quotas Using Cloudera Manager

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

  1. From the HDFS service page, select the File Browser tab.
  2. Browse the file system to find the directory for which you want to set quotas.
  3. Click the directory name so that it appears in the gray panel above the listing of its contents and in the detail section to the right of the File Browser table.
  4. Click the Edit Quota button for the directory. A Manage Quota pop-up displays, where you can set file count or disk space limits for the directory you have selected.
  5. When you have set the limits you want, click OK.

Setting HDFS Quotas Using the Command Line

To set space quotas on a directory: 
$ sudo -u hdfs hdfs dfsadmin -setSpaceQuota n directory
where n is a number of bytes and directory is the directory the quota applies to. You can specify multiple directories in a single command; n applies to each.
To remove space quotas from a directory: 
$ sudo -u hdfs hdfs dfsadmin -clrSpaceQuota directory
You can specify multiple directories in a single command.
To set name quotas on a directory: 
$ sudo -u hdfs hdfs dfsadmin -setQuota n directory
where n is the number of file and directory names in directory. You can specify multiple directories in a single command; n applies to each.
To remove name quotas from a directory: 
$ sudo -u hdfs hdfs dfsadmin -clrQuota directory
You can specify multiple directories in a single command.

For More Information

For more information, see the HDFS Quotas Guide.