Now that we have a general idea of what a cluster is and how it is built, let’s see what resources and queues are available.
There are three types of useful resources: hardware, logical, and time. The most useful hardware resources are:
- select – number of resource chunks. Minimal chunk has one CPU and some memory. A job may need several chunks or even different chunks;
- ncpus – number of cores requested per chunk;
- mem – amount of memory per chunk, may be specified in bytes (the default), kb (kilobytes), mb (megabytes), and gb (gigabytes);
- ngpus – number of GPU processors, if relevant.
The default allocation is select=1:ncpus=1:mem=1000mb.
Some important logical resources:
- mpiprocs – number of MPI processes per chunk; the default is 1
- ompthreads – number of OpenMP threads per chunk; the default value is equal to “ncpus”.
Time is also a resource. There is a walltime, i.e. the actual time the job runs. If the job started at 13:43 and finished at 17:54, the walltime is 4 hours and 11 minutes. The cputime refers to the CPU untilization. If a program uses 4 CPU for 3 hours and utilizes them at 100%, then the cputime is 12 hours. We use the walltime in all our definitions; however, the cputime may give you an indication of the efficiency of your application.
A queue is a container for jobs having some sort of similarity. It could be similar runtime, or hardware, or permissions. In our system we have two types of queues: private and public. Private queues are sleep, long and medium. They differ by the maximal allowed walltime. Medium queue allows up to 48 hours walltime and long – up to 60 days. The default walltime value for the long queue is 7 days; longer walltime has to be explicitly requested. The smaller the maximal walltime, the higher the priority of the queue, e.g. a job in a medium queue will start earlier than a similar job in the sleep queue.
For each research group there are group limits for the number of cores and the amount of memory that can be used by all private queues together. A job in a private queue can run only if there are free resources available to your group.
There are special private queues (heavyio) that allow usage of a specific hardware and only certain groups are allowed to use them.
Some users have personal limits on private queues, according to a policy in their groups. In order to see your personal limit, type:
qmgr -c 'p s' | grep u:$USER
The limits are “soft” and “hard”. The user is not allowed to run more jobs than are allowed by the hard limit. Please note that jobs running above the soft limit can be suspended in any queue.
For the group limits:
qmgr -c 'p s' | grep g:`id -gn`
Public queues allow using public resources or private resources that belong to other groups. They usually allow relatively short maximal walltime. Alternatively jobs in such queues may be suspended.
- p72 – public queue open for parallel jobs only. The minimal number of cores is 8. From the queue name it is clear that the maximal walltime is 72 hours.
- gpuq – dedicated to use nodes with GPU. Requires setting of ngpus resource. The queue is limited to 5 days.
- highmemq – dedicated to run on the nodes with large amount of memory. The queue is limited to 5 days.
- short – it is really short, 30 minutes, but very high priority. This queue is dedicated for testing.
- idle – practically unlimited but zero priority queue. This queue allows running jobs above the group CPU allocation on a free space basis; however, the jobs submitted to this queue may be suspended.
To see the full list of queues, type:
qstat -Q | grep -v aws | awk '{print $1}'
To see full specifications of a queue, type:
qstat -Qf [queue name]
Note, that you can submit the jobs to the routing queues only. The names of the execution queues end with "x".