Slurm

Revision as of 11:29, 29 September 2021

The Slurm job scheduler on the High Performance WiWi Cluster (HPC3)

1 Introduction

2 Cluster topology & hardware specs

The cluster is currently made up of 9 nodes:

• 3 x HP DL385
• 6 x HP ProLiant XL170r (accommodated in an HP Apollo r2200 chassis)

The servers' CPU and memory resources can briefly be summarized as follows:

Although all 9 nodes could serve as compute nodes, one of the DL 385 machines currently serves as a login node only. Due to the fact that the compute nodes are heterogeneous, they are grouped into so-called partitions according to the terminology of slurm. Furthermore, some of the nodes are "private", meaning that particular working groups have exclusive access to them as soon as they submit jobs. Whenever a private node is idle, users from other working groups also may use them for computational purposes. However, as soon as a high-priority job arrives, any running low-priority job on these machines are cancelled (re-queued). Details on how the access control is implemented on the HPC3 WiWi cluster are given in the next section.

The following table gives an overview of the nodes and the partitions they belong to:

Partition defpart is the default partition, gpu and apollo encompass the corresponding group of server nodes. Partitions gpucu, gpukr and apollokr are single-node partitions for access to private nodes of working groups ag_cuchiero and ag_krivobokova. Nodes n01-n05 are grouped into a partition apollo_nonreserved for dedicated access to Apollo nodes that are not subject to reserved resources.

3 Access control & quality of service (QoS)

Each system user (see HPC) is assigned a corresponding slurm user. Access control and queue management is based on slurm accounts which directly correspond to working groups. Each account is entitled to a particular set of quality of service (QoS) levels. On the HPC3 WiWi cluster, QoS mechanisms are used to implement the specific kind of resource reservation that was desired by the working groups that own private nodes. In fact, high-priority access to those nodes is achieved by what is called preemption in slurm. Whenever jobs with a privileged QoS level enter the queue, jobs with a standard QoS level that are running on the associated private node(s) are cancelled and re-queued.

The following table gives an overview of the working groups and the QoS levels they are allowed to select:

Note that privileged QoS levels can only be selected

1. by users that have the permission to use them (based on their working group affiliation), and
2. on partitions that contain private nodes only.

For example, using QoS level agcu on partition apollo is not permitted because high-priority access (with occasional preemption) should only take effect on partition gpucu (node gpu01).

4 Submitting jobs

4.1 Batch jobs

Slurm provides support for unattended execution of jobs on the cluster's resources, which is perhaps the most common way of using it (batch mode). For this purpose, a shell script is passed to the job scheduler, containing

• the commands to be executed and
• some extra information for the slurm job scheduler (optional).

Let us take a closer look at how to create such a script. We start with the first line, telling the OS which kind of UNIX shell to use for interpreting the commands in the script.

#!/bin/bash

Then we add a series of directives for the slurm job scheduler, each starting with a '#SBATCH'. Although the '#' character usually indicates a comment, this specific string gets interpreted by slurm and allows to set various options.

#SBATCH --mail-type=BEGIN,END
#SBATCH --mail-user=john.doe@univie.ac.at

For the moment, we only state an e-mail address here and an indication which events trigger a notification via mail. In this case, we receive an e-mail when the job has been started, that is, when it is removed from the queue of waiting jobs and actually allocates resources on the cluster.

Finally, we add commands to be executed for actual computation purposes. Let us assume in the following that the program we would like to run is called do-something, allowing single- or multi-threaded execution. Assume further that threading can be controlled by a command line parameter --threads. If we wanted to use all 16 or 32 processors of a standard allocation (1 socket), then the program could be run either by

do-something --threads 16

or by parallelizing single-threaded instances of itself:

do-something --threads 1 &
do-something --threads 1 &
...
do-something --threads 1 &

Note that the '&' character at the end of each line tells the shell to run the program in background mode. The second mode of execution is useful, for example, when each instance of do-something takes a different file as an input.

When saving the script to the disk as a file, for example job-script.sh, we can run it using the sbatch command:

sbatch -J Job1 job-script.sh

The command takes the contents of the file job-script.sh, and tries to allocate resources on the cluster. If there are enough resources available (at least one socket) then the job is started on the corresponding node. Otherwise the job is held in the queue. To keep track of one's jobs, an identifier (job name) can be assigned to a submitted job by using the parameter '-J', as shown above.

An overview of queued and running jobs can be obtained by the command

squeue

The output might look as follows:

 JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  224    apollo     Job7 ag_do-br PD       0:00      1 (Resources)
  218    apollo     Job1 ag_do-br  R       0:29      1 n01
  219    apollo     Job2 ag_do-br  R       0:29      1 n02
  220    apollo     Job3 ag_do-br  R       0:29      1 n03
  221    apollo     Job4 ag_do-br  R       0:29      1 n04
  222    apollo     Job5 ag_do-br  R       0:29      1 n05
  223    apollo     Job6 ag_do-br  R       0:29      1 n06

In this case, 6 jobs are running on the 'apollo' partition, each allocating a whole node, i.e., two sockets. Job #7 is currently held in the queue because the partition is fully occupied. This is indicated by the field 'ST' (state), telling us that the job is currently pending (PD). Jobs #1 - #6 are in state running. The last column in this table shows the nodes on which each of the listed jobs is running.

To specify the partition on which a job should run, we can use the option '-p'. For partition 'apollo', this would be

sbatch -p apollo -J Job1 job-script.sh

If no partition is stated in the sbatch command line, the default partition (all compute nodes) is assumed as a target.

The quality of service (QoS) to use can be specified by the option '-q', for example

sbatch -q agcu job-script.sh

Again, the default QoS ('normal') is used if none is provided. Note that privileged QoS specifiers are accepted only

• for users which are entitled to them
• for partitions on which they are admitted (see also #Access control & quality of service (QoS)).

To avoid long chains of command line arguments, one can pass most of the parameters to sbatch via directives in the job script, as they were already introduced above in the context of notification e-mails. For example,

#SBATCH --partition=apollo
#SBATCH --qos=normal

lead to the same result as the command line arguments '-p apollo' and '-q normal'.

4.2 Interactive jobs

5 Advanced topics

5.1 Adjusting the resource allocation

5.2 Job steps

Contents will follow...