Slurm

The Slurm job scheduler on the High Performance WiWi Cluster (HPC3)

2 Cluster topology & hardware specs

The cluster is currently made up of 9 nodes that are interconnected via a 10Gbit (Ethernet) network:

• 3 x HP DL385
• 6 x HP ProLiant XL170r (accommodated in an HP Apollo r2200 chassis)

The servers' CPU and memory resources can briefly be summarized as follows:

Server	CPU	Clock speed	Sockets	Cores/socket	Memory	GPU ready
DL385	AMD Epyc 7452	2.35 GHz (max. 3.35 GHz)	2	32	256 GB	yes
ProLiant XL170r	Xeon-G 6226R	2.9 GHz (max. 3.9 GHz)	2	16	384 GB	no

Although all 9 nodes could serve as compute nodes, one of the DL385 machines currently serves as a login node only. Due to the fact that the compute nodes are heterogeneous, they are grouped into so-called partitions according to the terminology of slurm. Furthermore, some of the nodes are "private", meaning that particular working groups have exclusive access to them as soon as they submit jobs. Whenever a private node is idle, users from other working groups also may use them for computational purposes. However, as soon as a high-priority job arrives, any running low-priority job on these machines are cancelled (re-queued). Details on how the access control is implemented on the HPC3 WiWi cluster are given in the next section.

The following table gives an overview of the nodes and the partitions they belong to:

Node name	Role	Partitions	"Private"
hpc3	login, control		no
gpu01	compute, GPU	defpart, gpu, gpucu	yes
gpu02	compute, GPU	defpart, gpu, gpukr	yes
n01-n05	compute	defpart, apollo, apollo_nonreserved	no
n06	compute	defpart, apollo, apollokr	yes

Partition defpart is the default partition, gpu and apollo encompass the corresponding group of server nodes. Partitions gpucu, gpukr and apollokr are single-node partitions for access to private nodes of working groups ag_cuchiero and ag_krivobokova. Nodes n01-n05 are grouped into a partition apollo_nonreserved for dedicated access to Apollo nodes that are not subject to reserved resources.

3 Access control & quality of service (QoS)

Each system user (see HPC) is assigned a corresponding slurm user. Access control and queue management is based on slurm accounts which directly correspond to working groups. Each account is entitled to a particular set of quality of service (QoS) levels. On the HPC3 WiWi cluster, QoS mechanisms are used to implement the specific kind of resource reservation that was desired by the working groups that own private nodes. In fact, high-priority access to those nodes is achieved by what is called preemption in slurm. Whenever jobs with a privileged QoS level enter the queue, jobs with a standard QoS level that are running on the associated private node(s) are cancelled and re-queued.

The following table gives an overview of the working groups and the QoS levels they are allowed to select:

Working group	QoS levels	Preemption on partition(s)
ag_cuchiero	normal, agcu	gpucu
ag_krivobokova	normal, agkr	gpukr, apollokr
ag_doerner	normal
ag_ehmke	normal
ag_hautsch	normal
ag_operres	normal

Note that privileged QoS levels can only be selected

1. by users that have the permission to use them (based on their working group affiliation), and
2. on partitions that contain private nodes only.

For example, using QoS level agcu on partition apollo is not permitted because high-priority access (with occasional preemption) should only take effect on partition gpucu (node gpu01).

4 Submitting jobs

4.1 Batch jobs

Slurm provides support for unattended execution of jobs on the cluster's resources, which is perhaps the most common way of using it (batch mode). For this purpose, a shell script is passed to the job scheduler, containing

• the commands to be executed and
• some extra information for the slurm job scheduler (optional).

Let us take a closer look at how to create such a script. We start with the first line, telling the OS which kind of UNIX shell to use for interpreting the commands in the script.

#!/bin/bash

Then we add a series of directives for the slurm job scheduler, each starting with a '#SBATCH'. Although the '#' character usually indicates a comment, this specific string gets interpreted by slurm and allows to set various options.

#SBATCH --mail-type=BEGIN,END
#SBATCH --mail-user=john.doe@univie.ac.at

For the moment, we only state an e-mail address here and an indication which events trigger a notification via mail. In this case, we receive an e-mail when the job has been started, that is, when it is removed from the queue of waiting jobs and actually allocates resources on the cluster.

Finally, we add commands to be executed for actual computation purposes. Let us assume in the following that the program we would like to run is called do-something, allowing single- or multi-threaded execution. Assume further that threading can be controlled by a command line parameter --threads. If we wanted to use all 16 or 32 processors of a standard allocation (1 socket), then the program could be run either by

do-something --threads 16

or by parallelizing single-threaded instances of itself:

do-something --threads 1 &
do-something --threads 1 &
...
do-something --threads 1 &

Note that the '&' character at the end of each line tells the shell to run the program in background mode. The second mode of execution is useful, for example, when each instance of do-something takes a different file as an input.

When saving the script to the disk as a file, for example job-script.sh, we can run it using the sbatch command:

sbatch -J Job1 job-script.sh

The command takes the contents of the file job-script.sh, and tries to allocate resources on the cluster. If there are enough resources available (at least one socket) then the job is started on the corresponding node. Otherwise the job is held in the queue. To keep track of one's jobs, an identifier (job name) can be assigned to a submitted job by using the parameter '-J', as shown above.

An overview of queued and running jobs can be obtained by the command

squeue

The output might look as follows:

 JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  224    apollo     Job7 ag_do-br PD       0:00      1 (Resources)
  218    apollo     Job1 ag_do-br  R       0:29      1 n01
  219    apollo     Job2 ag_do-br  R       0:29      1 n02
  220    apollo     Job3 ag_do-br  R       0:29      1 n03
  221    apollo     Job4 ag_do-br  R       0:29      1 n04
  222    apollo     Job5 ag_do-br  R       0:29      1 n05
  223    apollo     Job6 ag_do-br  R       0:29      1 n06

In this case, 6 jobs are running on the 'apollo' partition, each allocating a whole node, i.e., two sockets. Job #7 is currently held in the queue because the partition is fully occupied. This is indicated by the field 'ST' (state), telling us that the job is currently pending (PD). Jobs #1 - #6 are in state running. The last column in this table shows the nodes on which each of the listed jobs is running.

To specify the partition on which a job should run, we can use the option '-p'. For partition 'apollo', this would be

sbatch -p apollo -J Job1 job-script.sh

If no partition is stated in the sbatch command line, the default partition (all compute nodes) is assumed as a target.

The quality of service (QoS) to use can be specified by the option '-q', for example

sbatch -q agcu job-script.sh

Again, the default QoS ('normal') is used if none is provided. Note that privileged QoS specifiers are accepted only

• for users which are entitled to them
• for partitions on which they are admitted (see also #Access control & quality of service (QoS)).

To avoid long chains of command line arguments, one can pass most of the parameters to sbatch via directives in the job script, as they were already introduced above in the context of notification e-mails. For example,

#SBATCH --partition=apollo
#SBATCH --qos=normal

lead to the same result as the command line arguments '-p apollo' and '-q normal'.

4.2 Interactive jobs

Besides the batch-based job submission, it is also possible to run interactive jobs on the cluster. In essence, this means that as soon as resources are available, a shell prompt appears, allowing to run any application or script on the node(s) that have just been allocated. Relying on the defaults, this can be achieved by the following command:

srun --pty /bin/bash

Note that the option '--pty' is important, because otherwise the srun command spawns multiple instances of /bin/bash across the allocation (all the cores of a socket). This is unwanted behavior in the interactive shell context, because all inputs & outputs would appear multiple times. Of course, any UNIX shell can be used instead of '/bin/bash'.

5 File system access

The users' home directories (/home/<username>/) are transparently available on all compute nodes via NFS. Consequently, any apps that are installed in a user's home directory can be run on any node. This is also the recommended way of proceeding when custom software is needed (instead of a system-wide installation).

Please be aware that due to the NFS-based mount all file operations also have to pass the network layer. I/O-intensive tasks with thousands of read or write operations per second can therefore be subject to significant slow-downs compared to native file system access.

In such scenarios, it is advisable to use the local scratch space provided on each node, accessible via /scratch.local. Note that both scratch directories, /scratch.global and /scratch.local are world-writable and thus large amounts of data might pile up over time. For this reason, files that are older than 30 days will be automatically removed.

6 Advanced topics

6.1 Adjusting the resource allocation

The default allocation on the WiWi cluster is one socket with a varying number of CPU cores, depending on the node type. Of course, it is possible to allocate more than a socket for a job. This can be done by explicitly providing the number of tasks to the slurm commands sbatch, srun and salloc via the option '-n' or '--ntasks'. The slurm subsystem on the WiWi cluster is configured to use a 1:1 mapping between tasks and CPU cores for resource allocation. This means that

sbatch -n 32 job-script.sh

would allocate two sockets on the Apollo nodes and one socket on the DL385 (GPU) nodes.

6.2 Job steps

Contents will follow...