Slurm

Revision as of 14:00, 23 September 2021

The Slurm job scheduler on the High Performance WiWi Cluster (HPC3)

1 Introduction

2 Cluster topology & hardware specs

3 Submitting jobs

3.1 Batch jobs

Slurm provides support for unattended execution of jobs on the cluster's resources , which is perhaps the most common way of using it (batch mode). For this purpose, a shell script is passed to the job scheduler, containing

• the commands to be executed and
• some extra information for the slurm job scheduler (optional).

Let us take a closer look at how to create such a script. We start with the first line, telling the OS which kind of UNIX shell to use for interpreting the commands in the script.

#!/bin/bash

Then we add a series of directives for the slurm job scheduler, each starting with a '#SBATCH'. Although the '#' character usually indicates a comment, this specific string gets interpreted by slurm and allows to set various options.

#SBATCH --mail-type=BEGIN,END
#SBATCH --mail-user=john.doe@univie.ac.at

For the moment, we only state an e-mail address here and an indication which events trigger a notification via mail. In this case, we receive an e-mail when the job has been started, that is, when it is removed from the queue of waiting jobs and actually allocates resources on the cluster.

Finally, we add commands to be executed for actual computation purposes. Let us assume in the following that the program we would like to run is called do-something, allowing single- or multi-threaded execution. Assume further that threading can be controlled by a command line parameter --threads. If we wanted to use all 16 or 32 processors of a standard allocation (1 socket), then the program could be run either by

do-something --threads 16

or by parallelizing single-threaded instances of itself:

do-something --threads 1 &
do-something --threads 1 &
...
do-something --threads 1 &

Note that the '&' character at the end of each line tells the shell to run the program in background mode. The second mode of execution is useful, for example, when each instance of do-something takes a different file as an input.

When saving the script to the disk as a file, for example job-script.sh, we can run it using the sbatch command:

sbatch -J Job1 job-script.sh

The command takes the contents of the file job-script.sh, and tries to allocate resources on the cluster. If there are enough resources available (at least one socket) then the job is started on the corresponding node. Otherwise the job is held in the queue. To keep track of one's jobs, an identifier (job name) can be assigned to a submitted job by using the parameter '-J', as shown above.

An overview of queued and running jobs can be obtained by the command

squeue

The output might look as follows:

 JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  224    apollo     Job7 ag_do-br PD       0:00      1 (Resources)
  218    apollo     Job1 ag_do-br  R       0:29      1 n01
  219    apollo     Job2 ag_do-br  R       0:29      1 n02
  220    apollo     Job3 ag_do-br  R       0:29      1 n03
  221    apollo     Job4 ag_do-br  R       0:29      1 n04
  222    apollo     Job5 ag_do-br  R       0:29      1 n05
  223    apollo     Job6 ag_do-br  R       0:29      1 n06

In this case, 6 jobs are running on the 'apollo' partition, each allocating a whole node, i.e., two sockets. Job #7 is currently held in the queue because the partition is fully occupied. This is indicated by the field 'ST' (state), telling us that the job is currently pending (PD). Jobs #1 - #6 are in state running. The last column in this table shows the nodes on which each of the listed jobs is running.

To specify the partition on which a job should run, we can use the option '-p'. For partition 'apollo', this would be

sbatch -p apollo -J Job1 job-script.sh

If no partition is stated in the sbatch command line, the default partition (all compute nodes) is assumed as a target.

The quality of service (QoS) to use can be specified by the option -q. Again, the default QoS 'normal' is

#SBATCH --partition=apollo
#SBATCH --qos=normal

3.2 Interactive jobs