Longleaf Cluster SLURM examples

This document will provide you with the basic information needed to run various types of jobs on Longleaf. Before going through this document we suggest you first look at our SLURM guide to get a basic overview of SLURM.

Table of Contents

Submitting a serial job

Submitting a multi-core job

Submitting a GPU job

Application commands

Submitting an interactive bash session job

Submitting a serial job

Create a SLURM script called example.sl using a text editor. Enter the following into your example.sl script:

#!/bin/bash

#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=5g
#SBATCH -t 1-

# put module commands here
# module purge
# etc.

my_serial_job

Submit your job using the sbatch command:

sbatch example.sl

example.sl is our SLURM job submission script being used to run the executable my_serial_job (which needs to be on your PATH) on Longleaf.

A serial job uses only one CPU. So here we have created a script example.sl that launches one task (-n 1) and allocates one cpu for the task (--cpus-per-task=1) that can run for up to one day (-t 1) with a 5 GB memory limit (--mem=5g). Since we don't specify a SLURM partition in our sbatch options, this job will run in the general partition which is appropriate in this case since the general partition is used for running CPU-only that don't have special requirements. Note you can optionally add any module commands after the SBATCH directives.

You can also submit this job directly from the commandline without using a script. The command would look like the following:

$ sbatch -n 1 --cpus-per-task=1 --mem=5g -t 1- --wrap="my_serial_job"

Submitting a multi-core job

Create a SLURM script called example.sl using a text editor. Enter the following into your example.sl script:

#!/bin/bash

#SBATCH -n 1
#SBATCH --cpus-per-task=90
#SBATCH --mem=45g
#SBATCH -t 01:45:00
#SBATCH --mail-type=begin,end,fail
#SBATCH --mail-user=onyen@email.unc.edu

# put module commands here
# module purge
# etc.

my_multicore_job

Submit your job using the sbatch command:

sbatch example.sl

example.sl is our SLURM job submission script being used to run the executable my_multicore_job (which needs to be on your PATH) on Longleaf.

A multi-core job uses multiple CPUs but only on a single host. So here we have created a script example.sl to launch one task (-n 1) that uses 90 CPUs (--cpus-per-task=90) and which can run for up to one hour and forty-five minutes (-t 01:45:00) with a 45 GB memory limit (--mem=45g). Note by default SLURM on Longleaf is configured to allow only single-node jobs, so there is no need to add the -N 1 SBATCH directive in your job submission script to restrict the number of nodes for your job to 1. We've also added two SBATCH directives to send an email for when the job starts and ends/fails (--mail-type=begin,end,fail) to our email address (--mail-user=onyen@email.unc.edu). Since we don't specify a SLURM partition in our sbatch options, this job will run in the general partition which is appropriate in this case since the general partition is used for running CPU-only that don't have special requirements. Note you can optionally add any module commands after the SBATCH directives.

You can also submit this job directly from the commandline without using a script. The command would look like the following:

$ sbatch -n 1 --cpus-per-task=90 --mem=45g -t 01:45:00 --mail-type=begin,end,fail --mail-user=onyen@email.unc.edu --wrap="my_multicore_job"

Submitting a GPU job

Create a SLURM script called example.sl using a text editor. Enter the following into your example.sl script:

#!/bin/bash

#SBATCH -n 1
#SBATCH --cpus-per-task=12
#SBATCH --mem=20g
#SBATCH -t 2-
#SBATCH -p a100-gpu
#SBATCH --qos=gpu_access
#SBATCH --gres=gpu:1

# put module commands here
# module purge
# etc.

my_gpu_job

Submit your job using the sbatch command:

sbatch example.sl

example.sl is our SLURM job submission script being used to run the executable my_gpu_job (which needs to be on your PATH) on Longleaf.

A GPU job on Longleaf gets access to an entire physical GPU (i.e., all the GPU cores and device memory). Note to use a GPU on Longleaf, the job needs to be submitted to one of the Longleaf GPU partitions. Here we have created a script example.sl to launch one task (-n 1) that allocates 12 CPUs (--cpus-per-task=12), 20 GB of memory (--mem=20g), and can run for up two days (-t 2-) in the a100-gpu partition (-p a100-gpu) which is one of the GPU partitions on Longleaf. You can specify multiple GPU partitions if you don't have a specific GPU model requirement. For example, specifying -p a100-gpu,l40-gpu will make the job eligible to run in either the a100-gpu or l40-gpu partition and therefore your job could end up using either a A100 or L40/L40S GPU. We also need to add a qos SBATCH directive to get access to a GPU (--qos=gpu_access) and specify the number of GPUs our task will use (--gres=gpu:1). Note you can optionally add any module commands after the SBATCH directives.

You can also submit this job directly from the commandline without using a script. The command would look like the following:

$ sbatch -n 1 --cpus-per-task=12 --mem=20g -t 2- -p a100-gpu --qos=gpu_access --gres=gpu:1 --wrap="my_gpu_job"

Application commands

The executables used in the examples above (i.e., my_serial_job, my_multicore_job, my_gpu_job) can be binaries you've compiled or commonly used applications such as R, Python, MATLAB, etc. The following examples show what the sequence of module and executable commands would look like for some commonly used applications.

1. R

There are a few R modules available on Longleaf. Here we load the r/4.4.0 module in order to use R on Longleaf and then submit our R script that is called my_code.R:

module purge
module add r/4.4.0

R CMD BATCH --no-save my_code.R

2. Python

There are a few Python and Anaconda (a Python distribution) modules on Longleaf. Here we load the anaconda/2024.02 module in order to use Python on Longleaf and then submit our Python script that is called my_code.py:

module purge
module add anaconda/2024.02

python my_code.py

3. MATLAB

There are a few MATLAB modules on Longleaf. Here we load the matlab/2024b module in order to use MATLAB on Longleaf and then submit our MATLAB script that is called my_code.m:

module purge
module add matlab/2024b

matlab -singleCompThread -nosplash -nodesktop -r my_code

4. SAS

There are a few SAS modules on Longleaf. Here we load the sas/9.4m8 module in order to use SAS on Longleaf and then submit our SAS script that is called my_code.sas:

module purge
module add sas/9.4m8

sas -noterminal my_code.sas

5. Stata

There are a few Stata modules on Longleaf. Here we load the stata/18 module in order to use Stata on Longleaf and then submit our Stata script that is called my_code.do:

module purge
module add stata/18

stata-se -b my_code.do

If you are instead running multi-core Stata (i.e., Stata/MP) then you'd use the stata-mp binary:

module purge
module add stata/18

stata-mp -b my_code.do

Submitting an interactive bash session job

A bash session can be useful if you need to work interactively for debugging code, profiling, plotting/visualization, etc. Such sessions on Longleaf should be submitted to the interact partition.

srun -t 5:00:00 -p interact -n 1 --cpus-per-task=1 --mem=8g --x11=first --pty /bin/bash

This srun command will start a single-task (-n 1) single CPU (--cpus-per-task=1) bash shell session in the interact partition (-p interact) with a five hour time limit (-t 5:00:00) and a 8 GB memory limit (--mem=8g).

Note. If you encounter the following X11 forwarding error srun: error: No DISPLAY variable set, cannot setup x11 forwarding you need to enable X11 forwarding in your ssh login session. One way to do this is to use the ssh options -X or-Y or (-XY) in your ssh command when logging into Longleaf.

 

Last Update 2/13/2025 10:31:06 AM