Sycamore Cluster SLURM examples
This document will provide you with the basic information needed to run various types of jobs on Sycamore. Before going through this document we suggest you first look at our SLURM guide to get a basic overview of SLURM.
Table of Contents
Submitting a MPI job using mpirun
Submitting an interactive bash session job
Submitting a MPI job using mpirun
Create a SLURM script called example.sl using a text editor. Enter the following into your example.sl script:
#!/bin/bash
#SBATCH -n 384
#SBATCH --mem-per-cpu=200M
#SBATCH -t 00:30:00
module purge
module add openmpi_5.0.5/gcc_11.4.1
mpirun my_parallel_MPI_job
Submit your job using the sbatch command:
sbatch example.sl
example.sl is our SLURM job submission script being used to run the MPI executable my_parallel_MPI_job (which needs to be on your PATH) on Sycamore.
So we have created a script example.sl that will launch a 384-way MPI job therefore using 384 CPUs in total (-n 384), which can run for up to 30 minutes(-t 00:30:00), and where each MPI process has a 200M memory limit (--mem-per-cpu=200M). When submitted, the job will run the MPI executable my_parallel_job using the mpirun command. Since the job allocates more than 47 CPUs we'll need to use the batch partition, but since batch is the default partition for all Sycamore jobs it does not need to be specified here.
Note that within the scope of the job, we cleared out any existing modules that may have been in our Sycamore environment using the module purge command and then loaded the same MPI module (openmpi_5.0.5/gcc_11.4.1 in this example) that was used to compile our MPI executable.
You can also submit your job directly from the commandline without using a script. The commands would look like the following:
$ module purge
$ module add openmpi_5.0.5/gcc_11.4.1
$ sbatch -n 384 --mem-per-cpu=200M -t 00:30:00 --wrap="mpirun my_parallel_MPI_job"
Submitting non-MPI jobs
In this section we'll disuss submitting three types of non-MPI jobs:
- serial job (i.e., single-CPU)
- multi-core job (i.e., multiple CPUs on the same host)
- distributed parallel job (i.e., multiple CPUs spanning different hosts)
Serial job:
Create a SLURM script called example.sl using a text editor. Enter the following into your example.sl script:
#!/bin/bash
#SBATCH -p small
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=5g
#SBATCH -t 1-
# put module commands here
# module purge
# etc.
my_serial_job
Submit your job using the sbatch command:
sbatch example.sl
example.sl is our SLURM job submission script being used to run the executable my_serial_job (which needs to be on your PATH) on Sycamore.
A serial job uses only one CPU. So here we have created a script example.sl that launches one task (-n 1) and allocates one cpu for the task (--cpus-per-task=1) that can run for up to one day (-t 1) in the small partition (-p small) with a 5 GB memory limit (--mem=5g). Since the job allocates fewer than 48 CPUs we'll need to use the small partition. Note you can optionally add any module commands after the SBATCH directives.
You can also submit this job directly from the commandline without using a script. The command would look like the following:
$ sbatch -p small -n 1 --cpus-per-task=1 --mem=5g -t 1- --wrap="my_serial_job"
Multi-core job:
Create a SLURM script called example.sl using a text editor. Enter the following into your example.sl script:
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --cpus-per-task=90
#SBATCH --mem=45g
#SBATCH -t 01:45:00
# put module commands here
# module purge
# etc.
my_multicore_job
Submit your job using the sbatch command:
sbatch example.sl
example.sl is our SLURM job submission script being used to run the executable my_multicore_job (which needs to be on your PATH) on Sycamore.
A multi-core job uses multiple CPUs but only on a single host. So here we have created a script example.sl that requests a single host/node (-N 1) to launch one task (-n 1) that uses 90 CPUs (--cpus-per-task=90) and which can run for up to one hour and forty-five minutes (-t 01:45:00) with a 45 GB memory limit (--mem=45g).
You can also submit this job directly from the commandline without using a script. The command would look like the following:
$ sbatch -N 1 -n 1 --cpus-per-task=90 --mem=45g -t 01:45:00 --wrap="my_multicore_job"
Distributed parallel job:
Create a SLURM script called example.sl using a text editor. Enter the following into your example.sl script:
#!/bin/bash
#SBATCH -n 384
#SBATCH --mem=400g
#SBATCH -t 2-
# put module commands here
# module purge
# etc.
my_parallel_job
Submit your job using the sbatch command:
sbatch example.sl
example.sl is our SLURM job submission script being used to run the executable my_parallel_job (which needs to be on your PATH) on Sycamore.
A distributed parallel job can use multiple CPUs across many hosts. So here we have created a script example.sl that can use 384 CPUs (-n 384) and can run for up to two days (-t 2) with a 400 GB memory limit (--mem=400g). Note that since the job can use CPUs across hosts we do not include the SBATCH directive -N 1 to restrict the task to a single node.
You can also submit this job directly from the commandline without using a script. The command would look like the following:
$ sbatch -n 384 --mem=400g -t 2- --wrap="my_parallel_job"
The executables used in the examples above (i.e., my_serial_job, my_multicore_job, my_parallel_job) can be binaries you've compiled or commonly used applications such as R, Python, MATLAB, etc. The following examples show what the sequence of module and executable commands would look like for some commonly used applications.
1. R
Here we load the r/4.4.2 module in order to use R on Sycamore and then submit our R script that is called my_code.R:
module purge
module add r/4.4.2
R CMD BATCH --no-save my_code.R
2. Python
Here we load the python/3.13 module in order to use Python on Sycamore and then submit our Python script that is called my_code.py:
module purge
module add python/3.13
python my_code.py
3. MATLAB
Here we load the matlab/2024b module in order to use MATLAB on Sycamore and then submit our MATLAB script that is called my_code.m:
module purge
module add matlab/2024b
matlab -singleCompThread -nosplash -nodesktop -r my_code
Submitting an interactive bash session job
A bash session can be useful if you need to work interactively for debugging code, profiling, plotting/visualization, etc. Such sessions on Sycamore should be submitted to the inter partition.
srun -t 5:00:00 -p inter -n 1 --cpus-per-task=1 --mem=8g --x11=first --pty /bin/bash
This srun command will start a single-task (-n 1) single CPU (--cpus-per-task=1) bash shell session in the inter partition (-p inter) with a five hour time limit (-t 5:00:00) and a 8 GB memory limit (--mem=8g).
Note. If you encounter the following X11 forwarding error srun: error: No DISPLAY variable set, cannot setup x11 forwarding you need to enable X11 forwarding in your ssh login session. One way to do this is to use the ssh options -X or-Y or (-XY) in your ssh command when logging into Sycamore.
Using SLURM features
On Sycamore, we can also request access to a specific CPU type or feature using the sbatch options -C (or --constraint) and/or --prefer. Specifying a constraint is optional and jobs submitted without one will run on any available node. The -C option is a hard requirement whereas --prefer is a soft requirement. The following examples build on the job submission script in the first example above to show how you can incorporate feature requests into your job submission script to request a specific CPU model and/or the IB NDR Connection.
1. Hard Requirement for Epyc 9684x Processor
#!/bin/bash
#SBATCH -C 9684x
#SBATCH -n 384
#SBATCH --mem-per-cpu=200M
#SBATCH -t 00:30:00
module purge
module add openmpi_5.0.5/gcc_11.4.1
mpirun my_parallel_MPI_job
2. Soft Requirement for Epyc 9684x Processor
#!/bin/bash
#SBATCH --prefer 9684x
#SBATCH -n 384
#SBATCH --mem-per-cpu=200M
#SBATCH -t 00:30:00
module purge
module add openmpi_5.0.5/gcc_11.4.1
mpirun my_parallel_MPI_job
3. Hard Requirement for IB NDR Connection with soft requirement for Epyc 9684x CPU
#!/bin/bash
#SBATCH -C ndr
#SBATCH --prefer 9684x
#SBATCH -n 384
#SBATCH --mem-per-cpu=200M
#SBATCH -t 00:30:00
module purge
module add openmpi_5.0.5/gcc_11.4.1
mpirun my_parallel_MPI_job
Note by default the IB NDR Connection is used for Sycamore jobs so it's not typically necessary to specify that as a hard requirement.
Last Update 3/31/2025 5:58:16 AM