Getting Started on Sycamore
Table of Contents
Introduction
The Sycamore cluster is a Linux-based computing system with approximately 15,000 compute cores available to researchers across the campus free of charge. This cluster is optimized to run large-way, distributed memory, multi-node, parallel jobs using a message passing (e.g. MPI) or hybrid (OpenMP + MPI) programming model. Specifically, the system has a fast 400 Gbps low latency high bandwidth network fabric (Infiniband NDR interconnect) for running MPI applications. Sycamore can be used to run non-MPI, serial, and multi-core workloads as well.
System Information
| Model | AMD EPYC 9654 |
|---|---|
| Quantity: | 40 nodes |
| Operating system: | Red Hat Enterprise Linux 9.4 |
| Total cores per node: | 192 cores on two sockets (96 cores/socket) |
| Hardware threads per core: | 1 (Hyperthreading is not currently enabled on Sycamore) |
| Clock rate: | 2.4GHz |
| RAM per node: | 1.5 TB |
| Cache: | 64 KB L1 cache (per core), 1MB L2 cache (per core), 384 MB L3 cache (shared) |
| Local disk: | 203GB /tmp |
| Model | AMD EPYC 9684X |
|---|---|
| Quantity: | 38 nodes |
| Operating system: | Red Hat Enterprise Linux 9.4 |
| Total cores per node: | 192 cores on two sockets (96 cores/socket) |
| Hardware threads per core: | 1 (Hyperthreading is not currently enabled on Sycamore) |
| Clock rate: | 2.55GHz |
| RAM per node: | 1.5 TB |
| Cache: | 64 KB L1 cache (per core), 1MB L2 cache (per core), 1152 MB L3 cache (shared) |
| Local disk: | 203GB /tmp |
Getting an Account
Follow the steps listed on Request a Cluster Account page and select Sycamore Cluster under subscription type. You will receive an email notification once your account has been created.
Logging In
Linux:
Linux users can use ssh from within their Terminal application to connect to Sycamore.
If you wish to enable x11 forwarding use the “-X” ssh option. Be sure to use your UNC ONYEN and password for the login:
ssh <onyen>@sycamore.unc.edu
Windows:
Windows users should download MobaXterm (Home Edition). Then use the Session icon to create a Sycamore SSH session using sycamore.unc.edu for “Remote host” and your ONYEN for the “username” (Port should be left at 22).
Mac:
Mac users can use ssh from within their Terminal application to connect to Sycamore. Be sure to use your UNC ONYEN and password for the login:
ssh <onyen>@sycamore.unc.edu
To enable x11 forwarding Mac users will need to download, install, and run Xquartz on their local machine in addition to using the “-X” ssh option. Furthermore, in many instances for x11 forwarding to work properly Mac users need to use the Terminal application that comes with Xquartz instead of the default Mac terminal application.
A successful login takes you to “login node” resources that have been set aside for user access. The login node is where you will edit your code, execute basic UNIX commands, and submit your jobs from to the SLURM job scheduler.
DO NOT RUN YOUR CODE OR RESEARCH APPLICATIONS DIRECTLY ON THE LOGIN NODE. THESE MUST BE SUBMITTED TO SLURM!
In order to connect to Sycamore from an off-campus location, a connection to the campus network through a VPN client is required.
Main Directory Spaces
1. NAS home directory space
Quota (per user): 50 GB soft limit, 75 GB hard limit.
Backed up: Yes, via snapshots.
Your home directory will be in /nas/longleaf/home/<onyen>.
If your home directory ever becomes full (i.e., reaches the hard quota limit) it can affect your overall ability to use the cluster so you will want to monitor and manage your home directory usage.
We recommend not running any heavy I/O workloads out of your home folder. These type of workloads should be run on either the /users, /work, or /proj file systems discussed below.
Note. The Sycamore and Longleaf clusters share the same home file space. Thus if you are someone who uses both clusters, we encourage creating a longleaf and/or sycamore subdirectory under your home directory to separate out files as needed.
2. /users storage
Quota (per user): 10 TB.
Backed up: No.
Your users directory will be in /users/<o>/<n>/<onyen>. It can be thought of as a capacity expansion to your home directory. This high capacity storage is provided by the same hardware as /proj.
The following rules of thumb should be kept in mind regarding your /users storage:
- OK to compute against, however as IO increases, consider copying or moving data to your /work storage for processing.
- OK for holding inactive data sets like a near-line archive.
- If a meaningful amount of cold data accrues, it can be packaged and MOVED to cloud archive, providing more working space for your warm data.
Note that /users is not intended to be used for team oriented shared storage (instead the /proj file system serves this purpose); it is intended to be your personal storage location.
3. /work storage
Quota (per user): 10 TB.
Backed up: No.
Your /work directory will be in /work/users/<o>/<n>/<onyen>.
/work is built for high-throughput and data-intensive computing, and intended for data that is actively being computed on, accessed, and processed. Please note that work is NOT intended to be a personal storage location. For inactive data, please move it to /users or /proj - contact us if you have no place or insufficient quota to move inactive data off of /work.
File systems such as /work in an academic research setting typically employ a file deletion policy, auto-deleting files of a certain age. At this time, there are no time limits for files on /work. We rely upon the community to maintain standards of appropriate and reasonable use of the various storage tiers.
4. /proj storage
Quota: Varies by need.
Backed up: Not by default.
/proj space is available to PIs (only) upon request and is intended to be used as team oriented shared storage. The amount of /proj space initially given to a PI varies according to their needs. There is no file deletion policy for /proj space, but users should take care in managing the use of their /proj space to stay under assigned quotas.
For further information and to make a request for /proj space please email research@unc.edu.
Applications Environment
The application environment on Sycamore is presented as modules using lmod. Please refer to the Help document on modules for information and examples of using module commands.
Modules are essentially software installations for general use on the cluster. Therefore, you will primarily use module commands to add and remove applications from your Sycamore environment as needed for running jobs. It’s recommended that you keep your module environment as sparse as possible.
Applications used by many groups across campus have already been installed and made available on Sycamore. To see the full list of applications currently available run module avail.
Users are able to install their own applications and create their own modules.
Job Submission
Job submission is handled using SLURM. SLURM is an open source, fault-tolerant, and highly scalable cluster management and job scheduling software used on Linux compute clusters around the world. As a cluster workload manager, SLURM has three main functions:
- orchestrating access to resources to cluster users for some duration of time in order to run code
- providing a framework for starting, executing, and monitoring work on the allocated resources
- managing contention for resources by maintaining a queue of pending jobs
On Sycamore, in order to run your code (i.e., a "job"), you'll need to submit a job request to SLURM to run your code. As part of this job submission request, you'll need to use the appropriate SLURM flags to request resources for your job. Common resources to request for your job will include the number of CPUs, the amount of memory, runtime limit, etc. that your code needs to run properly to completion. Upon submitting your job to SLURM, it immediately goes into a queue where it waits to be dispatched (by SLURM) to run somewhere on the Sycamore cluster. SLURM uses a fair-share algorithm to schedule user jobs on Sycamore.
We first recommend looking at our SLURM guide for an overview of SLURM. After gaining a basic understanding of SLURM, you can refer to this doc for specific examples of how to submit SLURM jobs on Sycamore using the appropriate syntax.
Sycamore Partitions
A SLURM partition is a collection of nodes. When you submit your job it will run in a SLURM partition. You can optionally specify in your job submission which partition your job should use. If you don't specify the partition, your job will run in the default SLURM partition.
Sycamore has the following three SLURM partitions available to users for job submission:
1. batch
- To be used for jobs that require at least 48 CPUs.
- A user can allocate up to a maximum of 5000 CPUs in the batch partition.
- batch is the default partition.
- Maximum job runtime limit is 5 days.
2. lowpri
- To be used for jobs that require at least 48 CPUs.
- A user can allocate up to a maximum of 5000 CPUs in the lowpri partition.
- Maximum job runtime limit is 5 days.
- Has a higher SLURM priority than the batch partition (so jobs may dispatch quicker) but a job in the lowpri partition can be suspended at the discretion of the SLURM administrator in order to run jobs they deem more urgent or of higher priority.
3. small
- To be used for jobs that require 2-47 CPUs.
- A user can allocate up to a maximum of 192 CPUs in the small partition.
- Maximum job runtime limit is 5 days.
4. inter
- To be used for shorter-running interactive jobs as is typical for debugging code, profiling and testing code, interactive plotting and visualization, etc.
- A user can allocate up to a maximum of 96 CPUs in the inter partition.
- Maximum job runtime limit is 8 hours.
Additional Help
For further assistance email research@unc.edu with any questions or open a support ticket.
Last Update 3/31/2025 6:08:19 AM