Body
What is the CRUG Cluster?
One of the goals of Carleton’s Computational Research Users Group is to create a shared, powerful, and expandable computation cluster that is usable by as many of our users as possible and funded by grants and faculty start-up funds. Our users include faculty and students from all departments, and their needs are diverse.
To meet these needs in 2018, the college used NSF grant money to purchase a large computer (56 cores, 768 GB ram, and 19 TB Disk). The system is setup as a Proxmox Virtual Environment server, so that we can spin up individual, one-of-a-kind servers when needed. In practice, as much of the system as possible is dedicated to running a Slurm server and compute nodes. Slurm is a workload manager that allows users to submit jobs to a queue, and those jobs are then sent off to be run on compute nodes. Proxmox gives us the flexibility of running unique linux or windows virtual machines, and Slurm gives us the ability to add additional computation nodes when additional funding is found. Note that while all the cores and ram are available, the majority of the disk space is set aside for redundancy. While we can spin up a windows virtual machine, that hasn't been done since 2018. Whenever possible, the cluster runs on a linux operating system.
In 2019, ITS allocated some of the unused cpu and ram on their servers to CRUG use. We currently have 60 cores and 434 GB of ram in use on ITS servers.
In 2020, Aaron Bauer, using his start-up funds, purchased 50TB of of disk space to be shared with all CRUG users. He also purchased 50 TB of disk space to be used as a back up location.
In 2023, a second computer was added as a dedicated CRUG computer. This machine has 40 cores and 256GB of ram.
In 2024, CRUG purchased a server with two NVIDIA Ampere A30, 24GB GPUs. This machine has 24 cores and 128GB of ram.
Why do we need such a system?
Historically, our faculty have purchased research systems with funding from grants or start-up funds. These systems were “siloed”, meaning that the systems were used only by one faculty member and possibly their students and were off limits to the rest of the campus. Many of these systems were idle for much of the year, which seems wasteful when you consider that users without grants or start-up funding didn’t have access to large computational systems.
How do we fund and maintain such as system now and into the future?
Our answer is to leverage grant funding as much as possible and, when appropriate, to use start-up funding of new hires that have large computational needs. Faculty with funds and who are interested in the shared system are asked to test out the current system; if it meets their needs, they have the option to “buy into it.” These faculty work with our technical staff (Mike Tie, Bruce Duffy, and Randy Hoffner) to purchase new hardware in a way that expands or updates the current system while simultaneously maximizing the benefits to the new faculty member. Faculty who have “bought into the system” will have priority use on the hardware that they have purchased. For example, a faculty member who purchases a new compute node will be able to reserve that node when they need it; this guarantees that they have the resources they need to do their research when they need it (which is what the grant or start-up funds are intended for). Other users will have access to all idle processor time on nodes that haven't been reserved.
How do I use the system?
The system is designed to take advantage of the Slurm workload manager. We expect that the majority of the jobs will take advantage of multiple cores through some type of parallel processing. 95% of our needs seem to be embarrassingly parallel. Users “ssh” into shiver.crug.carleton.edu and submit Slurm jobs through the linux command line.
https://stolafcarleton.teamdynamix.com/TDClient/3356/Portal/KB/ArticleDet?ID=155691 describes the basics of using Slurm to run jobs on the CRUG cluster.
In general to run code on the cluster, you would
Use an sftp or scp client to upload your code and data to shiver.crug.carleton.edu (If you are are looking for a graphical interface for these commands, we recommend winscp for Windows users and cyberduck for Mac users or you can log in into the RStudio server running on shiver.crug.carleton.edu and use the RStudio upload file feature to transfer files.)
If your code requires an executable (C, C++, java), then compile your code on shiver.crug.carleton.edu.
Submit your code to the compute nodes using a command such as srun or sbatch. For example, you would use this command to run a program named helloworld on the FourHour partition:
srun -p FourHour helloworld
Note: shiver.crug.carleton.edu and all of the compute nodes share the same network based file system so your files are visible on all of the compute nodes.
Note: running commands on shiver.crug.carleton.edu does eat up cpu time. please limit your cpu usage on shiver; run your code on the compute nodes via slurm!
If you want a CRUG account or are new to this, please contact Mike Tie for help.
What if I have a job that only needs one core or if I want to run a graphical application?
Yes, we support this too. You can run graphical jobs by X-Forwarding them to your desktop. Mathematica and Rstudio graphical interfaces are good examples of this. However, please limit your core usage for these types of jobs to one or two cores. If you need more than two cores, then you need to submit your jobs through Slurm, so that the load gets distributed to the compute nodes. If that isn’t an option, you can reserve a compute node via Slurm, and then use the compute node to run your graphical app; please contact our technical staff for help.
Who gets access to the system?
Faculty and student researchers working with faculty have priority. There are three compute nodes available for students needing larger computational devices for course work - specifically comps.
Slurm supports "partitions". We have split our cluster into multiple partitions each with a time limit for the jobs running on them, and split up into machines purchased by startup funded computers (the facultynode and Preemptable partitions) and grant/college funded machines (FourHour, TwoDay, SevenDay and StudentSlurm partitions).
FourHour, TwoDay, SevenDay, and Preemptable are the partitions available to faculty and their student researchers. StudentSlurm is the partition used by students working on comps projects, but if the nodes in that partition are idle researchers can also run jobs there.
The facultynode partition is made up of machines purchased with faculty start-up funds. Unless you have purchased a machine, you should not be queuing directly to machines in this partition. These nodes are reserved for the faculty who purchased them. These faculty often move their devices into the Preemptable partition when they aren't heavily using their machine.
The Preemptable partition is made of up machines purchased with faculty start-up funds. If faculty member knows that they aren't going to be heavily using their machine, they reach out to Mike Tie, and he will add it to the Preemptable partition. Anyone can queue jobs to the Preemptable partition, but the purchaser has priority over the machine that they purchased. The purchaser's jobs have the highest priority on their machine. This means that jobs running on the machine will be terminated if the purchaser decides to run a job; the terminated jobs will completely restart after the purchaser's jobs finish. Faculty typically do not put their machines into the Preemptable partition unless they aren't expecting to be using it.
Is there a time limit for jobs?
Slurm supports "partitions". We have split our cluster into multiple partitions each with a time limit for the jobs running on them.
The SevenDay and StudentSlurm partitions will allow jobs to run for up to seven days before terminating them.
The TwoDay partition will allow jobs to run for up to two days before terminating them.
The FourHour partition will allow jobs to run for up to four hours before terminating them.
Please test your code on the FourHour partition before running jobs in the longer running partitions.
If you need to run jobs for more than seven days, then you need to write your code so that it saves it's state; so that you can re-queue and your code will pick up where it left off. If you purchased a machine, you can run jobs for as long as you want on your node.
What if my program needs more cores, memory, disk space, or gpus than the cluster currently has?
If you need more of any of the above, please find funding! We will happily add the new resources, and you will have priority use on the resources that you add to the system.
Is my data backed up?
Yes, but we can always use more funding for new disk space and newer back up space.
Is there regularly scheduled down time?
Yes; we reserve the last Monday of every month to patch and reboot all of the systems; it is critically important that we keep the systems patched.
What compute nodes are in the cluster?
| Partition |
Computer Node |
# of Cores |
RAM |
GPU |
| FourHour |
shark4 |
12 |
58 |
|
| FourHour |
shark13 |
12 |
93 |
|
| Preemptable |
shark4 |
12 |
58 |
|
| SevenDay |
shark1 |
12 |
58 |
|
| SevenDay |
shark2 |
12 |
58 |
|
| SevenDay |
shark10 |
16 |
125 |
|
| SevenDay |
shark11 |
16 |
125 |
|
| SevenDay |
shark12 |
16 |
125 |
|
| StudentSlurm |
shark5 |
12 |
93 |
|
| StudentSlurm |
shark6 |
12 |
62 |
|
| StudentSlurm |
shark9 |
12 |
93 |
|
| TwoDay |
shark3 |
12 |
58 |
|
| TwoDay |
shark7 |
12 |
93 |
|
| TwoDay |
shark8 |
12 |
93 |
|
| facultynode |
ada |
28 |
377 |
|
| facultynode |
borg |
32 |
375 |
|
| facultynode |
curie2 |
32 |
295 |
|
| facultynode |
edmonstone2024 |
32 |
295 |
|
| facultynode |
margulis2024 |
32 |
295 |
|
| GPU |
goblin1 |
12 |
48 |
NVIDIA A30 |
| GPU |
goblin2 |
12 |
48 |
NVIDIA A30 |
How do I reserve a system that I funded and have priority access on?
Talk with Mike Tie. He can remove your node to and from the facultynode and Preemptable partitions. Your jobs will always have the highest priority on your node.
How would I get additional disk space?
You will work with Mike Tie and ITS to purchase the extra space. You need to purchase drives 5 at a time which improves access speeds and adds some redundancy. Ideally you would purchase a second set of disks which would be used to store backs of the first set of disks.
GPUs:
In 2024, CRUG purchased a server with two NVIDIA Ampere A30, 24GB GPUs.
What if I want to run a Windows application, some other version of Linux, or Mac OSX?
We can spin/set up small virtual machines for unique versions of Linux or for Windows apps; please contact one of our technical staff. Unfortunately, Mac OSX is not supported in a virtual environment. While running a windows compute node is technically an option, we rarely do this. Historically all of our systems have been running linux.
How do I get software installed?
Speak with our technical staff.
Other questions?
Speak with Mike Tie.