Script files are “text only” that is don’t use Word and any other word processor to create the files. You can create your script files directly in your home directory on the cluster using: vi, vim, nano etc. Or you can create them on your computer using MacIntosh :TextEdit and Windows: Notepad and then move the file to the cluster via sftp.
Most people’s scripts will probably be straightforward, e.g. not parallel or using multiple nodes.
There are example scripts located on the cluster in /hb/software/scripts
The following is an example of a script file’s content.
| #!/bin/bash | Tells the OS which shell to run this script in |
| #SBATCH –job-name myname | Assigns the name “myname” to the job |
| #SBATCH -p <partition> default is 128×24 | Tells the job to run in the <queue> = name of queue |
| #SBATCH –mail-user=cruzid@ucsc.edu | Email address to send job-related status (may not work) |
| #SBATCH –mail-type=ALL | Specifies which events it should send email about(ALL, BEGIN, END, REQUEUE, FAIL) |
| #SBATCH -o myname.out-%j | Sends the standard output to the named file |
| #SBATCH -e myname.err-%j | Sends the standard error to the named file |
| #SBATCH -n x | Number of CPU cores you are requesting x = 5,10,20,etc |
| #SBATCH –mem=1G | The amount of memory your program requires per node. |
| myprogram | Your program information goes on the following line |
To verify that your script is constructed correctly in terms syntax you can use this web site:
SLURM script example using R:
Important Note: Do not leave any SPACES between #SBATCH directives.
Basic, single-processor job
This script can serve as the template for many single-processor applications. The mem-per-cpu flag can be used to request the appropriate amount of memory for your job. Please make sure to test your application and set this value to a reasonable number based on actual memory use. The %j in the -o (can also use –output) line tells SLURM to substitute the job ID in the name of the output file. You can also add a -e or –error with an error file name to separate output and error logs.
#!/bin/bash #SBATCH --job-name=serial_job_test # Job name #SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL) #SBATCH --mail-user=<email_address> # Where to send mail #SBATCH --ntasks=1 # Run on a single CPU #SBATCH --time=00:05:00 # Time limit hrs:min:sec (you may not want this) #SBATCH --output=serial_test_%j.out # Standard output and error log #SBATCH --mem=250M # Allocated 250 megabytes of memory for the job. module load python/2.7 python myscript.py an example of using R:
#!/bin/bash #SBATCH --job-name=Rtest #SBATCH --output=Rtest.%j.out #SBATCH --error=Rtest.%j.err ################# #time you think you need; leave out for unlimited #format could be dd-hh:mm:ss, hh:mm:ss, mm:ss, or mm #SBATCH --time=5:00 #SBATCH -p 128x44 #partition you want to use #SBATCH --nodes=1 #SBATCH --mail-type=END,FAIL # notifications for job done & fail #SBATCH --mail-user=cruzid@ucsc.edu # note the "CMD BATCH is an R specific command # note module load of gnu is optional (its the default) module load gnu/5.4.0 module load R/3.3.1 R CMD BATCH ./rtest.R
Threaded or multi-processor job
This script can serve as a template for applications that are capable of using multiple processors on a single server or physical computer. These applications are commonly referred to as threaded, OpenMP, PTHREADS, or shared memory applications. While they can use multiple processors, they cannot make use of multiple servers and all the processors must be on the same node.
These applications required shared memory and can only run on one node; as such it is important to remember the following:
- You must set
--nodes=1, and then set--cpus-per-taskto the number of OpenMP threads you wish to use. - You must make the application aware of how many processors to use. How that is done depends on the application:
- For some applications, set OMP_NUM_THREADS to a value less than or equal to the number of cpus-per-task you set.
- For some applications, use a command line option when calling that application.
#!/bin/bash #SBATCH --job-name=parallel_job_test # Job name #SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL) #SBATCH --mail-user=<email_address> # Where to send mail #SBATCH --nodes=1 # Use one node #SBATCH --ntasks=1 # Run a single task #SBATCH --cpus-per-task=4 # Number of CPU cores per task #SBATCH --output=parallel_%j.out # Standard output and error log #SBATCH --mem=1G # 1GB of memory is limit # Note: module load of gnu is optional (its the default) # You may need other modules, however, to run your program. module load gcc/5.4.0 <your program goes here>
MPI job
This script can serve as a template for MPI, or message passing interface, applications. These are applications that can use multiple processors that may, or may not, be on multiple servers.
Our testing has found that it is best to be very specific about how you want your MPI ranks laid out across nodes and even sockets (multi-core CPUs). SLURM and OpenMPI have some conflicting behavior if you leave too much to chance. Please refer to the full SLURM sbatch documentation, but the following directives are the main directives to pay attention to:
-c, --cpus-per-task=<ncpus>- Advise the Slurm controller that ensuing job steps will require ncpus number of processors per task.
-m, --distribution=arbitrary|<block|cyclic|plane=<options>[:block|cyclic|fcyclic]>- Specify alternate distribution methods for remote processes.
- We recommend
-m cyclic:cyclic, which tells SLURM to distribute tasks cyclically over nodes and sockets.
-N, --nodes=<minnodes[-maxnodes]>- Request that a minimum of minnodes nodes be allocated to this job.
-n, --ntasks=<number>- Number of tasks (MPI ranks)
--ntasks-per-node=<ntasks>- Request that ntasks be invoked on each node
--ntasks-per-socket=<ntasks>- Request the maximum ntasks be invoked on each socket
- Notes on socket layout:
- hpg2-compute nodes have 2 sockets, each with 16 cores.
- hpg1-compute nodes have 4 sockets, each with 16 cores.
The following example requests 24 tasks, each with one core. It further specifies that these should be split evenly on 2 nodes, and within the nodes, the 12 tasks should be evenly split on the two sockets. So each CPU on the two nodes will have 6 tasks, each with its own dedicated core. The distribution option will ensure that MPI ranks are distributed cyclically on nodes and sockets.
SLURM is very flexible and allows users to be very specific about their resource requests. Thinking about your application and doing some testing will be important to determine the best request for your specific use.
Download the mpi_job.sh script
#!/bin/bash #SBATCH --job-name=mpi_job_test # Job name #SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL) #SBATCH --mail-user=<email_address> # Where to send mail #SBATCH --ntasks=24 # Number of MPI ranks #SBATCH --cpus-per-task=1 # Number of cores per MPI rank #SBATCH --nodes=2 # Number of nodes #SBATCH --ntasks-per-node=12 # How many tasks on each node #SBATCH --output=mpi_test_%j.out # Standard output and error log # Note: module load of gnu and openmpi is optional (its the default) # You may need other modules, however, to run your program. module load gnu/5.4.0 openmpi/1.10.6 mpirun <your program>