{"id":31,"date":"2018-08-07T06:56:12","date_gmt":"2018-08-07T04:56:12","guid":{"rendered":"https:\/\/www.kth.se\/blogs\/pdc\/?p=31"},"modified":"2019-10-30T11:30:57","modified_gmt":"2019-10-30T10:30:57","slug":"getting-started-with-slurm","status":"publish","type":"post","link":"https:\/\/www.kth.se\/blogs\/pdc\/2018\/08\/getting-started-with-slurm\/","title":{"rendered":"Getting Started with SLURM"},"content":{"rendered":"<div class=\"post-content-wrapper\"><hr \/>\n<p><em>Note: This post has been updated to reflect the changes in the queueing system after the software upgrade of Beskow in June, 2019.<\/em><\/p>\n<hr \/>\n<p>Our <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\">supercomputer clusters at PDC<\/a>, equipped with thousands of multi-core processors, can be used to solve large scientific\/engineering problems. Because of their much higher performance compared to desktop computers or workstations, supercomputer clusters are also called high performance computing (HPC) clusters. Common application fields of HPC clusters include machine learning, galaxy simulation, climate modelling,\u00a0 bioinformatics, computational physics, quantum chemistry, etc.<\/p>\n<p>Building an HPC cluster demands sophisticated technologies and hardware, but fortunately a regular HPC user doesn&#8217;t have to worry too much about that. As an HPC user you can submit <strong>jobs<\/strong> that request the compute <strong>nodes<\/strong> (physical groups of processors) to do the calculations\/simulations you want. But note that you are not the only user of an HPC cluster, there are typically many users using the cluster for the same time period\u00a0and all of them will be submitting their own jobs. You may have realized by now that there needs to be some soft of queueing system that organizes the jobs and distributes them to the compute nodes. This post will briefly introduce SLURM, which is used in all PDC clusters and is the most widely used workload manager for HPC clusters.<\/p>\n<h2>What is SLURM?<\/h2>\n<p>SLURM, or Simple Linux Utility for Resource Management, is an open-source cluster management and job scheduling system. It provides three key functions<\/p>\n<ul>\n<li>Allocation of resources (compute nodes) to users&#8217; jobs<\/li>\n<li>Framework for starting\/executing\/monitoring jobs<\/li>\n<li>Queue management to avoid\u00a0resource contention<\/li>\n<\/ul>\n<p>In other words, SLURM oversees all the resources in the whole HPC cluster. Users then send their jobs (requests to run calculations\/simulations) to SLURM for later execution. SLURM will keep all the submitted jobs in a queue and decide what priorities the jobs have and how the jobs are distributed to the compute nodes. SLURM provides a series of useful commands for the user which we will now go through.<\/p>\n<p><!--more--><\/p>\n<h2>How to submit jobs<\/h2>\n<p>The\u00a0<em>sbatch<\/em>\u00a0command submits a job script to the SLURM queue for later execution<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>$ sbatch job_script.sh\r\n<\/pre>\n<\/div>\n<\/div>\n<p>where\u00a0<em>job_script.sh<\/em>\u00a0is a batch script written in bash syntax. The script usually contains options preceded with\u00a0<em>#SBATCH<\/em>\u00a0to tell SLURM what kind of resources\u00a0are to be allocated to this job. The most common options include the account (time-allocation) to be charged, the expected duration of the job, and the number of compute nodes needed by the job<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>#SBATCH -A &lt;time_allocation&gt;\r\n#SBATCH -t 00:10:00\r\n#SBATCH --nodes=2\r\n<\/pre>\n<\/div>\n<\/div>\n<p>Here,\u00a0<code>-A<\/code>\u00a0(<code>--account<\/code>) specifies the project account to be used by this job, <code>-t<\/code>\u00a0(<code>--time<\/code>) sets a limit on the total run time of this job, and <code>-N<\/code>\u00a0(<code>--nodes<\/code>) requests the number of compute nodes to be allocated to this job.<\/p>\n<p>In addition to\u00a0<code>--nodes<\/code>, it is also common to use the <code>--ntasks-per-node<\/code>\u00a0option for a finer control of parallel execution. As the name indicates, <code>--ntasks-per-node<\/code>\u00a0requests\u00a0that the number of tasks\/processes to be executed on each node will be set to the specified number.<\/p>\n<p>Other useful options include<\/p>\n<ul class=\"simple\">\n<li><code>-J<\/code>\u00a0or <code>--job-name<\/code>: specifies the name for the job.<\/li>\n<li><code>-e<\/code>\u00a0or <code>--error<\/code>: specifies the output file for the job script\u2019s standard error.<\/li>\n<li><code>-o<\/code>\u00a0or <code>--output<\/code>: specifies the output file for the job script\u2019s standard output.<\/li>\n<\/ul>\n<p>Please refer to\u00a0<a class=\"reference external\" href=\"https:\/\/slurm.schedmd.com\/sbatch.html#lbAE\">this page<\/a>\u00a0for a complete list of\u00a0<em>sbatch<\/em>\u00a0options.<\/p>\n<p>Below is a typical job script<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>#!\/bin\/bash -l\r\n\r\n#SBATCH -J myjob\r\n#SBATCH -A &lt;time_allocation&gt;\r\n#SBATCH -t 01:00:00\r\n\r\n#SBATCH --nodes=4\r\n#SBATCH --ntasks-per-node=32\r\n\r\n#SBATCH -e error_file.e\r\n#SBATCH -o output_file.o\r\n\r\n# Run the executable in parallel\r\n&lt;my_parallel_launcher&gt; -n 128 .\/myexe &gt; my_output_file\r\n<\/pre>\n<\/div>\n<\/div>\n<p>On the two PDC clusters, <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a> and <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>,\u00a0<em>srun<\/em>\u00a0and\u00a0<em>mpirun, <\/em>respectively,\u00a0are usually used to launch parallel jobs.\u00a0<em>srun<\/em> is the command to run parallel applications via SLURM. Parallel programs on\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a>\u00a0should always be invoked by the\u00a0<em>srun<\/em>\u00a0command<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>srun -n 128 .\/myexe &gt; my_output_file\r\n<\/pre>\n<\/div>\n<\/div>\n<p>where <code>-n<\/code> specifies the total number of cores to be used. Note that the <code>-n<\/code>\u00a0option of\u00a0<em>srun<\/em> should match the product of the <code>--nodes<\/code> and <code>--ntasks-per-node<\/code> options of\u00a0<em>sbatch<\/em>.<\/p>\n<p>On <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>,\u00a0<em>mpirun<\/em>\u00a0is the command for parallel execution<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre># Load the Intel MPI module\r\nmodule load intelmpi\/17.0.1\r\nmpirun -n 128 .\/myexe &gt; my_output_file<\/pre>\n<\/div>\n<\/div>\n<p>Note that the\u00a0<em>mpirun<\/em> launcher is only available after the IntelMPI module (or other MPI module like OpenMPI) is loaded.<\/p>\n<h2>How to request nodes<\/h2>\n<p>Different types of nodes are available on PDC&#8217;s clusters.\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a>\u00a0has two types of nodes.<\/p>\n<ul>\n<li>1676 nodes have Intel Xeon Haswell CPUs (32 cores\/node).<\/li>\n<li>384 nodes have Intel Xeon Broadwell CPUs (36 cores\/node).<\/li>\n<\/ul>\n<p>As you can see, the majority of the compute nodes on\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a>\u00a0are equipped with Haswell CPUs. To request that your job should exclusively use nodes with Haswell CPUs, you need to specify<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>#SBATCH --ntasks-per-node=32\r\n#SBATCH --constraint=Haswell<\/pre>\n<\/div>\n<\/div>\n<p>where the \u201c<code>--constraint<\/code>\u201d option makes sure that the job runs only on Haswell nodes. The Broadwell CPUs, however, are mostly reserved for users from <a href=\"https:\/\/www.pdc.kth.se\/publications\/pdc-newsletter-articles\/2017-no-2\/pdc-scania-collaboration-1.783448\">Scania<\/a>. If you would like to exclusively execute your job on Broadwell nodes, please contact <a href=\"https:\/\/www.pdc.kth.se\/support\/documents\/contact\/contact_support.html\">PDC Support<\/a>.<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\"><\/div>\n<\/div>\n<p><a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a> has thin and fat nodes that differ in their available memory.<\/p>\n<ul>\n<li>All the 55 thin nodes have 512 GB memory and 24 cores per node.<\/li>\n<li>5 of the 10 fat nodes have 1 TB memory and 48 cores per node.<\/li>\n<li>5 of the 10 fat nodes have 2 TB memory and 48 cores per node.<\/li>\n<\/ul>\n<p>If you need to use the fat nodes on\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>, specify &#8220;<code>#SBATCH --mem=1000000<\/code>&#8221;\u00a0for 1 TB memory and &#8220;<code>#SBATCH --mem=2000000<\/code>&#8221;\u00a0for 2 TB memory.<\/p>\n<h2>How to request GPUs<\/h2>\n<p>PDC&#8217;s\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a> cluster is also equipped with GPU (graphics processing unit) cards. You have probably heard of GPUs &#8211; they are specialized electronic circuits designed to process computer graphics with a highly parallel structure. Many modern GPUs have also been designed for general purpose computing, taking advantage of the SIMD (single instruction multiple data) architecture. General purpose GPUs are now very popular in the field of machine learning, bioinformatics, molecular dynamics, etc.<\/p>\n<p>Two types of GPU cards are available on the fat and thin nodes of\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>.<\/p>\n<ul>\n<li>All the 10 fat nodes have 2 Nvidia Quadro K420 cards\/node.<\/li>\n<li>46 of the thin nodes have 1 Nvidia Quadro K420 card\/node.<\/li>\n<li>9 of the thin nodes have 1 Nvidia Tesla K80 card\/node.<\/li>\n<\/ul>\n<p>If you need to use GPUs in your calculations, use the <code>--gres=&lt;gpu_list&gt;<\/code>\u00a0option in the job script. The format for the <code>&lt;gpu_list&gt;<\/code>\u00a0entry is \u201c<code>name[[:type]:count]<\/code>\u201d. Some examples are given below. Note that one K80 card is counted as two GPUs.<\/p>\n<ul class=\"simple\">\n<li><span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>#SBATCH --gres=gpu:1<\/code><\/span><\/li>\n<li><span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>#SBATCH --gres=gpu:K420:1<\/code><\/span><\/li>\n<li><span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>#SBATCH --gres=gpu:K80:2<\/code><\/span><\/li>\n<\/ul>\n<h2>How to monitor jobs<\/h2>\n<p>We have discussed how to submit jobs with\u00a0<em>sbatch<\/em>\u00a0and how to request certain nodes with specific CPUs and GPUs. Submitted jobs will stay in the SLURM queue and you can monitor the state of\u00a0your jobs with\u00a0<em>squeue<\/em> before they are finished<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>$ squeue -u &lt;username&gt;\r\n<\/pre>\n<\/div>\n<\/div>\n<p>The output of the\u00a0squeue command consists of several columns including job ID, partition, job name, username, job state, elapsed time, number of nodes, node list, etc.<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>   JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)\r\n  733608      main     job1    user1  R    6:11:55      1 t03n03\r\n  733814      main     job2    user2 PD       0:00      1 (Priority)<\/pre>\n<\/div>\n<\/div>\n<p>Here, <strong>partition<\/strong> means a logic group of nodes that can allow certain types of jobs to run. You can imagine that short jobs (e.g. 10 min) and long jobs (e.g. 24 h) will belong to different partitions. The same for small jobs (e.g. 1 node) and large jobs (e.g. 128 nodes). A partition is like a subset of the SLURM queue. Note that, however, there is no need to specify the partition in your job submission script, as the jobs are automatically routed to proper partitions based on the requested time, size of job, type of nodes, etc.<\/p>\n<p><strong>Job state<\/strong> is listed in the ST column of the output of the\u00a0<em>squeue<\/em>\u00a0command. The most common job state codes are<\/p>\n<ul class=\"simple\">\n<li>R: Running<\/li>\n<li>PD: Pending<\/li>\n<li>CG: Completing<\/li>\n<li>CA: Cancelled<\/li>\n<\/ul>\n<p>For more job state codes, please refer to\u00a0<a class=\"reference external\" href=\"https:\/\/slurm.schedmd.com\/squeue.html#lbAG\">this page<\/a>.<\/p>\n<p>For a job that is pending, <em>squeue<\/em>\u00a0gives the reason, which in most cases is <em>Resources<\/em>\u00a0or <em>Priority<\/em>. The former indicates that there currently are not enough resources to run your job, and you have to wait for somewhat longer. The latter suggests that you job has a low priority, possibly because you have recently used a significant amount of resources or that your project has exceeded the time allocation quota in the project you belong to.<\/p>\n<p>For a job that is running,\u00a0<em>squeue<\/em> shows the node (or node list if there&#8217;s more than one node) that the job is running on. On\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>, it is possible to log in to the compute node to check the status of your calculation. If you have a single-node job running on node\u00a0<em>t03n03<\/em>, you may log in to the compute node using the command below. For security reasons it should be<br \/>\ndone <strong>from your local computer<\/strong>:<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>$ ssh &lt;username&gt;@t03n03.pdc.kth.se\r\n<\/pre>\n<\/div>\n<\/div>\n<p>Note that, however, it is not possible to log in to compute node of\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a>.<\/p>\n<p>If you want to know more about your jobs listed by <em>squeue<\/em>, use\u00a0<em>scontrol<\/em>\u00a0with the job ID<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>$ scontrol show job &lt;job_id&gt;\r\n<\/pre>\n<\/div>\n<\/div>\n<p>The output will show detailed information about the job, including priority, submit time, start time, end time, working directory, standard output, etc. Read\u00a0<a class=\"reference external\" href=\"https:\/\/slurm.schedmd.com\/scontrol.html#lbAD\">this page<\/a>\u00a0to learn more about the\u00a0<em>scontrol<\/em>\u00a0command.<\/p>\n<p>To cancel a specific job, use\u00a0<em>scancel<\/em>\u00a0with job ID<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>$ scancel &lt;job_id&gt;<\/pre>\n<\/div>\n<\/div>\n<h2>Interactive nodes<\/h2>\n<p>Sometimes you may want to use a compute node interactively, for example for debugging purposes. This can be done with\u00a0<em>salloc<\/em><\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>$ salloc -A &lt;time_allocation&gt; -t 01:00:00 -N 1<\/pre>\n<\/div>\n<\/div>\n<p>If the cluster is busy at the time, you may need to wait for some time before an interactive node is allocated to you. After allocation of an interactive node you can run parallel programs using <em>srun<\/em> (on <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a>) or <em>mpirun<\/em> (on\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>). Note that <strong><em>srun<\/em> and <em>mpirun<\/em> are mandatory<\/strong> to make sure that the job runs on the compute node. If you execute your program on the login node without <em>srun<\/em> or <em>mpirun<\/em>, the job will be running on the login node, which will cause performance issues or even kill the node.<\/p>\n<p>You can check this out by the following example on\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>. First, ask for an interactive node and print the SLURM node list via the environment variable &#8220;$SLURM_NODELIST&#8221;. Then check the output of <em>hostname<\/em>\u00a0without and with <em>mpirun<\/em>. You can see that the <em>hostname<\/em> command runs on the login node, and that the &#8220;<em>mpirun -n 1 hostname<\/em>&#8221; command runs on the actual compute node. Similarly, on an interactive node of\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a>, parallel programs need to be invoked by the <em>srun<\/em> command.<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>tegner-login-1$ salloc -A &lt;time_allocation&gt; -t 01:00:00 -N 1\r\nsalloc: Granted job allocation 733877\r\nbash-4.2$ echo $SLURM_NODELIST\r\nt03n03\r\nbash-4.2$ hostname\r\ntegner-login-1.pdc.kth.se\r\nbash-4.2$ module load intelmpi\/17.0.1\r\nbash-4.2$ mpirun -n 1 hostname\r\nt03n03.pdc.kth.se\r\n<\/pre>\n<\/div>\n<\/div>\n<p>We have shown in the previous section that you can log in to the compute node of <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>\u00a0provided that you have an active job running on it. If you have an interactive node on <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a>, it is also possible to log in to the compute node <strong>from your local computer<\/strong> via e.g. <span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>ssh\u00a0&lt;username&gt;@t03n03.pdc.kth.se<\/code><\/span>.<\/p>\n<p>The allocated interactive node will be revoked if the actual run time exceeds the requested time limit. You may also manually terminate an interactive node with\u00a0<em>exit<\/em>\u00a0or <em>Ctrl-D<\/em>.<\/p>\n<h2>Environment variables<\/h2>\n<p>Sometimes it is useful to extract certain information from SLURM. This can be done via SLURM environment variables. For instance, the following script uses the SLURM environment variables to copy files from the working directory to a unique scratch directory. Note that <span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>$USER<\/code><\/span> and\u00a0<span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>${USER:0:1}<\/code><\/span>\u00a0stand for your username and the first letter of it, respectively.<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>workdir=$SLURM_SUBMIT_DIR\r\nscratch=\/cfs\/klemming\/scratch\/${USER:0:1}\/$USER\/$SLURM_JOB_ID\r\nmkdir -p $scratch\r\n\r\necho working dir: $workdir\r\necho scratch dir: $scratch\r\n\r\n# copy data to scratch folder\r\ncp -r $workdir\/* $scratch\r\n<\/pre>\n<\/div>\n<\/div>\n<p>See\u00a0<a class=\"reference external\" href=\"https:\/\/slurm.schedmd.com\/sbatch.html#lbAH\">this page<\/a>\u00a0for a complete list of SLURM environment variables.<\/p>\n<p>Moreover, <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a> and\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a> are <a href=\"https:\/\/www.snic.se\/\">SNIC<\/a> resources and therefore support\u00a0<a href=\"https:\/\/www.snic.se\/\">SNIC<\/a> environment variables that can be convenient to use for portability of batch scripts across SNIC clusters. These environment variables are available after loading the\u00a0snic-env module. Below are the <a href=\"https:\/\/www.snic.se\/\">SNIC<\/a> environment variables on\u00a0<a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/beskow-1.737436\">Beskow<\/a> and <a href=\"https:\/\/www.pdc.kth.se\/hpc-services\/computing-systems\/tegner-1.737437\">Tegner<\/a><\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>user@beskow-login2:~&gt; salloc -A &lt;time_allocation&gt; -t 00:10:00 -N 1\r\nsalloc: Granted job allocation 3432703\r\nuser@beskow-login2:~&gt; module load snic-env\r\nuser@beskow-login2:~&gt; printenv | grep SNIC\r\nSNIC_RESOURCE=beskow\r\nSNIC_BACKUP=\/afs\/pdc.kth.se\/home\/u\/user\r\nSNIC_NOBACKUP=\/cfs\/klemming\/nobackup\/u\/user\r\nSNIC_TMP=\/cfs\/klemming\/scratch\/u\/user\r\nSNIC_SITE=pdc<\/pre>\n<\/div>\n<\/div>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>tegner-login-1$ salloc -A &lt;time_allocation&gt; -t 00:10:00 -N 1\r\nsalloc: Granted job allocation 733885\r\nbash-4.2$ module avail snic-env\r\nbash-4.2$ printenv|grep SNIC\r\nSNIC_RESOURCE=tegner\r\nSNIC_BACKUP=\/afs\/pdc.kth.se\/home\/u\/user\r\nSNIC_NOBACKUP=\/cfs\/klemming\/nobackup\/u\/user\r\nSNIC_TMP=\/cfs\/klemming\/scratch\/u\/user\r\nSNIC_SITE=pdc<\/pre>\n<\/div>\n<\/div>\n<h2>SLURM job array<\/h2>\n<p>In scientific computing it is not uncommon that the user needs to submit a large number of similar jobs. One possible solution is to write a program or script that goes through each job folder, creates job script, and submits the job. Unfortunately, this kind of batch-processing program and script are not only error-prone, but also difficult for troubleshooting. A better way of handling many jobs is to use a SLURM <strong>job array<\/strong>, which allows to submit a series of jobs via a single submission script.<\/p>\n<p>Suppose you have 10 jobs residing in folders <em>data-0<\/em>,\u00a0<em>data-1<\/em>, &#8230;,\u00a0<em>data-9<\/em>. You may use the following script to submit all the jobs in one shot (note the <span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>-a<\/code><\/span>\u00a0option)<\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>#!\/bin\/bash -l\r\n\r\n#SBATCH -J array\r\n#SBATCH -A &lt;time_allocation&gt;\r\n#SBATCH -n 1\r\n#SBATCH -t 01:00:00\r\n#SBATCH -a 0-9\r\n\r\n# Fetch one directory from the array based on the task ID\r\n# Note: index starts from 0\r\nCURRENT_DIR=data-${SLURM_ARRAY_TASK_ID}\r\necho \"Running simulation $CURRENT_DIR\"\r\n\r\n# Go to job folder\r\ncd $CURRENT_DIR\r\necho \"Simulation in $CURRENT_DIR\" &gt; result\r\n\r\n# Run job\r\n&lt;my_parallel_launcher&gt; -n 32 .\/myexe &gt; my_output_file\r\n<\/pre>\n<\/div>\n<\/div>\n<p>See\u00a0<a class=\"reference external\" href=\"https:\/\/slurm.schedmd.com\/job_array.html\">this page<\/a>\u00a0for a detailed documentation of SLURM job array.<\/p>\n<p>SLURM job arrays can become handy in many scenarios. You may want to cross-validate the result\u00a0of your statistical analysis, carry out simulations from many different initial conditions, or benchmark your code\/method on a variety of systems. In any case, you may find SLURM job arrays useful in helping you handle large numbers of jobs. The only restriction is that all these jobs have to use the same number of compute nodes and wall-clock time limit, which in practice is usually not a problem.<\/p>\n<h2>Resource reservation<\/h2>\n<p>With SLURM it is possible to reserve resources (e.g. some compute nodes) for jobs to be executed in a specific project.\u00a0 A reservation gives the project exclusive rights to a set of nodes during a specific time period. This is very useful for arranging lab exercises for courses like the\u00a0<a href=\"https:\/\/www.pdc.kth.se\/training\/pdc-summer-school-1.749842\">PDC summer school<\/a>\u00a0to minimize waiting time for course participants. If during a course you have been informed that a reservation is available it is important to use it, otherwise the reserved node will be idle. Resource reservations can be created, updated, or removed by sysadmins, and viewed by normal users with\u00a0<em>scontrol<\/em><\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>user@beskow-login2:&gt; scontrol show reservation\r\nReservationName=summer-2018-08-15 StartTime=2018-08-15T13:45:00 \r\n   EndTime=2018-08-15T17:15:00 Duration=03:30:00\r\n   Nodes=nid00[004-027,029-067,072-108] NodeCnt=100 CoreCnt=3200 \r\n   Features=(null) PartitionName=4h1-800 Flags= TRES=cpu=6400\r\n   Users=(null) Accounts=edu18.summer Licenses=(null) \r\n   State=INACTIVE BurstBuffer=(null) Watts=n\/a\r\n<\/pre>\n<\/div>\n<\/div>\n<p>To use the reserved resources, add the\u00a0<span style=\"font-family: Consolas, Monaco, Lucida Console, monospace\"><code>--reservation<\/code><\/span> option to <em>sbatch<\/em><\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>#SBATCH -A edu18.summer\r\n#SBATCH -t 01:30:00\r\n#SBATCH --nodes=1\r\n#SBATCH --reservation=summer-2018-08-15\r\n<\/pre>\n<\/div>\n<\/div>\n<p>or <em>salloc<\/em><\/p>\n<div class=\"highlight-default\">\n<div class=\"highlight\">\n<pre>$ salloc -A edu18.summer -t 01:30:00 -N 1 --reservation=summer-2018-08-15<\/pre>\n<\/div>\n<\/div>\n<h2>Further information<\/h2>\n<p>You can read more about SLURM in\u00a0<a class=\"reference external\" href=\"https:\/\/slurm.schedmd.com\/quickstart.html\">this page<\/a>.<\/p>\n<h2>Summary<\/h2>\n<p>We hope that you have learned something about SLURM and its usage on HPC clusters after reading this post. Some key points are summarized below.<\/p>\n<ul>\n<li class=\"p1\"><em>sbatch<\/em> submits jobs and\u00a0accepts options preceded by <em>#SBATCH<\/em><\/li>\n<li class=\"p1\"><em>srun<\/em>\u00a0and\u00a0<em>mpirun<\/em> are launchers for parallel programs<\/li>\n<li class=\"p1\">There are different types of nodes in terms of CPU and memory<\/li>\n<li class=\"p1\">GPUs are available on\u00a0Tegner<\/li>\n<li class=\"p1\">You can monitor you jobs with <em>squeue<\/em>,\u00a0<em>scontrol<\/em>\u00a0and\u00a0<em>scancel<\/em><\/li>\n<li class=\"p1\">You can request interactive nodes with <em>salloc<\/em><\/li>\n<li class=\"p1\">SLURM and SNIC environment variables are sometimes handy<\/li>\n<li class=\"p1\">SLURM job arrays can be used to handle large numbers of similar jobs<\/li>\n<li class=\"p1\">SLURM can reserve resources for courses using PDC resources<\/li>\n<\/ul>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Note: This post has been updated to reflect the changes in the queueing system after the software upgrade of Beskow in June, 2019. Our supercomputer clusters at PDC, equipped with thousands of multi-core processors, can be used to solve large scientific\/engineering problems. Because of their much higher performance compared to desktop computers or workstations, supercomputer [&hellip;]<\/p>\n","protected":false},"author":1140,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[3,2],"tags":[11,10],"class_list":["post-31","post","type-post","status-publish","format-standard","hentry","category-best-practice","category-job-scheduling","tag-job-submission","tag-slurm"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p9W9Im-v","_links":{"self":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts\/31","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/users\/1140"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/comments?post=31"}],"version-history":[{"count":75,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts\/31\/revisions"}],"predecessor-version":[{"id":536,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts\/31\/revisions\/536"}],"wp:attachment":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/media?parent=31"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/categories?post=31"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/tags?post=31"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}