Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# Single job step
In SLURM, Job Steps are a way to launch distinct parallel (most commonly) and/or sequential tasks from within a single job script. Job Steps are executed using the SLURM command "srun"
This is an example where Slurm is instructed to run a single job step. A single job step is fast to write and simple to use.
This folder contains one batch script that runs a sequential program and one batch sript that runs a parallel program. The batch scripts it self, that defines the job steps, are similar with only slight modifications in the settings.
## How to run
To run the example do the following steps:
1. Log in to Lundgren
2. Change directory to the example code
3. Run `sbatch single_job_step_sequential.sh` or `sbatch single_job_step_parallel.sh`
4. Check queue status by running `squeue`
5. When the job is completed check the file _sequential_single_job_step.log_ or _parallel_single_job_step.log_ for the program log.
_If you run the parallel example, try change the batch file to run on 1 and or 4 cores._
## Detailed description of the example
The batch script is the main file for the job allocation and preparation. Inside the python script a few environmental variables are fetched and printed out.
### The batch script
The batch script, _single_job_step_sequential.sh_ / _single_job_step_parallel.sh_ contains three sections. The first section contains input arguments to the Slurm scheduler. The second section loads Python into environment so it is accessible and lastly the a job step is performed.
The input arguments are defined with a comment beginning with SBATCH followed by the argument key and value. For easier readablility the -- method is used.
- __job-name:__ The name of the job
- __time:__ The requeted time
- __ntasks:__ The number of tasks to be performed in this job
- __cpus-per-task:__ The requested number of cpus per task
- __mem-per-cpu:__ The requested memory adjusted per the number of cpu's
- __output:__ File name for standard output
Python needs to be loaded into the environment in order to be accessible this is done in the next step with the __module__ command.
The single job step is allocated and performed with the __srun__ command.
#### The python script
The python script represents the taskt to be done. In this case the task is read an input file and wait to simulate a calculation and afterwards print to an output file.
- The environment variable __JOB_ID__ can be used to create temporary files and folders.
- The environment variable __SLURM_CPUS_PER_TASK__ is used to restrict the worker pool to the allocated number of cpus when running in parallel.
### How to set the number of cores in different programing languages and softwares
Most programming languages and softwares tries to make use of all cores that are available. This can lead to an oversubscription on the resources. On a shared resource one must match the maximum used resources with the allocated ones. This section gives a reference in how to do it in commonly used softwares.
- __CPLEX:__ Use the parameter _global thread count_. Read more in the [documentation](https://www.ibm.com/docs/en/icos/22.1.2?topic=parameters-global-thread-count)
- __Gurobi:__ Use the configuration parameter _ThreadLimit_. Read more in the [documentation](https://docs.gurobi.com/projects/optimizer/en/current/reference/parameters.html#threadlimit)
- __MATLAB:__ Create a instance of the parpool object with the _poolsize_ set to the number of cores and use the pool when running in parallell. Read more in the [documentation](https://se.mathworks.com/help/parallel-computing/parpool.html)
- __Python:__ If the multiprocessing package is used, create an instance of the pool class with the _processes_ set to the number of cores and use the pool when running in parallell. Read more in the [documentation](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool)