Skip to content
Snippets Groups Projects
Commit 29e86c08 authored by Rasmus Ringdahl's avatar Rasmus Ringdahl
Browse files

feat: add multi core job example

Closes #4
parent c89d2ecd
No related branches found
No related tags found
1 merge request!2feat: add multi core job example
# Multi core jobs
A multi core job is a job that splits the computation to multiple cores. This type of job is the most suitable and most common ones to run on Lundgren. This includes optimization problems and heavy computations.
## How to run
To run the example do the following steps:
1. Log in to Lundgren
2. Change directory to the example code
3. Run `sbatch multi_core_job.sh`
4. Check queue status by running `squeue`
5. When the job is completed check the file _multi_core_job.log_
Try changing the number of cpus in _multi_core_job.sh_ and see the changes in processing time.
## Detailed description of the example
The batch script is the main file for the job allocation and preparation. Inside the python script a few environmental variables are fetched and printed out.
### The batch script
The batch script, multi_core_job.sh_, contains three sections. The first section contains input arguments to the Slurm scheduler. The second section loads Python into environment so it is accessible and lastly the a job step is performed.
The input arguments are defined with a comment beginning with SBATCH followed by the argument key and value. For easier readablility the -- method is used.
- __job-name:__ The name of the job is set to demo_multi_core
- __time:__ The requeted time is set to 5 minutes, _00:05:00_
- __ntasks:__ The number of tasks to be performed in this job is set to _1_.
- __cpus-per-task:__ The requested number of cores per task is set to _2_
- __mem:__ The requested memory is set to _50 MB_
- __output:__ The standard output should be sent to the file multi_core_job.log_
Python needs to be loaded into the environment in order to be accessible this is done in the next step with the __module__ command.
The job step with the single task is allocated and performed with the __srun__ command.
#### The python script
The python script represents the taskt to be done. In this case the task is to wait a random time and print the waiting is done.
The environment variable __SLURM_CPUS_PER_TASK__ is used to restrict the worker pool to the allocated number of cores.
### How to set the number of cores in different programing languages and softwares
Most programming languages and softwares tries to make use of all cores that are available. This can lead to an oversubscription on the resources. On a shared resource one must match the maximum used resources with the allocated ones. This section gives a reference in how to do it in commonly used softwares.
- __CPLEX:__ Use the parameter _global thread count_. Read more in the [documentation](https://www.ibm.com/docs/en/icos/22.1.2?topic=parameters-global-thread-count)
- __Gurobi:__ Use the configuration parameter _ThreadLimit_. Read more in the [documentation](https://docs.gurobi.com/projects/optimizer/en/current/reference/parameters.html#threadlimit)
- __MATLAB:__ Create a instance of the parpool object with the _poolsize_ set to the number of cores and use the pool when running in parallell. Read more in the [documentation](https://se.mathworks.com/help/parallel-computing/parpool.html)
- __Python:__ If the multiprocessing package is used, create an instance of the pool class with the _processes_ set to the number of cores and use the pool when running in parallell. Read more in the [documentation](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool)
#! /bin/bash
#SBATCH --job-name=demo_multi_core
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=50MB
#SBATCH --output=multi_core_job.log
# Loading Python into the environment
module load python/anaconda3-2024.02-3.11.7
# Start job stage
srun python multi_core_task.py
\ No newline at end of file
from datetime import datetime
from multiprocessing import Pool
import logging
import os
import random
import time
logger = logging.getLogger(__name__)
def sleep(input):
time.sleep(input[1])
logger.info('Task %d done.',input[0])
def main():
# Read environment variables.
NUMBER_OF_CORES = os.environ.get('SLURM_CPUS_PER_TASK','Unknown')
if NUMBER_OF_CORES in 'Unknown':
logger.error('Unkown number of cores, exiting.')
return
NUMBER_OF_CORES = int(NUMBER_OF_CORES)
logger.info('Running program with %d cores.',NUMBER_OF_CORES)
# Creating a list of tasks to be performed
# This represents the calculations
random.seed(1)
tasks = []
total_time = 0
for i in range(10):
time = random.randrange(1,29)
tasks.append((i, time))
total_time = total_time + time
# Creating a multiprocessing pool to perform the tasks
pool = Pool(processes=NUMBER_OF_CORES)
# Running submitting the tasks to the worker pool
tic = datetime.now()
logger.info('Submitting tasks to pool.')
pool.map(sleep, tasks)
toc = datetime.now()
logger.info('All tasks are done, took %d seconds, compared to %d seconds with single thread.',
(toc-tic).seconds, total_time)
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
main()
......@@ -21,3 +21,8 @@ A single core job is a job with only a single thread. This type of job is used w
A simple example could be a data parser that reads a file and transforms it into a more suitable format.
Learn more about the [example](https://gitlab.liu.se/rasri17/lundgren-examples/-/tree/main/1_single_core_job).
#### Example 2 - Mutli core job
A multi core job is a job that splits the computation to multiple cores. This type of job is the most suitable and most common ones to run on Lundgren. This includes optimization problems and heavy computations.
Learn more about the [example](https://gitlab.liu.se/rasri17/lundgren-examples/-/tree/main/2_multi_core_job).
\ No newline at end of file
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment