Warlock is a snakemake workflow to spawn multiple demons (deme-based oncology models) as jobs running around on a cluster environment 😈😈

public 1yr ago Version: 1.1.4 0 bookmarks

View Workflow

Help improve this workflow!

This workflow has been published but could be further improved with some additional meta data:

Keyword(s) in categories input, output, operation, topic

You can help improve this workflow by suggesting the addition or removal of keywords, suggest changes and report issues, or request to become a maintainer of the Workflow .

demon (deme-based oncology model) is a flexible framework for modelling intra-tumour population genetics with varied spatial structures and modes of cell dispersal. It is primarly designed for computational biologists and mathematicians who work in the field of ecology on a cellular level; investiaging mechanisms behind tumour evolution.

The following repository encapsulates demon into an automated and reproducible snakemake workflow in order to simplify parallel simulations to all users. Just as a regular warlock , it can spawn multiple demons (as cluster jobs), provided enough mystic powers (computational resources) are available.

Installation

The workflow is designed to run on Linux and macOS systems.
We have prepared a dedicated conda environment recipe which will contain all prerequisites required to execute the workflow. Thus Anaconda/Miniconda package manager is a natural dependency (see Appendix A for installation instructions.)

Clone the repository and navigate inside that directory
```
git clone https://github.com/AngryMaciek/warlock.git --recursive
cd warlock
```
Note: if you wish to compile a different version of demon (i.e. other branch) please remember to navigate to resources/demon_model first and git checkout a proper branch of that repository.

Create and activate conda environment

conda env create
conda activate warlock

Compile demon
- Linux:
```
g++ resources/demon_model/src/demon.cpp -o resources/demon_model/bin/demon -I$HOME/miniconda3/envs/warlock/include -lm
```
- macOS:
```
clang++ resources/demon_model/src/demon.cpp -o resources/demon_model/bin/demon -I$HOME/miniconda3/envs/warlock/include -lm
```
Note: remember to adjust miniconda3 (and its path) in the command above, in case you have a different manager installed on your system. All in all, the point is to provide the include directory of your warlock environment to the compiler.

Configuration

For a detailed description of all available simulation parameters please inspect GitHub repository of the core demon model.

These parameters are now required to be set inside a YAML-formatted pipeline configuration file. A template for this file is available here . Please see an example configuration file designed for the CI tests here . Note that multiple values for distinct parameters might be provided in lists. Current implementation of the workflow prepares a Cartesian product of all parameter's values and runs demon with each of them in parallel.

Additionally, please notice that two absolute paths are required to be set in the same configuration file: path to this cloned repository as well as path for the analyses output directory.

Execution

This workflow should be executed via a top-level bash script: warlock.sh which has the following description:

This is the main script to call the _warlock_ workflow.
Available options:
 -c/--configfile {XXX} (REQUIRED)
 Path to the snakemake config file.
 -e/--environment {local/slurm} (REQUIRED)
 Environment to execute the workflow in:
 * local = execution on the local machine.
 * slurm = slurm cluster support.
 -n/--cores {XXX} (OPTIONAL)
 Number of cores available for the workflow.
 (Default = 1)

Local

Example command for a local workflow execution:

bash warlock.sh --configfile {PATH} --environment local

Docker container

We have additionally prepared a development/execution Docker image which one may use in order to run warlock in a fully encapsulated environment (that is, a container).
Assuming the Docker Engine is running locally please build the image with:

docker build -t warlock:latest .

To test the container one can execute the following bash command:

docker run --name warlock warlock bash -c "source ~/.bashrc; bash testscript.sh"

Finally, enter the container to start your work with:

docker run -it warlock:latest

Alternatively, please note that the image is also uploaded to DockerHub , one may download it with:

docker pull angrymaciek/warlock:latest

SLURM cluster

Example command for a cluster-supported workflow execution:

bash warlock.sh --configfile {PATH} --environment slurm

Please note that, depending on the complexity of the simulations, it might be required to adjust parameters for cluster jobs. If the expected required resources (memory or computation time) are high please adjust time and mem fields in the cluster submission configuration file, located at: /workflow/profiles/slurm/slurm-config.json .

Running large workflows with hundreds of cluster jobs might take very long; consider executing warlock in a terminal multiplexer, e.g. tmux .

Output

After each pipeline run the main output directory will contain three subdirectories: configfiles , simulations and logs . Each simulation run with a specific set of parameters is encoded by a 4-letter code. The first directory contains configuration files for each of the simulation runs; simulations contain all demon output files; logs keep captured standard output and error streams for the commands.

Example

Feel free to run the pipeline and inspect the results yourself in an interactive jupyter notebook we prepared.
Alternatively, feel free to run a small test script on your local machine with bash testscript.sh .

Community guidelines

For guidelines on how to contribute to the project or report issues, please see contributing instructions .
For other inquires feel free to contact project lead by email: ✉️

Citation

Results obtained with demon have already been published in:

Noble R, Burri D, Le Sueur C, Lemant J, Viossat Y, Kather JN, Beerenwinkel N. Spatial structure governs the mode of tumour evolution. Nat Ecol Evol. 2022 Feb;6(2):207-217. doi: https://doi.org/10.1038/s41559-021-01615-9

Noble R, Burley JT, Le Sueur C, Hochberg ME. When, why and how tumour clonal diversity predicts survival. Evol Appl. 2020 Jul 18;13(7):1558-1568. doi: https://doi.org/10.1111/eva.13057

Appendix A: Miniconda installation

To install the latest version of Miniconda on a Linux system please execute:

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
source .bashrc

For other platforms, see all available installers here .

Code Snippets

run:
    with open(log.LOG_local_stderr, "w") as errlogfile:
        try:
            os.makedirs(
                params.DIR_demon_configfile_dirpath,
                exist_ok=True
            )
            with open(output.DAT_demon_configfile, "w") as f:
                f.write(ENCODED_DEMON_CONFIGFILES[params.STR_runID])
        except Exception:
            traceback.print_exc(file = errlogfile)
            raise Exception(
                "Workflow error at rule: prepare_configfiles"
            )