Porting and deploying the HiSea use case on C-SCALE

public 1yr ago Version: v0.1.0 0 bookmarks

View Workflow

workflow-coastal-hydrowaq — View Workflow

Help improve this workflow!

This workflow has been published but could be further improved with some additional meta data:

Keyword(s) in categories input, output, operation, topic

You can help improve this workflow by suggesting the addition or removal of keywords, suggest changes and report issues, or request to become a maintainer of the Workflow .

C-SCALE Workflow Solution: Coastal hydrodynamic and water quality modelling using Delft3D FM

C-SCALE workflow solutions are intended to give users a template and reusable components to start building monitoring, modelling and forecasting applications

With the coastal hydrodynamic and water quality modelling workflow solution a user can easily produce hydrodynamic and water quality hindcasts or forecasts for the coastal ocean for a Delft3D FM model schematisation.

The workflow solution has the following functionality

Download the necessary input data for the user's Delft3D Flexible Mesh model setup. Input data include Copernicus' Global Ocean Physics Reanalysis and Global ocean biogeochemistry hindcast , ERA5 and FES2012 .
Prepare the data for ingestion into the user's Delft 3D Flexible Mesh hydrodynamic and water quality model . This entails the preparation of forcings, initial conditions, and boundary condiditons.
Produce hydrodynamic and water quality hindcasts or forcasts based on the user's Delft3D Flexible Mesh hydrodynamic and water quality model setups.
Analyse the simulation outputs in an interactive Jupyter Notebook.

The workflow is executed using Snakemake , a workflow management system for creating and executing reproducible and scalable data analyses. The workflow solution can be deployed on any provider part of the C-SCALE data and compute federation offering cloud container compute.

The workflow components and flow is summarised in the below schematic:

Prerequisites

Github account. Sign up here .
DockerHub account and access to https://hub.docker.com/repository/docker/deltares/delft3dfm (contact [email protected] to arrange access)
CMEMS account, which can be obtained by registering at https://resources.marine.copernicus.eu/registration-form .
CDS API key. Follow instructions at https://cds.climate.copernicus.eu/api-how-to#install-the-cds-api-key to generate the key.
Docker needs to be installed on your computing environment.
Snakemake needs to be installed on your computing environment.

Instructions

1. Build the Docker containers

Instructions on how to build and run the Docker conterainers for each workflow component can be found in the README.md files of corresponding folders of this repo:

2. Install Snakemake

To install Snakemake do:

conda install -n base -c conda-forge mamba
mamba install -c conda-forge -c bioconda snakemake click

3. Run the workflow for the example fm_model included in this repo

In your CLI, cd to use-case-hisea/.
Open use-case-hisea/workflow/config.yml in your preferred text editor and specify
a. dependencies
b. data folder locations
c. delft3dfm folders and files
d. area of interest (this should cover the area of your delft3dfm model schematisation)
e. run mode
Execute the workflow from the use-case-hisea/ directory by doing: snakemake --cores 4 all
To run each rule individually, from the use-case-hisea/ directory do: snakemake -R <insert rule name> --cores 1

Additional instructions

Code Snippets

import pandas as pd
import click
import warnings
import sys
sys.path.append('../delft3dfm_helpers')
from delft3dfm_helpers import get_tref

# Ignore FutureWarning
warnings.filterwarnings("ignore", category=FutureWarning)

@click.command()
@click.option('--mdufile', type=str, help='path/to/mdufile.mdu')
@click.option('--filename_in', help='path/filename of input file')
@click.option('--filename_out', help='path/filename of output file')
@click.option('--date_max', help='date_max as YYYY-MM-DD')
def main(mdufile, filename_in, filename_out, date_max):

    # Define the reference time
    tref = get_tref(mdufile=mdufile)

    #print(tref, '\n')

    # Read the data from file
    data = pd.read_csv(filename_in, delim_whitespace=True, header=None)

    # Add a datetime column based on the first column
    data['datetime'] = tref + pd.to_timedelta(data.iloc[:, 0], unit='m')

    # get the number of days between the last datetime in the last row and date_max
    date_max = pd.to_datetime(date_max)
    last_datetime = data.iloc[-1]['datetime']
    days_diff = int((date_max - last_datetime) / pd.Timedelta(days=1))
    #print(days_diff, '\n')

    for i in range(days_diff+1):

        # Append an empty row
        data = data.append({}, ignore_index=True)

        # Get the previous value in column 0
        prev_value = data.iloc[-2, 0]
        #print(prev_value, '\n')

        # Get the unique difference between all the values in column 0
        unique_diff = data[0].diff().dropna().unique()
        #print(unique_diff, '\n')

        # Fill the value in column 0, last row with the previous value plus the unique differenc
        data.iloc[-1, 0] = prev_value + unique_diff

        # Set datetime of last row to the corresponding value from column 0
        data.iloc[-1, 2] = tref + pd.to_timedelta(data.iloc[-1, 0], unit='m')

        # Get the day and month of the datetime in the last row
        last_date = data.iloc[-1]['datetime']
        last_day_month = last_date.strftime('%m-%d')
        #print(last_date)
        #print(last_day_month)

        # Create a mask to filter the rows with the same day and month
        mask = data['datetime'].dt.strftime('%m-%d') == last_day_month

        # Compute the average of values in column 1 for these rows
        # Set the value in the last row of column 1 as the average
        data.iloc[-1, 1] = data.loc[mask, 1].mean()

    # convert column 0 to integers
    data[0] = data[0].astype(int)

    # write to file
    data.to_csv(filename_out, columns=[0,1], header=False, index=False, sep='\t', float_format='%.8f')

if __name__ == '__main__':
    main()

Python Pandas CLICK Delft3D From line 30 of workarounds/update_existing_RadSurf_daily_dot_tim.py

run:
    if params.run_mode == 'forecast':
        date_min, date_max = set_date_range(run_mode=params.run_mode,
                                            forecast_window_mid_pt=params.forecast_window_mid_pt,
                                            forecast_window=params.forecast_window,
                                            outfile='workflow/config.yml')
    else:
        date_min, date_max = set_date_range(run_mode=params.run_mode,
                                            tstart=params.tstart,
                                            tstop=params.tstop,
                                            outfile='workflow/config.yml')
    os.system('touch workflow/logs/$(date +%Y-%m-%d)_set_date_range.done')
    current_date = date.today().strftime('%Y-%m-%d')
    with open(os.path.join("workflow/logs/", f"{current_date}_set_date_range.done"), 'w') as f:
        f.write(f"date_min: {date_min}\n")
        f.write(f"date_max: {date_max}\n")

SnakeMake From line 39 of workflow/Snakefile

run:
    update_mdu_tstart_tstop(mdufile = params.mdufile, date_min=params.date_min, date_max=params.date_max)

SnakeMake From line 65 of workflow/Snakefile

shell:
    """
    sudo docker run \
        -v {params.cdsapirc_loc}:/root/.cdsapirc \
        -v {params.data_dir_path}/download/$(date +%Y-%m-%d):/data \
        download-input python download_era5.py \
            --longitude_min {params.lon_min} \
            --longitude_max {params.lon_max} \
            --latitude_min {params.lat_min} \
            --latitude_max {params.lat_max} \
            --date_min {params.date_min} \
            --date_max {params.date_max} && \
    touch workflow/logs/$(date +%Y-%m-%d)_download_era5.done
    """

SnakeMake From line 84 of workflow/Snakefile

shell:
    """
    sudo docker run \
        -v {params.data_dir_path}/download/$(date +%Y-%m-%d):/data \
        download-input python download_cmems_physics.py \
            --username {params.cmems_uname} \
            --password {params.cmems_pwd} \
            --longitude_min {params.lon_min} \
            --longitude_max {params.lon_max} \
            --latitude_min {params.lat_min} \
            --latitude_max {params.lat_max} \
            --date_min {params.date_min} \
            --date_max {params.date_max} && \
    touch workflow/logs/$(date +%Y-%m-%d)_download_cmems_physics.done
    """

SnakeMake From line 116 of workflow/Snakefile

shell:
    """
    sudo docker run \
        -v {params.data_dir_path}/download/$(date +%Y-%m-%d):/data \
        download-input python download_cmems_biogeochemistry.py \
            --username {params.cmems_uname} \
            --password {params.cmems_pwd} \
            --longitude_min {params.lon_min} \
            --longitude_max {params.lon_max} \
            --latitude_min {params.lat_min} \
            --latitude_max {params.lat_max} \
            --date_min {params.date_min} \
            --date_max {params.date_max} && \
    touch workflow/logs/$(date +%Y-%m-%d)_download_cmems_biogeochemistry.done
    """

SnakeMake From line 149 of workflow/Snakefile

shell:
    """
    sudo docker run \
        -v {params.data_dir_path}/download/$(date +%Y-%m-%d)/era5:/data/input \
        -v {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d):/data/output \
        getera ERA5_convert2_FM_and_merge_allVars.py \
            --input /data/input \
            --output /data/output && \
    touch workflow/logs/$(date +%Y-%m-%d)_preprocess_era5.done
    """

SnakeMake From line 174 of workflow/Snakefile

shell:
    """
    sudo mkdir -p {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/DFMWAQ_input && \
    touch workflow/logs/$(date +%Y-%m-%d)_create_random_DFMWAQ_folder.done
    """

SnakeMake From line 190 of workflow/Snakefile

shell:
    """
    sudo docker run \
        -v {params.data_dir_path}/download/$(date +%Y-%m-%d)/cmems:/data/input \
        -v {params.model_dir_path}:/data/model \
        -v {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d):/data/output \
        preprocessing boundary.py \
            --interp true \
            --simultaneous true \
            --steric true \
            --input /data/input \
            --model /data/model \
            --output /data/output && \
    touch workflow/logs/$(date +%Y-%m-%d)_preprocess_cmems.done
    """

SnakeMake From line 208 of workflow/Snakefile

run:
    fes2012_files_exist = check_fes2012_files_exist(params.fes2012_data_dir_path)
    if fes2012_files_exist:
        os.system('touch workflow/logs/check4fes2012_files.done')

SnakeMake From line 229 of workflow/Snakefile

shell:
    """
    sudo docker run \
        -v {params.fes2012_data_dir_path}:/data/input \
        -v {params.model_dir_path}:/data/model \
        -v {params.data_dir_path}/output/preprocessing/fes2012:/data/output \
        preprocessing tide.py \
            --fespath /data/input \
            --coords "{params.lon_min}, {params.lon_max}, {params.lat_min}, {params.lat_max}" \
            --pli {params.plifile1} \
            --pli {params.plifile2} \
            --output /data/output \
            --model /data/model && \
    touch workflow/logs/preprocess_fes2012.done
    """

SnakeMake From line 251 of workflow/Snakefile

shell:
    "cp -r \
        {params.data_dir_path}/output/preprocessing/fes2012/tide_east2.bc \
        {params.data_dir_path}/output/preprocessing/fes2012/tide_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/DFMWAQ_east2_tmp.ext \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/DFMWAQ_input \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/DFMWAQ_south2_tmp.ext \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Diat_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Diat_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/east2.pli \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Green_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Green_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/NO3_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/NO3_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Opal_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Opal_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/OXY_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/OXY_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/PO4_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/PO4_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/POC1_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/POC1_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/PON1_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/PON1_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/POP1_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/POP1_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/salinity_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/salinity_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Si_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/Si_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/south2.pli \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/steric_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/steric_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/temperature_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/temperature_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/uxuy_east2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/uxuy_south2.bc \
        {params.data_dir_path}/output/preprocessing/$(date +%Y-%m-%d)/era5_FM.nc \
        {params.model_dir_path}/input/. && \
        touch workflow/logs/$(date +%Y-%m-%d)_copy_preprocessed_data_to_model_input_folder.done"

SnakeMake From line 278 of workflow/Snakefile

shell:
    """
    python scripts/workarounds/update_existing_RadSurf_daily_dot_tim.py \
                               --mdufile {params.mdufile} \
                               --filename_in {params.model_dir_path}/input/RadSurf_daily.tim \
                               --filename_out {params.model_dir_path}/input/RadSurf_daily.tim \
                               --date_max {params.date_max} && \
    touch workflow/logs/$(date +%Y-%m-%d)_workaround_update_RadSurf_daily.done
    """

SnakeMake From line 329 of workflow/Snakefile

shell:
    """
    sudo docker run \
        -v {params.model_dir_path}:/data \
        --shm-size=4gb \
        --ulimit stack=-1 \
        -t deltares/delft3dfm:latest && \
    touch workflow/logs/$(date +%Y-%m-%d)_run_delft3dfm.done
    """

SnakeMake From line 359 of workflow/Snakefile

shell:
    """
    sudo mkdir -p {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/rst && \
    sudo mv {params.model_dir_path}/DFM_OUTPUT_tttz_waq/*rst* {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/rst/. && \
    sudo mkdir -p {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/map && \
    sudo mv {params.model_dir_path}/DFM_OUTPUT_tttz_waq/*map* {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/map/. && \
    sudo mkdir -p {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/his && \
    sudo mv {params.model_dir_path}/DFM_OUTPUT_tttz_waq/*his* {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/his/. && \
    sudo mkdir -p {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/dia && \
    sudo mv {params.model_dir_path}/DFM_OUTPUT_tttz_waq/*dia* {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/dia/. && \
    sudo mkdir -p {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/other && \
    sudo mv {params.model_dir_path}/DFM_OUTPUT_tttz_waq/* {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/other/. && \
    touch workflow/logs/$(date +%Y-%m-%d)_move_raw_data_to_output_folder.done
    """

SnakeMake From line 377 of workflow/Snakefile

shell:
    """
    sudo rm -r \
        {params.model_dir_path}/input/tide_east2.bc \
        {params.model_dir_path}/input/tide_south2.bc \
        {params.model_dir_path}/input/DFMWAQ_east2_tmp.ext \
        {params.model_dir_path}/input/DFMWAQ_input \
        {params.model_dir_path}/input/DFMWAQ_south2_tmp.ext \
        {params.model_dir_path}/input/Diat_east2.bc \
        {params.model_dir_path}/input/Diat_south2.bc \
        {params.model_dir_path}/input/east2.pli \
        {params.model_dir_path}/input/Green_east2.bc \
        {params.model_dir_path}/input/Green_south2.bc \
        {params.model_dir_path}/input/NO3_east2.bc \
        {params.model_dir_path}/input/NO3_south2.bc \
        {params.model_dir_path}/input/Opal_east2.bc \
        {params.model_dir_path}/input/Opal_south2.bc \
        {params.model_dir_path}/input/OXY_east2.bc \
        {params.model_dir_path}/input/OXY_south2.bc \
        {params.model_dir_path}/input/PO4_east2.bc \
        {params.model_dir_path}/input/PO4_south2.bc \
        {params.model_dir_path}/input/POC1_east2.bc \
        {params.model_dir_path}/input/POC1_south2.bc \
        {params.model_dir_path}/input/PON1_east2.bc \
        {params.model_dir_path}/input/PON1_south2.bc \
        {params.model_dir_path}/input/POP1_east2.bc \
        {params.model_dir_path}/input/POP1_south2.bc \
        {params.model_dir_path}/input/salinity_east2.bc \
        {params.model_dir_path}/input/salinity_south2.bc \
        {params.model_dir_path}/input/Si_east2.bc \
        {params.model_dir_path}/input/Si_south2.bc \
        {params.model_dir_path}/input/south2.pli \
        {params.model_dir_path}/input/steric_east2.bc \
        {params.model_dir_path}/input/steric_south2.bc \
        {params.model_dir_path}/input/temperature_east2.bc \
        {params.model_dir_path}/input/temperature_south2.bc \
        {params.model_dir_path}/input/uxuy_east2.bc \
        {params.model_dir_path}/input/uxuy_south2.bc \
        {params.model_dir_path}/input/era5_FM.nc \
        {params.model_dir_path}/DFM_interpreted_idomain_myortho3_net.nc \
        {params.model_dir_path}/log*.irlog \
        {params.model_dir_path}/myortho3_*_net.nc \
        {params.model_dir_path}/tttz_waq_*.cache \
        {params.model_dir_path}/tttz_waq_*.mdu \
        {params.model_dir_path}/unstruc.dia && \
    sudo cp {params.mdufile} {params.data_dir_path}/output/fm_model_runs/$(date +%Y-%m-%d)/. && \
    touch workflow/logs/$(date +%Y-%m-%d)_cleanup_model_folder.done
    """