Pipeline to fetch metadata and raw FastQ files from public and private databases

public public 1yr ago Version: 1.10.0 0 bookmarks
View Workflow
Report
fetchngs
fetchngs
View Workflow

Introduction

nf-core/fetchngs is a bioinformatics pipeline to fetch metadata and raw FastQ files from both public and private databases. At present, the pipeline supports SRA / ENA / DDBJ / GEO / Synapse ids (see usage docs ).

Usage

Note If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.

First, prepare a samplesheet with your input data that looks as follows:

ids.csv : 

SRR9984183
SRR13191702
ERR1160846
ERR1109373
DRR028935
DRR026872

Each line represents a database id. Please see next section for supported ids.

Now, you can run the pipeline using: 

nextflow run nf-core/fetchngs \
 -profile <docker/singularity/.../institute> \
 --input ids.csv \
 --outdir <OUTDIR>

Warning: Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters ; see docs .

For more details, please refer to the usage documentation and the parameter documentation .

Supported ids

Via a single file of ids, provided one-per-line (see example input file ) the pipeline performs the following steps:

SRA / ENA / DDBJ / GEO ids

  1. Resolve database ids back to appropriate experiment-level ids and to be compatible with the ENA API

  2. Fetch extensive id metadata via ENA API

  3. Download FastQ files:

    • If direct download links are available from the ENA API, fetch in parallel via curl and perform md5sum check

    • Otherwise use sra-tools to download .sra files and convert them to FastQ

  4. Collate id metadata and paths to FastQ files in a single samplesheet

Synapse ids

  1. Resolve Synapse directory ids to their corresponding FastQ files ids via the synapse list command.

  2. Retrieve FastQ file metadata including FastQ file names, md5sums, etags, annotations and other data provenance via the synapse show command.

  3. Download FastQ files in parallel via synapse get

  4. Collate paths to FastQ files in a single samplesheet

Pipeline output

The columns in the output samplesheet can be tailored to be accepted out-of-the-box by selected nf-core pipelines (see usage docs ), these currently include:

To see the the results of a test run with a full size dataset refer to the results tab on the nf-core website pipeline page. For more details about the output files and reports, please refer to the output documentation .

Credits

nf-core/fetchngs was originally written by Harshil Patel ( @drpatelh ) from Seqera Labs, Spain and Jose Espinosa-Carrasco ( @JoseEspinosa ) from The Comparative Bioinformatics Group at The Centre for Genomic Regulation, Spain . Support for download of sequencing reads without FTP links via sra-tools was added by Moritz E. Beber ( @Midnighter ) from Unseen Bio ApS, Denmark . The Synapse workflow was added by Daisy Han @daisyhan97 and Bruno Grande @BrunoGrandePhD from Sage Bionetworks, Seattle .

Contributions and Support

If you would like to contribute to this pipeline, please see the contributing guidelines .

For further information or help, don't hesitate to get in touch on the Slack #fetchngs channel (you can join with this invite ).

Citations

If you use nf-core/fetchngs for your analysis, please cite it using the following doi: 10.5281/zenodo.5070524

An extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.

You can cite the nf-core publication as follows:

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .

Code Snippets

17
18
19
20
21
22
23
24
25
26
"""
multiqc_mappings_config.py \\
    $csv \\
    multiqc_config.yml

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    python: \$(python --version | sed 's/Python //g')
END_VERSIONS
"""
NextFlow From line 17 of local/multiqc_mappings_config.nf 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
"""
curl \\
    $args \\
    -L ${fastq[0]} \\
    -o ${meta.id}.fastq.gz

echo "${meta.md5_1}  ${meta.id}.fastq.gz" > ${meta.id}.fastq.gz.md5
md5sum -c ${meta.id}.fastq.gz.md5

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    curl: \$(echo \$(curl --version | head -n 1 | sed 's/^curl //; s/ .*\$//'))
END_VERSIONS
"""
NextFlow From line 23 of local/sra_fastq_ftp.nf 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
"""
curl \\
    $args \\
    -L ${fastq[0]} \\
    -o ${meta.id}_1.fastq.gz

echo "${meta.md5_1}  ${meta.id}_1.fastq.gz" > ${meta.id}_1.fastq.gz.md5
md5sum -c ${meta.id}_1.fastq.gz.md5

curl \\
    $args \\
    -L ${fastq[1]} \\
    -o ${meta.id}_2.fastq.gz

echo "${meta.md5_2}  ${meta.id}_2.fastq.gz" > ${meta.id}_2.fastq.gz.md5
md5sum -c ${meta.id}_2.fastq.gz.md5

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    curl: \$(echo \$(curl --version | head -n 1 | sed 's/^curl //; s/ .*\$//'))
END_VERSIONS
"""
NextFlow From line 38 of local/sra_fastq_ftp.nf 
21
22
23
24
25
26
27
28
29
30
31
32
"""
echo $id > id.txt
sra_ids_to_runinfo.py \\
    id.txt \\
    ${id}.runinfo.tsv \\
    $metadata_fields

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    python: \$(python --version | sed 's/Python //g')
END_VERSIONS
"""
NextFlow From line 21 of local/sra_ids_to_runinfo.nf 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
"""
head -n 1 `ls ./samplesheets/* | head -n 1` > samplesheet.csv
for fileid in `ls ./samplesheets/*`; do
    awk 'NR>1' \$fileid >> samplesheet.csv
done

head -n 1 `ls ./mappings/* | head -n 1` > id_mappings.csv
for fileid in `ls ./mappings/*`; do
    awk 'NR>1' \$fileid >> id_mappings.csv
done

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    sed: \$(echo \$(sed --version 2>&1) | sed 's/^.*GNU sed) //; s/ .*\$//')
END_VERSIONS
"""
NextFlow From line 18 of local/sra_merge_samplesheet.nf 
17
18
19
20
21
22
23
24
25
26
"""
sra_runinfo_to_ftp.py \\
    ${runinfo.join(',')} \\
    ${runinfo.toString().tokenize(".")[0]}.runinfo_ftp.tsv

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    python: \$(python --version | sed 's/Python //g')
END_VERSIONS
"""
NextFlow From line 17 of local/sra_runinfo_to_ftp.nf 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
"""
synapse \\
    -c $config \\
    get \\
    $args \\
    $meta.id

echo "${meta.md5} \t ${meta.name}" > ${meta.id}.md5

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    synapse: \$(synapse --version | sed -e "s/Synapse Client //g")
END_VERSIONS
"""
NextFlow synapseclient From line 23 of local/synapse_get.nf 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
"""
synapse \\
    -c $config \\
    list \\
    $args \\
    $id \\
    $args2 \\
    > ${id}.list.txt

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    syanpse: \$(synapse --version | sed -e "s/Synapse Client //g")
END_VERSIONS
"""
NextFlow synapseclient From line 22 of local/synapse_list.nf 
17
18
19
20
21
22
23
24
25
26
27
"""
head -n 1 `ls ./samplesheets/* | head -n 1` > samplesheet.csv
for fileid in `ls ./samplesheets/*`; do
    awk 'NR>1' \$fileid >> samplesheet.csv
done

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    sed: \$(echo \$(sed --version 2>&1) | sed 's/^.*GNU sed) //; s/ .*\$//')
END_VERSIONS
"""
NextFlow From line 17 of local/synapse_merge_samplesheet.nf 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
"""
synapse \\
    -c $config \\
    show \\
    $args \\
    $id \\
    $args2 \\
    > ${id}.metadata.txt

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    synapse: \$(synapse --version | sed -e "s/Synapse Client //g")
END_VERSIONS
"""
NextFlow synapseclient From line 22 of local/synapse_show.nf 
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
"""
export NCBI_SETTINGS="\$PWD/${ncbi_settings}"

fasterq-dump \\
    $args \\
    --threads $task.cpus \\
    --outfile $outfile \\
    ${key_file} \\
    ${sra.name}

pigz \\
    $args2 \\
    --no-name \\
    --processes $task.cpus \\
    *.fastq

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    sratools: \$(fasterq-dump --version 2>&1 | grep -Eo '[0-9.]+')
    pigz: \$( pigz --version 2>&1 | sed 's/pigz //g' )
END_VERSIONS
"""
NextFlow From line 28 of fasterqdump/main.nf 
17
18
19
20
21
22
23
24
25
26
"""
multiqc_mappings_config.py \\
    $csv \\
    multiqc_config.yml

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    python: \$(python --version | sed 's/Python //g')
END_VERSIONS
"""
NextFlow From line 17 of local/multiqc_mappings_config.nf 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
"""
curl \\
    $args \\
    -L ${fastq[0]} \\
    -o ${meta.id}.fastq.gz

echo "${meta.md5_1}  ${meta.id}.fastq.gz" > ${meta.id}.fastq.gz.md5
md5sum -c ${meta.id}.fastq.gz.md5

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    curl: \$(echo \$(curl --version | head -n 1 | sed 's/^curl //; s/ .*\$//'))
END_VERSIONS
"""
NextFlow From line 23 of local/sra_fastq_ftp.nf 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
"""
curl \\
    $args \\
    -L ${fastq[0]} \\
    -o ${meta.id}_1.fastq.gz

echo "${meta.md5_1}  ${meta.id}_1.fastq.gz" > ${meta.id}_1.fastq.gz.md5
md5sum -c ${meta.id}_1.fastq.gz.md5

curl \\
    $args \\
    -L ${fastq[1]} \\
    -o ${meta.id}_2.fastq.gz

echo "${meta.md5_2}  ${meta.id}_2.fastq.gz" > ${meta.id}_2.fastq.gz.md5
md5sum -c ${meta.id}_2.fastq.gz.md5

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    curl: \$(echo \$(curl --version | head -n 1 | sed 's/^curl //; s/ .*\$//'))
END_VERSIONS
"""
NextFlow From line 38 of local/sra_fastq_ftp.nf 
21
22
23
24
25
26
27
28
29
30
31
32
"""
echo $id > id.txt
sra_ids_to_runinfo.py \\
    id.txt \\
    ${id}.runinfo.tsv \\
    $metadata_fields

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    python: \$(python --version | sed 's/Python //g')
END_VERSIONS
"""
NextFlow From line 21 of local/sra_ids_to_runinfo.nf 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
"""
head -n 1 `ls ./samplesheets/* | head -n 1` > samplesheet.csv
for fileid in `ls ./samplesheets/*`; do
    awk 'NR>1' \$fileid >> samplesheet.csv
done

head -n 1 `ls ./mappings/* | head -n 1` > id_mappings.csv
for fileid in `ls ./mappings/*`; do
    awk 'NR>1' \$fileid >> id_mappings.csv
done

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    sed: \$(echo \$(sed --version 2>&1) | sed 's/^.*GNU sed) //; s/ .*\$//')
END_VERSIONS
"""
NextFlow From line 18 of local/sra_merge_samplesheet.nf 
17
18
19
20
21
22
23
24
25
26
"""
sra_runinfo_to_ftp.py \\
    ${runinfo.join(',')} \\
    ${runinfo.toString().tokenize(".")[0]}.runinfo_ftp.tsv

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    python: \$(python --version | sed 's/Python //g')
END_VERSIONS
"""
NextFlow From line 17 of local/sra_runinfo_to_ftp.nf 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
"""
synapse \\
    -c $config \\
    get \\
    $args \\
    $meta.id

echo "${meta.md5} \t ${meta.name}" > ${meta.id}.md5

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    synapse: \$(synapse --version | sed -e "s/Synapse Client //g")
END_VERSIONS
"""
NextFlow synapseclient From line 23 of local/synapse_get.nf 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
"""
synapse \\
    -c $config \\
    list \\
    $args \\
    $id \\
    $args2 \\
    > ${id}.list.txt

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    syanpse: \$(synapse --version | sed -e "s/Synapse Client //g")
END_VERSIONS
"""
NextFlow synapseclient From line 22 of local/synapse_list.nf 
17
18
19
20
21
22
23
24
25
26
27
"""
head -n 1 `ls ./samplesheets/* | head -n 1` > samplesheet.csv
for fileid in `ls ./samplesheets/*`; do
    awk 'NR>1' \$fileid >> samplesheet.csv
done

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    sed: \$(echo \$(sed --version 2>&1) | sed 's/^.*GNU sed) //; s/ .*\$//')
END_VERSIONS
"""
NextFlow From line 17 of local/synapse_merge_samplesheet.nf 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
"""
synapse \\
    -c $config \\
    show \\
    $args \\
    $id \\
    $args2 \\
    > ${id}.metadata.txt

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    synapse: \$(synapse --version | sed -e "s/Synapse Client //g")
END_VERSIONS
"""
NextFlow synapseclient From line 22 of local/synapse_show.nf 
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
"""
export NCBI_SETTINGS="\$PWD/${ncbi_settings}"

fasterq-dump \\
    $args \\
    --threads $task.cpus \\
    --outfile $outfile \\
    ${key_file} \\
    ${sra.name}

pigz \\
    $args2 \\
    --no-name \\
    --processes $task.cpus \\
    *.fastq

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    sratools: \$(fasterq-dump --version 2>&1 | grep -Eo '[0-9.]+')
    pigz: \$( pigz --version 2>&1 | sed 's/pigz //g' )
END_VERSIONS
"""
NextFlow From line 28 of fasterqdump/main.nf
ShowHide 21 more snippets with no or duplicated tags.

Login to post a comment if you would like to share your experience with this workflow.

Do you know this workflow well? If so, you can request seller status , and start supporting this workflow.

Free

Created: 1yr ago
Updated: 1yr ago
Maitainers: public
URL: https://nf-co.re/fetchngs
Name: fetchngs
Version: 1.10.0
Badge:
workflow icon

Insert copied code into your website to add a link to this workflow.

Downloaded: 0
Copyright: Public Domain
License: None
  • Future updates

Related Workflows

psychip_snakemake
Show Details View Workflow
ENCODE pipeline for histone marks developed for the psychENCODE project
public

public

psychip pipeline is an improved version of the ENCODE pipeline for histone marks developed for the psychENCODE project. The o...
raw sequence reads Alignment Sequence alignment report macs2 ucsc-bedclip bedGraphToBigWig BEDTools BWA Picard SAMtools Snakemake
Free
74
ncov_2
Show Details View Workflow
Near-real time tracking of SARS-CoV-2 in Connecticut
public

public

Repository containing scripts to perform near-real time tracking of SARS-CoV-2 in Connecticut using genomic data. This pipeli...
JSON nextclade Augur Biopython FOCUS Pandas Snakemake bs4 epiweeks geopy matplotlib numpy pycountry pycountry-convert uszipcode
Free
28
cellranger-snakemake-gke
Show Details View Workflow
snakemake workflow to run cellranger on a given bucket using gke.
public

public

A Snakemake workflow for running cellranger on a given bucket using Google Kubernetes Engine. The usage of this workflow ...
macs2 ucsc-bedclip bedGraphToBigWig BEDTools BWA Picard SAMtools Snakemake
Free
74
atlas
Show Details View Workflow
ATLAS - Three commands to start analyzing your metagenome data
public

public

Metagenome-atlas is a easy-to-use metagenomic pipeline based on snakemake. It handles all steps from QC, Assembly, Binning, t...
raw sequence reads Genome assembly Annotation track checkm2 gunc prodigal snakemake-wrapper-utils MEGAHIT Atlas BBMap Biopython BioRuby Bwa-mem2 cd-hit CheckM DAS Diamond eggNOG-mapper v2 MetaBAT 2 Minimap2 MMseqs MultiQC Pandas Picard pyfastx SAMtools SemiBin Snakemake SPAdes SqueezeMeta TADpole VAMB CONCOCT ete3 gtdbtk h5py networkx numpy plotly psutil utils metagenomics
Free
175
rna-seq-star-deseq2
Show Details View Workflow
RNA-seq workflow using STAR and DESeq2
public

public

This workflow performs a differential gene expression analysis with STAR and Deseq2. The usage of this workflow is described ...
raw sequence reads Gene expression profiling ID list snakemake-wrapper-utils biomaRt PCAtools BiocParallel BWA Cutadapt DESeq2 GFFutils MultiQC Pandas SAMtools Snakemake STAR cli stringr tidyverse RNA-Seq
Free
41
dna-seq-gatk-variant-calling
Show Details View Workflow
This Snakemake pipeline implements the GATK best-practices workflow
public

public

This Snakemake pipeline implements the GATK best-practices workflow for calling small germline variants. The usage of thi...
VCF raw sequence reads Variant calling genetic variants gatk rust-bio-tools snakemake-wrapper-utils tabix BCFtools BWA FastQC MultiQC Pandas Picard SAMtools Snakemake Trimmomatic Variant Effect Predictor (VEP) common matplotlib numpy seaborn DNA
Free
57