taxonomic_profiling_pipeline

public public 1yr ago 0 bookmarks

This pipeline was developed using the Snakemake workflow management system

You would need to have the Snakefile, the env folder and its contents (YAML files with environment d

Code Snippets

  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
import os
import re
from collections import deque

# ranks that krakefaction needs for lineage
TAXA_RANKS = ['d','p','c','o','f','g','s', 's1']

"""
# =============================================================================
# =============================================================================
"""

"""
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Class - TaxonNode
------
PURPOSE
-------
This class represents a node of a taxonomic level, with links to subtaxa and 
supertaxon.
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
"""
class TaxonNode:

    def __init__(self, clade_count, taxa_count, rank, taxid, name, supertaxon=None):
        self.clade_count = int(clade_count) # num of db entries of this + subtaxa
        self.taxa_count = int(taxa_count)   # num of db entries of this exact taxa
        self.rank = rank.lower()
        self.taxid = taxid
        self.name = name.strip().replace(' ', '_')
        self.supertaxon = supertaxon        # link to supertaxon node
        self.subtaxa = []                   # list of links to subtaxa nodes
        self.subtaxa_sum = int(taxa_count)
        self.lineage = ''

    def create_lineage(self):
        """Returns appropriate lineage string, based on ancestors"""
        # first case (domain)
        if self.rank == 'd':
            return 'd__' + self.name
        elif self.rank in TAXA_RANKS:
            return self.supertaxon.lineage + '|{}__{}'.format(self.rank, self.name)
        elif self.supertaxon:
            return self.supertaxon.lineage
        else:
            return ''

    def add_child(self, child):
        """Add a subtaxon child node to this node
           Add the new node's taxa_count to the subtaxa_sum of all supertaxa"""
        # add to parent's subtaxa list
        self.subtaxa.append(child)
        # add parent as child's supertaxa
        child.supertaxon = self

        if child.taxa_count != 0:
            # add tax_count to parent's sum
            self.subtaxa_sum += child.taxa_count

            curr_parent = self
            # add to any further ancestor's sums
            while curr_parent.supertaxon is not None:
                curr_parent = curr_parent.supertaxon
                curr_parent.subtaxa_sum += child.taxa_count

        # now that child has parent, create lineage
        child.lineage = child.create_lineage()

        return

    def has_full_subtaxa(self):
        """Return boolean whether all subtaxa have been added to this node"""
        return self.subtaxa_sum == self.clade_count

    def __str__(self):
        s = 'Taxon name: {} - Rank {}\n'.format(self.name, self.rank)
        s += '\tLineage: {}\n'.format(self.lineage)
        s += '\tTaxa ID: {}\n'.format(self.taxid)
        s += '\tClade count: {}\n'.format(self.clade_count)
        s += '\tTaxa count: {}\n'.format(self.taxa_count)
        s += '\tSubtaxa sum: {}\n'.format(self.subtaxa_sum)
        if self.supertaxon is not None:
            s += '\tSupertaxa: {}\n'.format(self.supertaxon.name)
        else:
            s += '\tSupertaxa: None\n'
        s += '\tSubtaxa: {}\n'.format(' '.join([subtaxon.name for subtaxon in self.subtaxa]))
        return s

# def get_superlineage(node):
#     """Recursively finds the most recent lineage string from ancestors"""
#     # if supertaxon's lineage is not an empty string, return it
#     if node.supertaxon.lineage:
#         return node.supertaxon.lineage
#     # Else, recurse
#     else:
#         return get_superlineage(node.supertaxon)

"""
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
"""

"""
# =============================================================================
MAIN
# =============================================================================
"""
def main():
    """Verify inputs, then run script"""
    # get inspection filename and filtered kraken output file paths from snakemake
    db_inspection, infile_paths = get_inputs()
    run(db_inspection, infile_paths)

"""
# =============================================================================
# =============================================================================
"""

def get_inputs():
    """Get the inputs from snakemake.
       Database inspection file, and paths to filtered kraken output files"""
    # retrieve the name of the inspection file from snakemake
    db_inspection = snakemake.input[0]

    # retrieve the paths of filtered files from snakemake
    infile_paths = snakemake.input[1:]

    # verifiy files
    check_file_existence(db_inspection, infile_paths)

    return(db_inspection, infile_paths)


def check_file_existence(db_ins, in_paths):
    """Validate file existence, raising error upon failure"""
    # check database inspection file
    if not os.path.isfile(db_ins):
        raise RuntimeError(
            "ERROR: could not open database inspection file {}\n".format(db_ins))

    # check each kraken input file
    for f in in_paths:
        if not os.path.isfile(f):
            raise RuntimeError(
                "ERROR: could not open input file {}\n".format(f))

    return

"""
# =============================================================================
"""
def run(db_inspection, infile_paths):
    """Main logical control of the script occurs within"""
    taxon_dict = get_taxa_dict(db_inspection)

    for infile in infile_paths:
        f = open(infile, 'r')
        reads = f.readlines()
        f.close()
        create_outfile(reads, taxon_dict, infile)
"""
# =============================================================================
"""    

def get_taxa_dict(db_ins):
    """Produce a taxonomy dictionary with taxids as keys and lineage strings as values"""
    # get database inspection file as list of lines
    with open(db_ins, 'r') as f:
        inspection_lines = f.readlines()

    # create the taxon tree
    root = create_tree(inspection_lines)

    # create the taxid/lineage dictionary
    taxa_dict = {}
    build_taxa_dict(taxa_dict, root)

    return taxa_dict


def create_tree(inspection_lines):
    """Create the taxonomy tree and return the root node"""
    # make deque for leftpopping of inspection lines
    inspection_lines = deque(inspection_lines)

    # create root node
    root_line = inspection_lines.popleft().split('\t')
    root_node = TaxonNode(*root_line[1:])

    # stack for tracking nodes with more subtaxa to add
    stack = []
    stack.append(root_node)

    # while there are still lines to make nodes from
    while inspection_lines:
        # get the node at the top of the stack
        curr_node = stack[-1]
        new_node = TaxonNode(*inspection_lines.popleft().split('\t')[1:])

        # if current node has full subtaxa list, pop stack until otherwise
        if curr_node.has_full_subtaxa():
            curr_node = find_next_parent(stack)

        # add the new node to the parent node
        curr_node.add_child(new_node)
        # put new node on top of stack if it still needs subtaxa
        if not new_node.has_full_subtaxa():
            stack.append(new_node)


    return root_node


def find_next_parent(stack):
    """Recursively pops stack until a node that still needs children is found"""
    node = stack[-1]
    if not node.has_full_subtaxa():
        return stack[-1]
    else:
        stack.pop()
        return find_next_parent(stack)


def build_taxa_dict(t_dict, node):
    """Recursively add taxid/lineage from tree to taxa dictionary"""
    # add current node if it has lineage
    if node.lineage:
        t_dict[str(node.taxid)] = node.lineage
    else:
        # if it has no lineage, it is root or subroot
        t_dict[str(node.taxid)] = 'root'
    for sub in node.subtaxa:
        build_taxa_dict(t_dict, sub)


def create_outfile(reads, t_dict, filename):
    """Creates the formatted output table file"""
    with open(filename.replace('filtered', 'translated'), 'w') as f:
        for line in reads:
            columns = line.split('\t')
            # data to be first entry in output file
            datum = columns[1]
            # find the taxid
            tax_search = re.search('(?<=taxid )(\d*)', columns[2])
            taxid = tax_search.group(1)
            if taxid in t_dict.keys():
                #print(datum + '\t' + t_dict[taxid])
                if line == reads[-1]:
                    f.write(datum + '\t' + t_dict[taxid])
                else:
                    f.write(datum + '\t' + t_dict[taxid] + '\n')
            #else:
               # l.debug('Unable to translate: {}'.format(columns[2]))


def print_tree(root):
    """Prints out tree top down"""
    print(root)
    for sub in root.subtaxa:
        print_tree(sub)

    return

if __name__ == '__main__':
    main()
48
49
shell:
    'fastp -i {input.fwd} -I {input.rev} -o {output.fwd} -O {output.rev} --html {output.html} --json {output.json} --thread {threads}'
63
64
shell:
    '(bowtie2 -p {threads} -x phiX -1 {input.fwd} -2 {input.rev} --un-conc-gz results/unmapped/{wildcards.sample}_R%_unmapped.fastq.gz) > /dev/null 2> {log}'
SnakeMake From line 63 of main/Snakefile
75
76
shell:
    'fastqc {input} --outdir=results/fastqc'
83
84
shell:
    "multiqc -o results -n multiqc_report.html {input}"
100
101
102
103
104
105
106
107
108
109
110
111
shell:
    "kraken2 "
    "--db {params.db} "
    "--threads {threads} "
    "--output {output.kraken_class} "
    "--confidence {params.confidence} "
    "--minimum-base-quality {params.base_qual} "
    "--report {output.kraken_report} "
    "--paired "
    "--use-names "
    "--gzip-compressed "
    "{input.fwd} {input.rev}"
119
120
121
shell:
    """ktUpdateTaxonomy.sh
    ktImportTaxonomy -m 3 -t 5 {input} -o {output}"""
136
137
shell:
    "cat {input} | grep -v \"{params.filter_string}\" > {output}"
SnakeMake From line 136 of main/Snakefile
147
148
shell:
    "kraken2-inspect --db {input} > {output}"
SnakeMake From line 147 of main/Snakefile
158
159
script:
    "scripts/kraken2-translate.py"
SnakeMake From line 158 of main/Snakefile
168
169
shell:
    "krakefaction -u {input.untrans} -t {input.trans} -o {output}"
SnakeMake From line 168 of main/Snakefile
ShowHide 5 more snippets with no or duplicated tags.

Login to post a comment if you would like to share your experience with this workflow.

Do you know this workflow well? If so, you can request seller status , and start supporting this workflow.

Free

Created: 1yr ago
Updated: 1yr ago
Maitainers: public
URL: https://github.com/BeeCSI-Microbiome/taxonomic_profiling_pipeline
Name: taxonomic_profiling_pipeline
Version: 1
Badge:
workflow icon

Insert copied code into your website to add a link to this workflow.

Downloaded: 0
Copyright: Public Domain
License: None
  • Future updates

Related Workflows

cellranger-snakemake-gke
snakemake workflow to run cellranger on a given bucket using gke.
A Snakemake workflow for running cellranger on a given bucket using Google Kubernetes Engine. The usage of this workflow ...