DOC_ID : P04-0002
Editor : Anita
Reviewer : Lincat

Background

QIIME 2 is a powerful, extensible, and decentralized microbiome analysis package with a focus on data and analysis transparency. QIIME 2 enables researchers to start an analysis with raw DNA sequence data and finish with publication-quality figures and statistical results. 

                                                                                                                         Learn more   

Environment requirement

• Hardware : CPU : 10 cores
   RAM : 45 Gb
• Software :The pipeline is developed based on GA System. All software used in the pipeline are included in pre-defined software environments (GA application collection ). Please install suitable environment before start the pipeline(How to do it!) .

Reference library

• Classifier reference  BuildNameContentQIIME2v2020.6gg-13-8-99-515-806-nb-classifier.qzasilva-138-99-515-806-nb-classifier.qza

Project folder structure

• Project Name                  project.sh
                q2manifest.txt
                   q2metadata.tsv
  • Raw
                   SingleEnd: SampleName_R1.fastq.gz
                   PairedEnd: SampleName_R1.fastq.gz, SampleName_R2.fastq.gz         
  • Processed
                  rep-seq-dada2.qza/qzv
                  table-dada2.qza
                  taxonomy.qza/qzv
                  feature-table.biom               …
  • QC
                fastqc: sample_fastqc.html / sample_fastqc.json
                qualimap : sample folder
  • Log
                Summary.log
                service_id.ER, OU
  • Report
               MultiQC : multiqc_report.html
               core-metrics-results
               Alpha_diversity_indices.qza
               taxa-bar-plots.qzv
               feature-table.tsv             …*:Files prepared by users are marked in Red

Manifest File

Pair-end manifest file :

Description : 
User have to create manifest file first. Manifest file described sample identifiers to  absolute filepaths  of raw reads file (fastq.gz or .fastq). The manifest file should indicates the direction of the reads in each raw read file. The manifest format is Metadata-compatible, user can re-use the manifest file to bootstrap Sample Metadata, too.The following example illustrates a simple fastq manifest file for paired-end read data for four samples. The fastq.gz absolute filepaths may contain environment variables $HOME or $PWD (e.g. $PWD/raw/S1_R1.fastq.gz ). 

Ref : https://docs.qiime2.org/2020.6/tutorials/importing/#fastq-manifest-formats

Manifest format 1

Manifest format 2

►Download demo file: q2manifest.txt 

Single-end manifest file :

Just like with fastq.gz, the absolute filepaths in the manifest for any fastq files must be accurate. The following example illustrates a simple fastq manifest file for fastq single-end read data for two samples.

Metadata :

Description :
Metadata provides the key information of each sample. User should manually define sample metadata in metadata file before beginning a QIIME 2 analysis. The ID column is essential and the first column of  metadata file. The ID must be unique and also list in manifest file too. User can all kinds of factor associated with sample ID. These factor would be used in following taxonomical, statistical and diversity analysis.  You have to add   comment directive to declare column types in your factor. The comment directive must appear directly below the header. The row’s first cell must be #q2:types to indicate the row is a comment directive. Subsequent cells may contain the values categorical or numeric (both case-insensitive). Your metadata file can be tested by online tools in official web site.

Reference:  ver. 2020.6 [ref : https://docs.qiime2.org/2020.6/tutorials/metadata/]

ex .

QIIME 2 metadata的檔案格式為.tsv，檔案內第一個欄位”#SampleID”是必須的，其餘欄位可依user自行配置填寫，欄位間需用”Tab”鍵做間隔。

 ►Download demo file: q2metadata.tsv 

The diagram of workflow

QIIME2_DPP_PE : QIIME2_DPP_PE perform Data import, denoise, dereplicate, and filters chimeras of paired-end sequences.(Ref)

QIIME2_TA : We use taxonomy classifiers to determine the closest taxonomic affiliation with some degree of confidence or consensus (which may not be a species name if one cannot be predicted with certainty!), based on alignment, k-mer frequencies, etc. (Ref)

QIIME2_DA : QIIME 2 analysis allows the use of phylogenetic trees for diversity metrics such as Faith’s Phylogenetic Diversity and UniFrac distance as well as feature-based analyses in Gneiss. The tree provides an inherent structure to the data, allowing us to consider an evolutionary relationship between organisms. Alpha diversity measures the level of diversity within individual samples. Beta diversity measures the level of diversity or dissimilarity between samples. (Ref)

PICRUSt2 :  PICRUSt2 (Phylogenetic Investigation of Communities by Reconstruction of Unobserved States) is a software for predicting functional abundances based only on marker gene sequences. (Ref)

FAPROTAX : FAPROTAX is a manually constructed database that maps prokaryotic taxa (e.g. genera or species) to metabolic or other ecologically relevant functions (e.g. nitrification, denitrification or fermentation), based on the literature on cultured representatives. (Ref)

Module list

Workflow script configuration

Color theme of Script 
Gray(Dark Gray): Script comment
Red (Pale Red):  module name
Green (Dark Emerald):  Variables defined by use

The workflow script demonstrate genome variant calling pipeline implemented by using QIIME2, PICRUSt2 and FAPROTAX modules. The work script are included several functional section. The detail of each section show below.

1. Parameter setup : 

# *****************************************
# *****       Global parameters     ******
# *****************************************
# set your email for job message senting
email = user@your.email

# set your project account for submit job
projectID = MST-XXXXXXXX  

2. Data preprocessing : 

QIIME2_DPP_PE perform Data import, denoise, dereplicate, and filters chimeras of paired-end sequences.

Ref : https://docs.qiime2.org/2020.6/tutorials/overview/

# ****************************************************
# ******              Data preprocessing           ******
# ***SampleN_R1,R2 => rep-seq-dada2.qza ***
# **** SampleN_R1,R2 => table-dada2.qza ****
# *** SampleN_R1,R2 => taxonomy.qza/qzv ***
# ****************************************************

# ===== Data preprocessing =====
QIIME2_DPP_PE=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt‘, metaData=“q2metadata.tsv”‘ QIIME2_DPP_PE.2.X.Y.sh)
echo -e $(date)’\tQIIME2_DPP_PE\t’$QIIME2_DPP_PE’\tSubmit’ >> log/Summary.log

# ===== (!!Adjust denoise parameters!!) Data preprocessing =====
QIIME2_DPP_PE=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt‘, metaData=“q2metadata.tsv”,denoise=220:220:0:0:2.0:2.0:2:pooled‘ QIIME2_DPP_PE.2.X.Y.sh)
echo -e $(date)’\tQIIME2_DPP_PE\t’$QIIME2_DPP_PE’\tSubmit’ >> log/Summary.log

3. Diversity analysis : 

QIIME 2 analysis allows the use of phylogenetic trees for diversity metrics such as Faith’s Phylogenetic Diversity and UniFrac distance as well as feature-based analyses in Gneiss. The tree provides an inherent structure to the data, allowing us to consider an evolutionary relationship between organisms. 
Alpha diversity measures the level of diversity within individual samples. Beta diversity measures the level of diversity or dissimilarity between samples.

Ref : https://docs.qiime2.org/2020.6/tutorials/overview/

# ****************************************************
# *******              Diversity analysis           *******
# ****** rep-seq-dada2.qza => *.qza/ *.qzv ******
# ******** table-dada2.qza => *.qza/ *.qzv ********
# ****************************************************

# ===== Diversity analysis =====
QIIME2_DA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=‘q2manifest.txt’, metaData=‘q2metadata.tsv’‘ QIIME2_DA.2.X.Y.sh)
echo -e $(date)’\tQIIME2_DA\t’$QIIME2_DA’\tSubmit’ >> log/Summary.log

# ****************************************************
# *******              Diversity analysis           *******
# ****** rep-seq-dada2.qza => *.qza/ *.qzv ******
# ******** table-dada2.qza => *.qza/ *.qzv ********
# ****************************************************

# ===== (!!!Further analysis the column in metadata!!!) Diversity analysis =====
QIIME2_DA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=‘q2manifest.txt’, metaData=‘q2metadata.tsv’,fieldName=“Subject:sex”‘ QIIME2_DA.2.X.Y.sh)
echo -e $(date)’\tQIIME2_DA\t’$QIIME2_DA’\tSubmit’ >> log/Summary.log

4. Taxonomic analysis : 

We use taxonomy classifiers to determine the closest taxonomic affiliation with some degree of confidence or consensus (which may not be a species name if one cannot be predicted with certainty!), based on alignment, k-mer frequencies, etc. 

Ref : https://docs.qiime2.org/2020.6/tutorials/overview/

# ****************************************************
# *******             Taxonomy analysis          *******
# ****** rep-seq-dada2.qza => *.qza/ *.qzv ******
# ******** table-dada2.qza => *.qza/ *.qzv ********
# ****************************************************

# =====Taxonomy analysis =====
QIIME2_TA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=‘$(pwd)’/,sampleName=‘q2manifest.txt’, metaData=“q2metadata.tsv”‘ QIIME2_TA.2.X.Y.sh)
echo -e $(date)’\tQIIME2_TA\t’$QIIME2_TA’\tSubmit’ >> log/Summary.log

# =====(!!user self-provided classifier!!) Taxonomy analysis =====
QIIME2_TA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=‘$(pwd)’/,sampleName=‘q2manifest.txt’, metaData=“q2metadata.tsv”,classifier=userpath/AA-classifier.qza,classifierName=AA‘ QIIME2_TA.2.X.Y.sh)
echo -e $(date)’\tQIIME2_TA\t’$QIIME2_TA’\tSubmit’ >> log/Summary.log

5. Predicte gene abundances : 

PICRUSt2 (Phylogenetic Investigation of Communities by Reconstruction of Unobserved States) is a software for predicting functional abundances based only on marker gene sequences.

Ref : https://github.com/picrust/picrust2/wiki

# ****************************************************
# *******       Predicte gene abundances    *******
# ****** feature-table.biom => *.tsv.gz/ *.txt ******
# ****************************************************

# =====Predicte gene abundances =====
PICRUSt2=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v “projDir=‘$(pwd)’/,sampleName=‘dna-sequences.fasta’,tableBiom=‘feature-table.biom’” PICRUSt2.2.X.Y.sh)
echo -e $(date)’\tPICRUSt2\t’$PICRUSt2’\tSubmit’ >> log/Summary.log

6. Annotation gene marker and phenotype : 

FAPROTAX is a manually constructed database that maps prokaryotic taxa (e.g. genera or species) to metabolic or other ecologically relevant functions (e.g. nitrification, denitrification or fermentation), based on the literature on cultured representatives.Functions represented in FAPROTAX focus on marine and lake biogeochemistry, particularly sulfur, nitrogen, hydrogen and carbon cycling, although other functions (e.g. plant pathogeneicity) are also included.

Ref : http://www.loucalab.com/archive/FAPROTAX/lib/php/index.php?section=Instructions

# ****************************************************
# *** annotation gene marker and phenotype ***
# ****** *_taxonomy.biom => *.tsv.gz/ *.txt ******
# ****************************************************

# ===== annotation gene marker and phenotype =====
FPO=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v “projDir=‘$(pwd)’/,inFile=‘table-with-taxonomy.biom’,output=‘Case’” FAPROTAX.2.X.Y.sh)
echo -e $(date)’\tFAPROTAX\t’$FPO’\tSubmit’ >> log/Summary.log

Demo Workflow script :

#!/bin/bash

# ************* QIIME2 Microbiota analysis
# ************* Platform Taiwania
# ************* Script version 2.0 (Applies to modules from version 2.0.0)
## [projDir] project <= manifest.txt, metadata.tsv
##   +– [rawDir] raw <= Sample1,2,3_R1/_R2.fastq.gz
##   +– [prcDir] processed => *.qza, *.qzv
##   +– [anaDir] analyzed
##   +– [rptDir] report => *.qzv,*.tsv.gz, *.tsv, *.txt
##   +– [tmpDir] temp
##   +– [logDir] log => Summary.log
##   +– [qcDir] QC

## 1. ##
# *****************************************
# ******     Global parameters       ******
# *****************************************

# set your email for job message senting
email = user@your.email
# set your project account for submit job
projectID = MST-XXXXXXXX  

# *****************************************
# ************* raw read QC   *************
# *****************************************
#===== fastqc module =====
#Use Raw_Read_Quality_Check-Ver2 pipeline

## 2. ##
# *****************************************
# ******      Data preprocessing      *****
# *** SampleN_R1,R2 => rep-seq-dada2.qza***
# **** SampleN_R1,R2 => table-dada2.qza****
# *** SampleN_R1,R2 => taxonomy.qza/qzv****
# *****************************************

# ===== Data preprocessing =====
QIIME2_DPP_PE=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt’, metaData=”q2metadata.tsv”‘ QIIME2_DPP_PE.2.X.Y.sh)
echo -e $(date)‘\tQIIME2_DPP_PE\t’$QIIME2_DPP_PE’\tSubmit’ >> log/Summary.log

# ===== (!!Adjust denoise parameters!!) Data preprocessing =====
QIIME2_DPP_PE=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt’, metaData=”q2metadata.tsv”,denoise=220:220:0:0:2.0:2.0:2:pooled’ QIIME2_DPP_PE.2.X.Y.sh)
echo -e $(date)‘\tQIIME2_DPP_PE\t’$QIIME2_DPP_PE’\tSubmit’ >> log/Summary.log

## 3. ##
# *****************************************
# *******    Diversity analysis    ********
# *** rep-seq-dada2.qza => *.qza/ *.qzv ***
# **** table-dada2.qza => *.qza/ *.qzv ****
# *****************************************

# ===== Diversity analysis =====
QIIME2_DA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt’, metaData=’q2metadata.tsv” QIIME2_DA.2.X.Y.sh)
echo -e $(date)‘\tQIIME2_DA\t’$QIIME2_DA’\tSubmit’ >> log/Summary.log

# ===== (!!!Further analysis the column in metadata!!!) Diversity analysis =====
QIIME2_DA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt’, metaData=’q2metadata.tsv’,fieldName=”Subject:sex”‘ QIIME2_DA.2.X.Y.sh)
echo -e $(date)‘\tQIIME2_DA\t’$QIIME2_DA’\tSubmit’ >> log/Summary.log

## 4. ##
# *****************************************
# *******      Taxonomy analysis   ********
# *** rep-seq-dada2.qza => *.qza/ *.qzv ***
# **** table-dada2.qza => *.qza/ *.qzv ****
# *****************************************

# =====Taxonomy analysis =====
QIIME2_TA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt’, metaData=”q2metadata.tsv”‘ QIIME2_TA.2.X.Y.sh)
echo -e $(date)‘\tQIIME2_TA\t’$QIIME2_TA’\tSubmit’ >> log/Summary.log

# =====(!!user self-provided classifier!!) Taxonomy analysis =====
QIIME2_TA=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v ‘projDir=’$(pwd)‘/,sampleName=’q2manifest.txt’, metaData=”q2metadata.tsv”,classifier=userpath/AA-classifier.qza,classifierName=AA’ QIIME2_TA.2.X.Y.sh)
echo -e $(date)‘\tQIIME2_TA\t’$QIIME2_TA’\tSubmit’ >> log/Summary.log

## 5. ##
# ****************************************
# *****   Predicte gene abundances   *****
# **feature-table.biom => *.tsv.gz/*.txt**
# ****************************************

# =====Predicte gene abundances =====
PICRUSt2=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v “projDir=’$(pwd)‘/,sampleName=’dna-sequences.fasta’,tableBiom=’feature-table.biom'” PICRUSt2.2.X.Y.sh)
echo -e $(date)‘\tPICRUSt2\t’$PICRUSt2’\tSubmit’ >> log/Summary.log

## 6. ##
# ****************************************
# * annotation gene marker and phenotype *
# ** *_taxonomy.biom => *.tsv.gz/ *.txt **
# ****************************************

# =====annotation gene marker and phenotype =====
FPO=$(qsub -P $projectID -W group_list=$projectID -m e -M $email -v “projDir=’$(pwd)‘/,inFile=’table-with-taxonomy.biom’,output=’Case'” FAPROTAX.2.X.Y.sh)
echo -e $(date)‘\tFAPROTAX\t’$FPO’\tSubmit’ >> log/Summary.log

Resource

Demo data set :  QIIM2 Demo data set (ref)

Report template: Microbiome QIIME2 report Template (ref)