Data Request

Raw sequence data is not available from the VAMPS website. To request original raw data files from projects that are currently on vamps please note the following options:

Tag Generation/Sequence Processing

In October of 2018, we switched from using the Invitrogen Platinum Taq DNA Polymerase High Fidelity (Cat. No. 11304102) to the Invitrogen Platinum SuperFi DNA Polymerase (Cat. #12351-050). High Fidelity polymerase offers 6X fidelity vs Taq, whereas SuperFi polymerase offers 100X fidelity vs Taq. Modified PCR master mix recipe is as follows:

• 91.25 uL of water
• 25 uL of 5X SuperFi Buffer
• 2.5 uL of 10 mM dNTP Mix
• 1.25 uL of Platinum SuperFi Polymerase
• 4 uL of 10 uM Fusion Primer Mix
• 1 uL of template*

• 94°C for 3 min
• 30 cycles of:
• 94°C for 30 sec
• 57°C for 45 sec
• 72°C for 1 min
• 72°C for 2 min
• Hold at 4°C

In 2013 we switched to using Illumina platforms for 16S sequencing. Similar to the strategy for 454, we use fusion primers composed of the Illumina adaptors, multiplexing identifiers, and domain-specific primers. Thermocycling and reaction mixtures are different from 454 sequencing for Illumina amplicon PCR.

For Archaeal V6:

• 94C for 2 min
• 30 cycles of (94C for 30sec, 60C for 45sec, 72C for 1min)
• 72C for 2 min
• Hold at 4C

For Bacterial and Archaeal V4V5:

• 94C for 2 min
• 30 cycles of (94C for 30sec, 57C for 45sec, 72C for 1min)
• 72C for 2 min
• Hold at 4C

Bacterial V6 uses first the domain-specific primers for 25 cycles, then the products are cleaned and used in a second 5-cycle PCR with fusion primers.

For Bacterial V6:

• 94C for 2 min
• 25 cycles of (94C for 30sec, 60C for 45sec, 72C for 1min)
• 72C for 2 min
• Hold at 4C
• Visualize and quantitate the amplicons in the BioAnalyzer or Caliper.
• Purify the amplicons using Qiagen MinElute columns.
• Use product in second PCR:
• 94C for 2 min
• 5 cycles of (94C for 30sec, 60C for 45sec, 72C for 1min)
• 72C for 2 min
• Hold at 4C

After we produce amplicons, we can clean and/or size-select for the target products using Agencourt AMPure XP beads. Then we quantitate products with an Invitrogen Picogreen assay, pool at desired concentrations (e.g. equimolar), and quantitate the final pool with qPCR.

Conserved sequences that flank the hypervariable V6-V4 region of rRNAs serve as primer sites to generate PCR amplicons. Each PCR reaction produces products that can be informatically identified using a unique "key" incorporated between the 454 Life Sciences primer A or B and the 5' flanking rRNA primer. The use of a 5-bp key allows for the synthesis of as many as 81 oligonucleotides that differ by at least two sites. Our multiplexing strategy allows the concurrent collection of 10,000-50,000 tags from each of 8-40 samples in a single nine- hour sequencing run without use of partitioning gaskets that reduce the number of sequencing wells on the PicoTiterPlate^TM. Amplicons can be pooled before the emPCR step and each pool is run on a large region of the plate.

454 Amplicon PCR (Christina Holmes/Ekaterina Andreishcheva) for four reactions:

• 96 ul water
• 13.4 ul 10X Platinum buffer
• 9.4 ul 50 mM MgSO4
• 2.6 ul 10 mM Pure Peak dNTPs
• 4 ul 10uM Fusion Primer A
• 4 ul 10uM Fusion Primer B
• 2.6 ul 2.5 U/ul Platinum HiFi Pol
• [2 ul template (~5-25 ng)*]
• 33 ul total volume/reaction

*If template stock is dilute or otherwise resistant to amplification, more template can be added in place of water.

The 5 reactions are the three replicates of the environmental template, positive control, and negative control. Template (plasmid pool for positive control; water for negative control) is added as final step.

Program:

• 94C for 2 min
• 30 cycles of (94C for 30sec, 60C for 45sec, 72C for 1min)
• 72C for 2 min
• Hold at 4C
• Visualize and quantitate the amplicons in the BioAnalyzer or Caliper.
• Purify the amplicons using Agencourt AMPure XP beads as described in 454 Sequencing Technical Bulletin 2011-007, and resuspend in 100 uL of Buffer EB.
• Store purified amplicons at -20C.

Supplies:

• DNA 1000 Kit, Agilent, 5067-1504, 25 chips. $372
• Agencourt AMPure XP, 60 mL, A63881. $1050
• PurePeak DNA Polymerization Mix 10 mM 1 ml, Pierce/ThermoFisher, NU606001, $115
• Platinum HiFi Taq polymerase plus buffer and MgSO4, Invitrogen.

Sequencing Pipeline

Our sequencing pipeline is public and the details are stored on GitHub. The links are here:

• https://github.com/annaship/py_mbl_sequencing_pipeline
• https://github.com/annaship/py_mbl_sequencing_pipeline/blob/master/pipeline/illumina_files.py

The Steps in order are: Demultiplexing, Merging, Uniqueing, Chimera Checking and Taxonomy Assignment

We use Meren's scripts to perform some of these steps. You can find them here on his GitHub site: https://github.com/merenlab/illumina-utils/tree/master/scripts

1. Demultiplexing
• https://github.com/merenlab/illumina-utils/blob/master/scripts/iu-demultiplex


2. Merging (with Quality Control)
• From https://github.com/merenlab/illumina-utils/tree/master/scripts
• overlap: iu-merge-pairs
• filter mismatches: iu-filter-merged-reads
• for v6 only: iu-trim-V6-primers


3. Uniqueing
• https://gist.github.com/meren/abd5c6f014da9aa554a4


4. Chimera Checking
• We use vsearch for chimera checking (combining result from both ref and denovo)
• [sample ref:] vsearch -uchime_ref ACAGTG_ACTGC_1_MERGED-MAX-MISMATCH-3.unique.chg -db rRNA16S.gold.fasta -uchimeout ACAGTG_ACTGC_1_MERGED-MAX-MISMATCH-3.unique.chimeras.db -chimeras ACAGTG_ACTGC_1_MERGED-MAX-MISMATCH-3.unique.chimeras.db.chimeric.fa -strand plus -notrunclabel
• [sample denovo:] vsearch -uchime_denovo TTAGGC_NNNNCGACG_1_MERGED-MAX-MISMATCH-3.unique.chg -uchimeout TTAGGC_NNNNCGACG_1_MERGED-MAX-MISMATCH-3.unique.chimeras.txt -chimeras TTAGGC_NNNNCGACG_1_MERGED-MAX-MISMATCH-3.unique.chimeras.txt.chimeric.fa -notrunclabels


5. Taxonomy Assignment (using GAST)
• gast_ill (https://github.com/annaship/BPC-scripts/blob/master/gast_ill)
• You also will need Taxonomy.pm and taxonomy reference database files.

Quality Control

• There are several steps in the quality control process. Some of these steps are tied up in other pipeline steps (See Sequencing Pipeline above).
• While demultiplexing we check if barcodes and indexes are the same as in provided metadata, if not reads are discarded.
• Merging read 1 and read 2 into fasta files. At this point we also discard low quality reads. Primers and barcodes are striped out.
• For part of it "iu-merge-pairs" and "iu-filter-merged-reads" Meren's utilities are used. (See Sequencing Pipeline above)
• Filters used during merging (sequence will be discarded if it fails any of these tests):
1. Is the prefix correct for read_1 and read_2?
2. P value? (P value is the ratio of the number of mismatches and the length of the overlap. Merged sequences can be discarded based on this ratio)
3. Does the read have less than the maximum number of mismatches?
4. Does the read have any 'N's?
5. Better than Q30? (Phred quality score, if Phred assigns a Q score of 30 (Q30) to a base, this is equivalent to the probability of an incorrect base call 1 in 1000 times)
6. More than minimum expected overlap?


• Default filter values:

"P" value .................................................   0.300000
Maximum number of mismatches in the overlapped region ......  None
Minimum overlap size .......................................  15
Minimum Q-score for mismatches .............................  15
Q30 enforced? ..............................................  True
                

• Chimera checking using vsearch, both denovo and reference based.

What is GAST?

GAST stands for Global Alignment for Sequence Taxonomy.

It uses a reference database of SSU sequences to determine the taxonomy of hypervariable region tags. The specifics are described in the citation below.

• GAST Code on GitHub
• Taxonomy.pm perl module
• Available GAST-formatted SSU Reference Sets.

Huse SM, Dethlefsen L, Huber JA, Welch DM, Relman DA, Sogin ML (2008) 
Exploring Microbial Diversity and Taxonomy Using SSU rRNA Hypervariable Tag Sequencing. 
PLoS Genet 4(11): e1000255. https://doi.org/10.1371/journal.pgen.1000255        
          

Exporting (Downloading) Data

There is no raw sequence data available on VAMPS.

The primary way to export data from VAMPS is to use the 'Download Data' page which is accessed from the main 'Sample Selection' page. After you've selected the datasets you want to include in your download select the blue 'Download Data' button.
You can download data from VAMPS in various formats. Listed on the download page are the formats (with descriptions) available to you.

• Fasta (3 formats available)
• Counts Matrix (2 formats available)
• Metadata (2 formats available)
• Biom (Modified JSON format)

Importing Data

Description of Import Options

Fasta files and count matrix files can be uploaded to VAMPS as long as they conform to the correct format which is described on the upload page for each type.

Multiple-Dataset fasta file

Metadata file (for projects already in VAMPS)

This is a csv (comma separated values) file but the data are actually separated by <TABS> to make it more human readable. These files must conform to the qiime mapping file format. Since these are processed (trimmed) data that are being uploaded, the 'BarcodeSequence' and 'LinkerPrimerSequence' fields can be left empty but the header names have to be present. The #SampleID field must be present and there can be no duplicate sample names. The Description field must also be present and must be the last field. There is a handy 'validate_mapping_file.py' script available in qiime to assist you with this file.

Phyloseq and ggplot2

What is Phyloseq?

Phyloseq is an R library for microbiome data: (https://joey711.github.io/phyloseq/).
VAMPS uses R (https://www.r-project.org/) and Python (https://www.python.org/) scripts to produce some of the visualizations.

To use the Phyloseq library with your VAMPS data download the three Phyloseq files from the 'Display Choices' Page:
Import them directly into R (or R-script) as shown below to create a Phyloseq Object:

• biom_file <- 'phyloseq.biom'
• tax_file <- 'phyloseq_taxonomy.txt'
• map_file <- 'phyloseq_metadata.txt'
• library(phyloseq)
• library(ggplot2)
• TAX<-as.matrix(read.table(tax_file, header=TRUE, sep = "\t", row.names = 1, as.is=TRUE))
• OTU <- import_biom(biom_file)
• MAP <- import_qiime_sample_data(map_file)
• TAX <- tax_table(TAX)
• OTU <- otu_table(OTU)
• physeq <- phyloseq(OTU,TAX,MAP)

See the Phyloseq website for more help and examples.

What is ggplot2

Reference Databases

How are the reference databases created?

We create Ref16S, a reference database of aligned full-length sequences based on all available sequences in SILVA exported using the ARB software. New updates to both SILVA and RDP are incorporated as they become available.

• We flag as low-quality and delete all sequences with a sequence quality score <= 50, an alignment score <= 50 or a pintail (chimera) score <= 40.
• We flag as redundant and delete all exact copies of the full-length sequence.
• We classify all bacterial and archaeal sequences directly with the Ribosomal Database Project Classifier (RDP). We used only RDP classifications with a bootstrap value of >=80%. If the bootstrap value was <80%, the taxonomic assignment was moved to a higher classification level until an 80% or better bootstrap value was achieved. For example, if the genus assignment had a bootstrap value of 70%, but the family had a value of 85%, that sequence would be assigned only as far as family and not to genus. RDP Classifier does not classify sequences below the genus level.
• We incorporate other taxonomy sources, such as Entrez Genome accession numbers or researcher knowledge of specific entries, as they become available. These "other sources" are used preferentially over the RDP for bacteria and archaea. RDP does not classify eukaryotes. For eukaryota taxonomies, we use the EMBL taxonomy from the SILVA database where we do not have other sources.
  
• We create hypervariable region specific databases (RefV6, RefV3, RefV9, etc.).
  • For each hypervariable region we calculate the Ref16S alignment coordinates.
  • We then excise from the Ref16s aligned sequences the section corresponding to the hypervariable region.
  • The gaps are removed from the aligned sequences to create a set of unaligned sequences. Any hypervariable sequences that contain an 'N' are deleted. Any hypervariable sequences shorter than 50 nt are deleted. Any full-length sequences that were not sequenced all the way through the specific hypervariable region are deleted. Two reference IDs are assigned to each reference hypervariable region. A ref16s_id (previously alt_local_gi) links the hypervariable sequence with its source full-length sequence, and a second, e.g., refv6_id (previously known as local_gi) is used to identify all entries having the exact same sequence of the hypervariable region. The taxonomy is carried directly from the full-length source.
  • Unique reference hypervariable regions sequences are exported to a blastable database for use in the assignment of taxonomy to pyrosequencing reads through GAST.

Sample Submission Process

https://vamps2.mbl.edu/metadata/metadata_new

Metadata

Required metadata fields:

File and Name Formats

Project Name

Dataset Name

FASTA File

Metadata File

• QIIME style mapping file: This file MUST have a header named '#SampleID', 'sample_name', 'dataset' or 'dataset_name' as the first column name.
  • The Header line is required
  • The columns must be tab delimited (unless commas are required by the specific page)
  • There must be one row for each dataset

TaxBySeq File and Metadata from Old (legacy) VAMPS

• See VAMPS Exports
• Download the TaxBySeq and Metadata files to your harddrive.
• Import them as a pair either directly (still compressed) or uncompressed.

JSON Configuration File

• Required to be valid JSON (try https://jsonlint.com)
• Requires "source":"VAMPS*"
• Requires id_name_hash.ids: [ ] --A valid list of dataset ids that you have permissions to view.
• Optional:
  1. normalization  (Valid values: "none", "maximum", "frequency")
  2. selected_distance  (Valid values: "morisita-horn", "jaccard", "kulczynski", "canberra" or "bray-curtis")
  3. tax_depth  (Valid values: one of "domain", "phylum", "klass", "order", "family", "genus", "species", "strain")
  4. include_nas  (Valid values: "yes" or "no")
  5. domains  (Valid values: any or all of "Archaea","Bacteria","Eukarya","Organelle","Unknown")
  6. min_range  (Valid values: integers 0-99)
  7. max_range  (Valid values: integers 1-100)
• Image -- (optional) To render a table, chart or other visual element automatically on upload try:
"image":"dheatmap" (added at same level as "source")
possible values: "dheatmap", "piecharts", "barcharts", "counts_matrix", "metadata_table", "fheatmap", "dendrogram01", "dendrogram03", "pcoa", "pcoa3d", "geospatial" or "adiversity"

SAMPLE:             
{
  "source":"VAMPS",
  "post_items":
  { 
    "normalization":"maximum",
    "selected_distance":"morisita_horn",
    "tax_depth":"phylum",
    "domains":["Archaea","Bacteria","Eukarya","Organelle","Unknown"],
    "include_nas":"yes",
    "min_range":0,
    "max_range":100
  },
    "id_name_hash":
    {
      "ids":["49","50","51","52"]
    }
}
              

Definitions

Project

Dataset

FASTA Files

FASTA definition line (or defline)