Is deduplication for you?
Re: @brianbushnell on seqanswers:Deduplication of reads is possible with dedupe, but it requires a lot of memory (~1kb per read) so it is only practical with HiSeq data if you have high-memory nodes. If you have a reference, deduplication based on mapping will use much less memory, though it will also be much slower. And whether or not deduplication is a good idea depends on the experiment; i.e., probably not for quantitative RNA-seq, and also probably not for unamplified libraries in general.

Suggested order:
1. Initial quality control BGI did this
2. Quality and/or length-based reads discarding; trimming/discarding of N-containing reads BGI did this
3. Adapter removal BGI did this
4. phix/contaminant removal
5. [optional] Error-correction --use ecc.sh
6. [optional] Deduplication (depending on experiment) --use dedupe.sh
7. Merging of PE reads --use bbmerge.sh
8. Quality trimming + phix/contaminant removal --use bbduk.sh
9. Contamination check

More advice:
As for the order of quality trimming and merge, they could be done either way. But, BBMerge internally performs quality-trimming for reads that don't merge prior to quality trimming, so for that particular program, quality-trimming after merging tends to be better.

Note that after merging, you will have 2 sets of reads (merged and unmerged) that must be processed independently, since not all of them will successfully merge

Also useful reading about merging your paired end reads before assembly:
http://thegenomefactory.blogspot.com/2012/11/tools-to-merge-overlapping-paired-end.html

Okay, time to try...

Step 1: Quality trimming

#quality control trimming of paired-end reads

#DR086_E.bombifrons_MOE quality control and plots

bbduk.sh -Xmx1g in1=DR086_Erbom_MOE_1.fq in2=DR086_Erbom_MOE_2.fq out1=DR086_Erbom_MOE_1_clean.fq out2=DR086_Erbom_MOE_2_clean.fq minlen=25 qtrim=rl trimq=10 qhist=qc_plots/DR086_Erbom_MOE_qhist.txt aqhist=qc_plots/DR086_Erbom_MOE_aqhist.txt lhist=qc_plots/DR086_Erbom_MOE_readLength.txt bqhist=qc_plots/DR086_Erbom_MOE_bqhist.txt overwrite=true

About 8-9% of each read was trimmed.

Normal level sequence duplication in mammals is 20 million reads

Normal sequence bias at beginning of reads due to nonrandom hybridization of random primers.

#I put this all in a shell script

sh BGI_qc.sh >log.txt

I tried to do error correction but these files are HUGE!! Need to remove duplicates first.

sh ecc.sh in1=DR086_Erbom_MOE_1.fq in2=DR086_Erbom_MOE_2.fq out1=DR086_Erbom_MOE_1_corrected.fq out2=DR086_Erbom_MOE_2_corrected.fq

/usr/local/bin/bbmap//calcmem.sh: line 111: [: unlimited: integer expression expected

java -ea -Xmx-410m -Xms-410m -cp /usr/local/bin/bbmap/current/ jgi.KmerNormalize bits=16 ecc=t passes=1 keepall dr=f prefilter in1=DR086_Erbom_MOE_1.fq in2=DR086_Erbom_MOE_2.fq out1=DR086_Erbom_MOE_1_corrected.fq out2=DR086_Erbom_MOE_2_corrected.fq

Invalid maximum heap size: -Xmx-410m

Could not create the Java virtual machine.

#even when I increased the memory, no cigar

sh ecc.sh in1=DR086_Erbom_MOE_1.fq in2=DR086_Erbom_MOE_2.fq out1=DR086_Erbom_MOE_1_corrected.fq out2=DR086_Erbom_MOE_2_corrected.fq -Xmx20g

Exception in thread "Thread-60" java.lang.NoClassDefFoundError: java/util/concurrent/ThreadLocalRandom

        at jgi.KmerNormalize$ProcessThread.run(KmerNormalize.java:2807)

Step 2: Deduplication

dedupe.sh in=NBS1170_Desrot_MOE_1_clean.fq out=NBS1170_Desrot_MOE_1_clean_dedupe.fq am ac fo renameclusters=f storename=f pto cc qin=33 csf=stats.txt dot=graph.dot -Xmx20g overwrite=t

dedupe.sh in=NBS1170_Desrot_MOE_2_clean.fq out=NBS1170_Desrot_MOE_2_clean_dedupe.fq am ac fo renameclusters=f storename=f pto cc qin=33 csf=NBS1170_Desrot_MOE_2_clean_dedupe_stats.txt dot=NBS1170_Desrot_MOE_2_clean_dedupe_graph.dot -Xmx20g overwrite=t

Step 3: Merge Paired End Reads

#merge PE 

bbmerge.sh in1=NBS1170_Desrot_MOE_1_clean.fq.gz in2=NBS1170_Desrot_MOE_2_clean.fq.gz out=NBS1170_Desrot_MOE_merged.fq -Xmx20g

Step 3.5: Could also remove the duplicates after merged paired ends

#remove duplicates of the two lanes

dedupe.sh in=NBS1170_Desrot_MOE_merged.fq out=NBS1170_Desrot_MOE_merged_dedupe.fq am ac renameclusters=f storename=f pto cc qin=33 csf=dedupe_logs/NBS1170_Desrot_MOE_merged_dedupe_stats.txt dot=dedupe_logs/NBS1170_Desrot_MOE_merged_dedupe_stats_graph.dot -Xmx20g overwrite=t

Compilation function of each BBmap shell scripts:

addadapters.sh

Randomly adds adapters to a file, or grades a trimmed file.

bbcountunique.sh

Generates a kmer uniqueness histogram, binned by file position.

There are 3 columns for single reads, 6 columns for paired:

count      number of reads or pairs processed

r1_first   percent unique 1st kmer of read 1

r1_rand    percent unique random kmer of read 1

r2_first   percent unique 1st kmer of read 2

r2_rand    percent unique random kmer of read 2

pair       percent unique concatenated kmer from read 1 and 2

bbduk.sh

Compares reads to the kmers in a reference dataset, optionally 

allowing an edit distance. Splits the reads into two outputs - those that 

match the reference, and those that don't. Can also trim (remove) the matching 

parts of the reads rather than binning the reads. Can use for quality trimming.

bbduk2.sh

Compares reads to the kmers in a reference dataset, optionally 

allowing an edit distance. Splits the reads into two outputs - those that 

match the reference, and those that don't. Can also trim (remove) the matching 

parts of the reads rather than binning the reads.

bbest.sh

Calculates EST (expressed sequence tags) capture by an assembly from a sam file.

Designed to use BBMap output generated with these flags: k=13 maxindel=100000 custom tag ordered

bbfakereads.sh

Generates fake read pairs from ends of contigs or single reads.

bbmap.sh

Fast and accurate short-read aligner for DNA and RNA.

bbmapskimmer.sh

Fast and accurate short-read aligner for DNA and RNA. (not sure difference from above)

bbmask.sh

Masks sequences of low-complexity, or containing repeat kmers, or covered by mapped reads.

bbmerge.sh

Merges paired reads into single reads by overlap detection.

With sufficient coverage, can also merge nonoverlapping reads using gapped kmers.

bbmergegapped.sh

Merges paired reads into single reads by overlap detection.

With sufficient coverage, can also merge nonoverlapping reads using gapped kmers.

bbnorm.sh

Normalizes read depth based on kmer counts.

Can also error-correct, bin reads by kmer depth, and generate a kmer depth histogram.

bbqc.sh

Performs quality-trimming; artifact, human, and phiX removal; adapter-trimming; error-correction and normalization. Designed for Illumina fragment libraries only.

bbrename.sh

Renames reads to <prefix>_<number> where you specify the prefix and the numbers are ordered.

bbsplit.sh

Maps reads to multiple references simultaneously.

Outputs reads to a file for the reference they best match, with multiple options for dealing with ambiguous mappings.

bbsplitpairs.sh

Separates paired reads into files of 'good' pairs and 'good' singletons by removing 'bad' reads that are shorter than a min length.

Designed to handle situations where reads become too short to be useful after trimming.  This program also optionally performs quality trimming.

bbwrap.sh

Wrapper for BBMap to allow multiple input and output files for the same reference.

calcmem.sh

????

calctruequality.sh

Calculates the observed quality scores from a sam file.

callpeaks.sh

No description--but appears related to keeping only certain reads within a certain region of histogram.

countbarcodes.sh

Counts the number of reads with each barcode.

countgc.sh

Counts GC content of reads or scaffolds.

crosscontaminate.sh

Generates synthetic cross-contaminated files from clean files.

Intended for use with synthetic reads generated by RandomReads.

cutprimers.sh

Cuts out sequences corresponding to primers identified in sam files.

decontaminate.sh

Decontaminates multiplexed assemblies via normalization and mapping.

dedupe.sh

Accepts one or more files containing sets of sequences (reads or scaffolds).

Removes duplicate sequences, which may be specified to be exact matches, subsequences, or sequences within some percent identity. Can also find overlapping sequences and group them into clusters.

dedupe2.sh

Accepts one or more files containing sets of sequences (reads or scaffolds).

Removes duplicate sequences, which may be specified to be exact matches, subsequences, or sequences within some percent identity.Can also find overlapping sequences and group them into clusters. (Not sure difference between dedupe.sh)

demuxbyname.sh

Demultiplexes reads based on their name (suffix or prefix) into multiple files.

ecc.sh

Corrects substitution errors in reads using kmer depth information.

Can also normalize and/or bin reads by kmer depth.

filterbarcodes.sh

Filters barcodes by quality, and generates quality histograms.

filterbycoverage.sh

Filters an assembly by contig coverage.

filterbyname.sh

Filters reads by name.

getreads.sh

Gets reads by number. The first read (or pair) has ID 0, the second read (or pair) has ID 1, etc.

grademerge.sh

Grades correctness of merging synthetic reads with headers generated by RandomReads and re-headered by RenameReads

gradesam.sh

Grades mapping correctness of a sam file of synthetic reads with headers generated by RandomReads3.java

idmatrix.sh

Generates an identity matrix via all-to-all alignment.

khist.sh

Generates a histogram of kmer counts for the input reads or assemblies.

Can also normalize, error-correct, and/or bin reads by kmer depth.

kmercount.sh

--counts kmers?

kmercountexact.sh

Counts the number of unique kmers in a file.

makechimeras.sh

Makes chimeric PacBio reads from nonchimeric reads.

mapPacBio.sh

Fast and accurate short-read aligner for DNA and RNA.

mapPacBio8k.sh

Description:  Fast and accurate short-read aligner for DNA and RNA.To index:   bbmap.sh ref=<reference fasta>

To map:     bbmap.sh in=<reads> out=<output sam>

To map without writing an index:

    bbmap.sh ref=<reference fasta> in=<reads> out=<output sam> nodisk

mapnt.sh

Maps sequences to the nt database.

matrixtocolumns.sh

Turns identity matrices into 2-column format for plotting.

mergeOTUs.sh

Merges coverage stats lines (from pileup) for the same OTU,

according to some custom naming scheme.

megebarcodes.sh

Concatenates barcodes and quality onto read names.

msa.sh

Aligns a query sequence to reference sequences.

Outputs the best matching position per reference sequence.

If there are multiple queries, only the best-matching query will be used.

phylip2fasta.sh

Calculates per-scaffold coverage information from an unsorted sam file.

pileup.sh

Calculates per-scaffold coverage information from an unsorted sam file.

printtime.sh

Prints time elapsed since last called on the same file.

randomreads.sh

Generates random synthetic reads from a reference genome.  Read names indicate their genomic origin. Allows precise customization of things like insert size and synthetic mutation type, sizes, and rates. Read names generated by this program are used by MakeRocCure (samtoroc.sh) and GradeSamFile (gradesam.sh). They can also be used by BBMap (bbmap.sh) and BBMerge (bbmerge.sh) to automatically calculate true and false positive rates, if the flag 'parsecustom' is used.

readlength.sh

Generates a length histogram of input reads.

reformat.sh

Reformats reads to change ASCII quality encoding, interleaving, file format, or compression format.

removehuman.sh

Removes all reads that map to the human genome with at least 95% identity after quality trimming.

removesmartbell.sh

Remove Smart Bell adapters from PacBio reads

repair.sh

Re-pairs reads that became disordered or had some mates eliminated. (not repair)

rqcfilter.sh

Performs quality-trimming, artifact removal, linker-trimming, adapter trimming, and spike-in removal using BBDukF. Performs human contaminant removal using BBMap.

samtoroc.sh

Creates a ROC curve from a sam file of synthetic reads with headers generated by RandomReads3.java

seal.sh

Performs high-speed alignment-free sequence quantification,

by counting the number of long kmers that match between a read and

a set of reference sequences.  Designed for RNA-seq with alternative splicing.

shuffle.sh

Reorders reads randomly.

stats.sh

Generates basic assembly statistics such as scaffold count, N50, L50, GC content, gap percent, etc.

staswraapper.sh

Runs stats.sh on multiple assemblies to produce one ouput line per file.

synthmda.sh

Generates synthetic reads following an MDA-amplified singe cell's coverage distribution.

testformat.sh

Tests the format of a sequence file based on name and contents.

textfile.sh

Translates nucleotide sequences to all 6 amino acid frames.