Simons Genome Diversity Project (SGDP)

Last update: Thu Apr 26 16:57:22 EDT 2018 (by Shop Mallick)

This points to data for 300 public SGDP samples across 142 diverse populations.

Update history:
Thu Apr 26 16:57:22 EDT 2018: bug in phased data identified (see section E)
Mon Apr 9 09:03:21 EDT 2018: EGA signed release form updated
Wed Jan 24 14:56:58 EST 2018: Pointers to mappability filters (section F3)
Thu Jan 11 10:23:40 EST 2018: Fermikit contig pointer added (section D)
Mon Apr 3 17:39:13 EDT 2017: Template PDF for signed letter provided (section Z)
Sat Feb 18 00:16:43 EST 2017: 21 signed letter vcfs (variants only) made available directly by request (following signed letter agreement)
Thu Jan 12 15:21:12 EST 2016: 21 signed letter vcfs made available directly for easy access (following signed letter agreement)
Wed Dec 7 16:01:12 EST 2016: Variant only VCFs tar ball built: public samples directly available, pointers to signedLetter samples given
Oct 6 2016: bams for Y-chromosomes available
Oct 6 2016: phased data available

Latest version:

Version 2: The dataset has been reprocessed using the newer, more accurate "bwa mem" algorithm. Data are available in several forms:


(A) Samples, raw data, alignments, genotypes

* Metadata for the samples is here: SGDP_metadata.279public.21signedLetter.samples.txt
* Raw data for 279 genomes for which the informed consent documentation is consistent with fully public data release are available through the EBI European Nucleotide Archive under accession numbers PRJEB9586 and ERP010710.
* For the remaining 21 genomes (designated by code Y in the seventh column of Supplementary Data Table 1), data are deposited at the European Genome-phenome Archive (EGA), which is hosted by the EBI and the CRG, under accession number EGAS00001001959. See section (Z) below for instructions.


(AA) BAMS for Y-chromosomes only

* Y-chromosome bams are separately available from here: https://sharehost.hms.harvard.edu/genetics/reich_lab/sgdp/Y-bams/Y.tar (This is 129Gb).

(AB) VCFs of variants of publicly available samples

* 279 variant only vcfs are available from here: https://sharehost.hms.harvard.edu/genetics/reich_lab/sgdp/vcf_variants/vcfs.variants.public_samples.279samples.tar (This is 57Gb).

We provide 263 C-panel fully public samples (as reported in Supplementary Information section 3) and 16 B-panel fully public samples for a total of 279 vcfs in a single tarball. Vcfs are bgzipped and provided with tbi files, with md5sums for both. Variant only vcfs are extracted from EMIT_ALL_SITES vcfs (which contain non-variant sites and are considerably bigger) which are intended to be available from EVA (validation pending). Note: These have been annotated with the results of Nick Patterson's filtering engine in the FL field, which should be helpful for variant analysis.

(AC) VCFs of variants of signed letter available samples

Variant only vcfs are available upon request and submission of signed letter (as described in (Z) Requestors of Signed Letter samples: instructions).

(B) Compact: SGDP-lite and Ctools

Compact versions of the SGDP dataset and software for accessing it are available to give speedy access to variant/non-variant data in a dataset of only ~140Gb.

An important development to facilitate fast analysis of genomic data for large numbers of samples has been to push data into a novel format ("hetfa"), where both variant and non-variant sites are represented by an IUB encoding following genotyping. This enables extremely quick access to both single-site and region data. This has been further packaged into "SGDP-lite" (written by Nick Patterson), along with the results of a novel filtering engine to allow users to balance reliability of calls with coverage.

(B1) SGDP-lite
These data are available from here:
Latest version:
v3: http://sharehost.hms.harvard.edu/genetics/reich_lab/cteam_lite_public3.tar

Previous versions:
v2: http://sharehost.hms.harvard.edu/genetics/reich_lab/old_versions/cteam_lite_public2.tar

(The size of SGDP-lite makes it possible for this to be quickly transferred, compared with the full dataset which is >70 Tbytes making it possibly impractical for some users both in terms of transfer costs and storage size).

(B2) Ctools
To accompany this, a powerful engine for population genetics manipulation of SGDP-lite, called "Ctools" (written by Nick Patterson and further developed by Mengyao Zhao), is available: https://github.com/DReichLab/cTools>


(C) STR genotypes

The short tandem repeat (STR) genotypes are available through dbVar under accession number nstd128 (http://www.ncbi.nlm.nih.gov/dbvar). If you simply wish to access variants (and non-variants) easily, SGDP-lite is recommended (see (a)).


(D) Fermikit

Described in SI4.
.. unitig variants are available from https://github.com/lh3/sgdp-fermi
.. contigs are available from https://www.ncbi.nlm.nih.gov/assembly/GCA_000786075.2/

(E) Phased genotypes

... are available from: https://sharehost.hms.harvard.edu/genetics/reich_lab/sgdp/phased_data

[Thu Apr 26 16:57:22 EDT 2018] Note: a bug in the processing chain indicates that heterozygous sites that occur in positions without a chimpanzee allele are incorrectly assigned homozygous reference state.
This artficially increases the size of some homozygous chunks and can affect some analysis; for example, it is known to inflate recent effective population sizes in MSMC estimates. We are working to rebuild these data.

(F,G,H) Filters and tools

(F1) Sample specific filters: bam2cnv:

... is available from: bam2cnv

(F2) Sample independent filters (for these samples)

... is available from: universal_mask

(F3) Generic mappability filters

For generating the map35_50% mask, please follow the procedure described here: http://lh3lh3.users.sourceforge.net/snpable.shtml

(G) vcf2hetfa:

... is available from: vcf2hetfa.pl

(H) filtstats:

... is available from (as part of the cTools package): filtstats


(Z) Requestors of Signed Letter samples: instructions

BAMS: Data for these 21 genomes can be obtained by submitting to the EGA (European Genome-phenome Archive) Data Access Committee a dated, signed letter (in pdf form only) containing the following text: (a) I will not distribute the data outside my collaboration; (b) I will not post the data publicly; (c) I will make no attempt to connect the genetic data to personal identifiers for the samples; and (d) I will not use the data for any commercial purposes. A template PDF is here: https://sharehost.hms.harvard.edu/genetics/reich_lab/sgdp/docs/EGA_DAA__SGDP_2018.pdf
Please send BAM requests to: Michelle Lee (Michelle_Lee at hms.harvard.edu) or to EGA directly (accession number EGAS00001001959)

VCFs: VariantOnly: available on request, requiring a signed letter as described above. Please send VCF requests to: Michelle Lee (Michelle_Lee at hms.harvard.edu), copying Shop Mallick (shop at genetics.med.harvard.edu) . VCFs for public samples are available directly (see Section AB above).

Previous versions:

Version 1 (outdated):
First release of genotypes are available, built using the "bwa aln" algorithm from the Simons Foundation are here: http://www.simonsfoundation.org/life-sciences/simons-genome-diversity-project-dataset/ (includes some experimentally phased samples)

Dataset size is ~10Tb. Reprints and permissions information is available at www.nature.com/reprints. Correspondence and requests for materials should be addressed to S.M. (shop@genetics.med.harvard.edu) or D.R. (reich@genetics.med.harvard.edu).

Acknowledgements

This project was funded by the Simons Foundation.