18  Reproducible, publishable results

A result is only as good as your ability to show how you got it. Reviewers, readers and your own future self need to know which data, software and settings produced each number. This chapter shows what to record, how to describe it in a paper, how to share the data, and how to check that an analysis gives the same answer when it is run again.

18.1 The question

Could someone else, or you in a year, get exactly the same result from the same data? And can you write down, for a paper, everything that was done?

18.2 What to record

Sandve and colleagues give ten simple rules for reproducible computational research (Sandve et al. 2013). Three of them matter most for a pipeline:

• Rule 1: “For Every Result, Keep Track of How It Was Produced”.
• Rule 3: “Archive the Exact Versions of All External Programs Used”.
• Rule 6: “For Analyses That Include Randomness, Note Underlying Random Seeds”.

For an epigenomics analysis, this means recording:

• the input data: which files, and their checksums (a fingerprint of a file’s content, such as MD5);
• the reference: the genome assembly, the gene annotation and the blacklist, with their versions (Chapter 3);
• the software: every tool and package, with its version;
• the settings: every parameter of every step;
• the commands: what was actually run, in which order.

A versions list alone is not enough. Tools depend on other software, which can change too. So a complete record also fixes the whole software environment, for example as a container image with a unique digest.

NoteIn EpiNexus 1.2

Every run writes its own record. EpiNexus runs each analysis in one software image, built from fixed versions of the operating system packages, R and Bioconductor packages and Python packages. The run records the image’s digest, so the same image can be used again. It also fixes the order of the aligner’s output, and the random seeds of the steps that use random numbers.

The record is part of the GEO submission package (see below), in the folder provenance/.

18.3 The record of one run

Table 18.1 lists what the GEO package of the heart run (Chapter 14) holds.

Table 18.1: The run record in the provenance/ folder of a GEO submission package (run cafc042f).

File	What it holds
tool_versions.json	Every tool and package version (for example Bowtie2 2.5.4, MACS3 3.0.4, DESeq2 1.50.2), the EpiNexus version (1.2.0) and the software image’s digest
environment.txt	The full list of installed packages in the image (868 lines), and the dates of the package snapshots it was built from
commands.tsv	All 129 commands that ran, with their start time, duration and exit code
params.effective.json, settings.json	Every parameter and setting the run used
samplesheet.csv, inputs.json	The samples, and where their files came from (here, GEO and ENA accessions with checksums)
reference.json	The genome build, the annotation and the blacklist

18.4 Writing the methods

EpiNexus writes a methods text for each run, from this record. It names each tool with its version and each setting that mattered. For the heart run it begins:

8 ChIP-seq libraries were processed (single-end) against Mouse (mm10/GRCm38). … Adapters and low-quality bases were trimmed with Trim Galore (v0.6.10) using Cutadapt (v4.7) … Reads were aligned with Bowtie2 (v2.5.4).

The text is a draft, not the final word. Read it, check that it says what you did, and add what EpiNexus cannot know: the biology, the samples, and any steps you did outside EpiNexus.

NoteIn EpiNexus 1.2

The methods text is in the run’s report (Reports page) and in the GEO package (methods.md). Exported figures are 300 dpi. The report states the EpiNexus version and the software image used.

18.5 Sharing the data: GEO

Journals usually ask you to deposit sequencing data in a public archive such as GEO (Barrett et al. 2013). GEO needs the raw reads, the processed files and a metadata sheet that describes each sample.

NoteIn EpiNexus 1.2

On the Export page, GEO Submission Package builds a zip file for one run. It holds:

• metadata.tsv: the GEO sheet, filled in from the run where possible. Fields EpiNexus cannot know are marked [fill in].
• processed/: peaks, count tables, differential results, annotation and more, with MD5 checksums in md5sums.txt.
• methods.md, qc_table.tsv and the provenance/ folder.
• linked_files.tsv: download links for the large signal tracks, valid for 24 hours.

Raw FASTQ files are not included: upload the files you sequenced yourself. Samples that you imported from GEO or SRA are marked as a “reanalysis” of the original sample. The package’s README explains that GEO asks submitters not to upload raw data that is already in SRA.

The same page also offers a shell script with the run’s commands and a JSON file with all its parameters (Reproducibility Files).

Figure 18.1: The lower part of the Export page for the heart run (cafc042f): the GEO Submission Package and the Reproducibility Files (shell script, parameters and methods text). EpiNexus 1.2 interface; not covered by the book’s licence.

18.6 Running an analysis again

The best test of reproducibility is to run the analysis again and compare the results.

NoteIn EpiNexus 1.2

On a finished run’s page, Run again starts a new run with the same samples and settings, using the current software (Chapter 4). The new run gets its own record, so the two can be compared. Before it starts, a dialog says what will be the same and what may differ (Figure 18.2).

Figure 18.2: The dialog that Run again opens. EpiNexus 1.2 interface; not covered by the book’s licence.

The heart dataset was run twice: first with EpiNexus 1.1.0 (run 93a23cb5), then again with 1.2.0 (run cafc042f). We compared the processed files of the two GEO packages (Table 18.2).

Table 18.2: The processed files of two runs of the same data, with EpiNexus 1.1.0 and 1.2.0.

Files	Same in both runs?	Why
Peaks (4), reproducible peaks (2), peak annotation (4)	Yes, byte for byte	These steps did not change
Super-enhancers (4), enhancer–gene links (4)	Yes, byte for byte	These steps did not change
GO enrichment of each sample’s peaks (4), QC table	Yes, byte for byte	These steps did not change
Motifs in all peaks (AME)	Same results; family names replaced	Version 1.2 takes the motif families from JASPAR
Differential tables, GO of the changed regions	No	Version 1.2 changed how H3K27ac regions are defined and tested
Motifs in changed regions, pooled regions	New in 1.2	New analyses
MultiQC report	No	It names its own run, and it lists the new pooled peak call as an extra sample. Three fingerprint scores (the CHANCE values) differ by less than 0.3 %

Twenty-three files were identical, byte for byte, although the runs used different versions of EpiNexus and different machines. Apart from the MultiQC report, the files that changed are those of the steps that version 1.2 changed. This is what a reproducible pipeline should do: the same data and the same method give the same result, and a change in the result can be traced to a change in the method.

WarningPitfall: re-running with new software without saying so

If you run an analysis again with a newer version, some results may change. Report which version produced the results in your paper, and keep the record of each run.

WarningPitfall: editing results by hand

Every manual step, such as deleting rows from a table in a spreadsheet, is a step that no record holds. Sandve and colleagues’ second rule is “Avoid Manual Data Manipulation Steps” (Sandve et al. 2013). Filter and sort with tools that keep a record, or describe each manual step in your methods.

18.7 Further reading

• Sandve et al. (2013): ten simple rules for reproducible computational research.
• Barrett et al. (2013): the GEO database.

Barrett, Tanya, Stephen E. Wilhite, Pierre Ledoux, Carlos Evangelista, Irene F. Kim, et al. 2013. ‘NCBI GEO: Archive for Functional Genomics Data Sets—Update’. Nucleic Acids Research 41 (D1): D991–95. https://doi.org/10.1093/nar/gks1193.

Sandve, Geir Kjetil, Anton Nekrutenko, James Taylor, and Eivind Hovig. 2013. ‘Ten Simple Rules for Reproducible Computational Research’. PLoS Computational Biology 9 (10): e1003285. https://doi.org/10.1371/journal.pcbi.1003285.