Downloading data in bulk
2026-11-16
Source:vignettes/download_data_in_bulk.Rmd
download_data_in_bulk.RmdDownloading data in bulk
This tutorial will follow a Flattened Data Layout. With a project that has this example layout:
.
├── biospecimen_experiment_1
│ ├── fileA.txt
│ └── fileB.txt
├── biospecimen_experiment_2
│ ├── fileC.txt
│ └── fileD.txt
├── single_cell_RNAseq_batch_1
│ ├── SRR12345678_R1.fastq.gz
│ └── SRR12345678_R2.fastq.gz
└── single_cell_RNAseq_batch_2
├── SRR12345678_R1.fastq.gz
└── SRR12345678_R2.fastq.gz
Tutorial Purpose
In this tutorial you will:
- Download all files/folder from a project
- Control manifest CSV generation during download
- Download all files/folders for a specific folder within the project
- Loop over all files/folders on the project/folder object instances
1. Download all files/folder from a project
First let’s set up some constants we’ll use in this script
library(synapser)
synLogin()
# Create some constants to store the paths to the data
DIRECTORY_TO_SYNC_PROJECT_TO <- path.expand(file.path("~", "my_ad_project"))
FOLDER_NAME_TO_SYNC <- "biospecimen_experiment_1"
DIRECTORY_TO_SYNC_FOLDER_TO <- file.path(
DIRECTORY_TO_SYNC_PROJECT_TO, FOLDER_NAME_TO_SYNC
)Finally we’ll sync the project from synapse to your local machine
# We'll set the `if_collision` to `keep.local` so that we don't overwrite any files.
project |> synSyncFromSynapse(path = DIRECTORY_TO_SYNC_PROJECT_TO, if_collision = "keep.local")
# Print out the contents of the directory where the data was synced to
# Explore the directory to see the contents have been recursively synced.
print(list.files(DIRECTORY_TO_SYNC_PROJECT_TO))While syncing your project you’ll see results like:
[syn74583648:My uniquely named project about Alzheimer's Disease]: Syncing Project from Synapse.
[syn74584000:biospecimen_experiment_1]: Syncing Folder from Synapse.
[syn74584007:single_cell_RNAseq_batch_2]: Syncing Folder from Synapse.
[syn74584001:biospecimen_experiment_2]: Syncing Folder from Synapse.
[syn74584006:single_cell_RNAseq_batch_1]: Syncing Folder from Synapse.
[syn74584146]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/biospecimen_experiment_1/fileB.png
[syn74584154]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/biospecimen_experiment_2/fileD.png
[syn74584155]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/biospecimen_experiment_2/fileC.png
[syn74584188]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/single_cell_RNAseq_batch_1/SRR12345678_R1.fastq.png
[syn74584147]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/biospecimen_experiment_1/fileA.png
[syn74584206]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/single_cell_RNAseq_batch_2/SRR12345678_R1.fastq.png
[syn74584189]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/single_cell_RNAseq_batch_1/SRR12345678_R2.fastq.png
[syn74584207]: Downloaded to <your_DIRECTORY_TO_SYNC_PROJECT_TO>/single_cell_RNAseq_batch_2/SRR12345678_R2.fastq.png
Downloading files: 100%|████████████████████| 1.31M/1.31M [00:02<00:00, 606kB/s]
2. Control manifest CSV generation during download
By default (manifest = "all"),
synSyncFromSynapse() writes a manifest.csv
into every synced directory. The manifest.csv is interoperable with synSyncToSynapse(),
the Synapse UI download cart, and synDownloadListFiles().
Use manifest = "root" to write a single manifest at the
root path, or manifest = "suppress" to skip manifest
generation entirely.
project |> synSyncFromSynapse(
path = DIRECTORY_TO_SYNC_PROJECT_TO,
if_collision = "keep.local",
manifest = "root"
)
print(list.files(DIRECTORY_TO_SYNC_PROJECT_TO))3. Download all files/folders for a specific folder within the project
Following the same set of steps let’s sync a specific folder
folder <- Folder(name = FOLDER_NAME_TO_SYNC, parent_id = project$id)
folder |> synSyncFromSynapse(path = DIRECTORY_TO_SYNC_FOLDER_TO, if_collision = "keep.local")
print(list.files(path.expand(DIRECTORY_TO_SYNC_FOLDER_TO)))While syncing your folder you’ll see results like:
Syncing Folder (syn53205630:experiment_notes) from Synapse.
Syncing Folder (syn53205632:notes_2022) from Synapse.
Syncing Folder (syn53205631:notes_2023) from Synapse.
[1] "notes_2022" "notes_2023"
You’ll notice that no files are downloaded. This is because the
client will see that you already have the content within this folder and
will not attempt to download the content again. If you were to use an
if_collision of "overwrite.local" you would
see that when the content on your machine does not match Synapse the
file will be overwritten.
4. Loop over all files/folders on the project/folder object instances
Using synSyncFromSynapse() will load into memory the
state of all Folders and Files retrieved from Synapse. This will allow
you to loop over the contents of your container.
for (folder_at_root in project$folders) {
print(sprintf("Folder at root: %s", folder_at_root$name))
for (file_in_root_folder in folder_at_root$files) {
print(sprintf("File in %s: %s", folder_at_root$name, file_in_root_folder$name))
}
for (folder_in_folder in folder_at_root$folders) {
print(sprintf("Folder in %s: %s", folder_at_root$name, folder_in_folder$name))
for (file_in_folder in folder_in_folder$files) {
print(sprintf("File in %s: %s", folder_in_folder$name, file_in_folder$name))
}
}
}The result of traversing some of your project structure should look like:
[1] "Folder at root: experiment_notes"
[1] "Folder in experiment_notes: notes_2022"
[1] "File in notes_2022: fileA.txt"
[1] "File in notes_2022: fileB.txt"
[1] "Folder in experiment_notes: notes_2023"
[1] "File in notes_2023: fileC.txt"
[1] "File in notes_2023: fileD.txt"
[1] "Folder at root: single_cell_RNAseq_batch_1"
[1] "File in single_cell_RNAseq_batch_1: SRR12345678_R1.fastq.gz"
[1] "File in single_cell_RNAseq_batch_1: SRR12345678_R2.fastq.gz"
[1] "File in single_cell_RNAseq_batch_1: SRR92345678_R1.fastq.gz"
[1] "File in single_cell_RNAseq_batch_1: SRR92345678_R2.fastq.gz"
[1] "Folder at root: single_cell_RNAseq_batch_2"
[1] "File in single_cell_RNAseq_batch_2: SRR12345678_R1.fastq.gz"
[1] "File in single_cell_RNAseq_batch_2: SRR12345678_R2.fastq.gz"