Skip to content

Commit 7b2b4ce

Browse files
fedorovfedorov
committed
Update to include AWS download instructions
1 parent 40e289a commit 7b2b4ce

File tree

1 file changed

+75
-105
lines changed

1 file changed

+75
-105
lines changed

‎notebooks/getting_started/part3_exploring_cohorts.ipynb‎

Lines changed: 75 additions & 105 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
{
22
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"id": "view-in-github",
7+
"colab_type": "text"
8+
},
9+
"source": [
10+
"<a href=\"https://colab.research.google.com/github/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/part3_exploring_cohorts.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
11+
]
12+
},
313
{
414
"cell_type": "markdown",
515
"metadata": {
@@ -23,7 +33,7 @@
2333
"---\n",
2434
"Initial version: Nov 2022\n",
2535
"\n",
26-
"Updated: "
36+
"Updated: May 2023"
2737
]
2838
},
2939
{
@@ -105,12 +115,7 @@
105115
"cell_type": "code",
106116
"execution_count": null,
107117
"metadata": {
108-
"colab": {
109-
"base_uri": "https://localhost:8080/",
110-
"height": 424
111-
},
112-
"id": "sVBOII5GP9cw",
113-
"outputId": "fc199d98-c22c-4d88-a89a-962698ddd4ee"
118+
"id": "sVBOII5GP9cw"
114119
},
115120
"outputs": [],
116121
"source": [
@@ -160,7 +165,7 @@
160165
"\n",
161166
"While defining your cohort, you may be looking for subsets of collections, patients or studies that meet your search critieria. But when it comes to downloading the cohort, you will always need to get the URLs of the individual files from the cohort.\n",
162167
"\n",
163-
"The following cell uses the query from the above to get the list of study identifiers that meet our search critieria, and then selects all of the rows correspoing to the DICOM instances (files)that are included in those studies. The column that contains the URI that can be used to download the file is in the `gcs_url` column. We will also query the `instance_size` column, which we can use to calculate the size of the files corresponding to the cohort.\n",
168+
"The following cell uses the query from the above to get the list of study identifiers that meet our search critieria, and then selects all of the rows correspoing to the DICOM instances (files)that are included in those studies. The column that contains the URI that can be used to download the file from Google GCS bucket is in the `gcs_url` column, while the location of that same file in Amazon AWS bucket is in the `aws_url` column. We will also query the `instance_size` column, which we can use to calculate the size of the files corresponding to the cohort.\n",
164169
"\n",
165170
"As we learned in the part 2 of this tutorial series, each row of the IDC metadata table corresponds to a single DICOM file, and the attributes we mentioned above are assigned at the granularity of the individual files.\n",
166171
"\n",
@@ -171,12 +176,7 @@
171176
"cell_type": "code",
172177
"execution_count": null,
173178
"metadata": {
174-
"colab": {
175-
"base_uri": "https://localhost:8080/",
176-
"height": 441
177-
},
178-
"id": "D4dkFpEVfDO1",
179-
"outputId": "8a5c6f34-d92f-4d6c-dccf-792819882901"
179+
"id": "D4dkFpEVfDO1"
180180
},
181181
"outputs": [],
182182
"source": [
@@ -189,6 +189,7 @@
189189
"selection_query = \"\"\"\n",
190190
"SELECT\n",
191191
" gcs_url,\n",
192+
" aws_url,\n",
192193
" instance_size\n",
193194
"FROM\n",
194195
" bigquery-public-data.idc_current.dicom_all\n",
@@ -238,69 +239,36 @@
238239
"id": "ojgbk_nMhbv0"
239240
},
240241
"source": [
241-
"Now that we have URLs of the individual files, we can use `gsutil` command line tool from Google Cloud SDK to download them. When using Colab, Google Cloud SDK is pre-installed, but if you want to download files to your computer directly you will need to install SDK first! Note that download instructions are documented [here](https://learn.canceridc.dev/data/downloading-data).\n",
242+
"Now that we have URLs of the individual files, we can use the open source `s5cmd` command line tool to download the files either from the GCS or AWS locations. Note that download instructions are documented [here](https://learn.canceridc.dev/data/downloading-data).\n",
242243
"\n",
243-
"To download the files, we will first save GCS URLs into the manifest file, and then pass that manifest to `gsutil`."
244+
"[s5cmd](https://github.com/peak/s5cmd) is an open source very fast S3 and local filesystem execution tool, which, as experiments showed, is significantly faster than the Google-provided `gsutil`. Let's first install it."
244245
]
245246
},
246247
{
247248
"cell_type": "code",
248249
"execution_count": null,
249250
"metadata": {
250-
"colab": {
251-
"base_uri": "https://localhost:8080/"
252-
},
253-
"id": "8CVLUiMMijjE",
254-
"outputId": "3afc6cb2-5fea-4785-e900-65eaddeffc53"
251+
"id": "mr-F6YXrWHOm"
255252
},
256253
"outputs": [],
257254
"source": [
258-
"selection_df[\"gcs_url\"].to_csv(\"manifest.txt\", header=False, index=False)\n",
255+
"!wget https://github.com/peak/s5cmd/releases/download/v2.0.0/s5cmd_2.0.0_Linux-64bit.tar.gz && tar zxf s5cmd_2.0.0_Linux-64bit.tar.gz\n",
259256
"\n",
260-
"!rm -rf downloaded_cohort_files && mkdir downloaded_cohort_files\n",
261-
"!cat manifest.txt | gsutil -m cp -I downloaded_cohort_files"
262-
]
263-
},
264-
{
265-
"cell_type": "markdown",
266-
"metadata": {
267-
"id": "9yJD5pUSjXo0"
268-
},
269-
"source": [
270-
"Once the cell above is done, you can expand the left-side \"Files\" panel in the Colab interface and confirm that `downloaded_cohort_files` is not empty."
257+
"!./s5cmd --help"
271258
]
272259
},
273260
{
274261
"cell_type": "markdown",
275-
"metadata": {
276-
"id": "70hNEc71T_VZ"
277-
},
278262
"source": [
279-
"### Faster download using `s5cmd`\n",
280-
"\n",
281-
"[s5cmd](https://github.com/peak/s5cmd) is an open source very fast S3 and local filesystem execution tool, which, as experiments showed, is significantly faster than the Google-provided `gsutil`."
282-
]
283-
},
284-
{
285-
"cell_type": "code",
286-
"execution_count": null,
263+
"In order to use `s5cmd` we need to create the manifest that contains the commands to download individual files. Note that the `gs` bucket prefix for the Google GCS locations is replaced with `s3`. This is because `s5cmd` is a provider-agnostic tool that implements S3 API, and Google GCS buckets support S3 API."
264+
],
287265
"metadata": {
288-
"colab": {
289-
"base_uri": "https://localhost:8080/"
290-
},
291-
"id": "mr-F6YXrWHOm",
292-
"outputId": "de9cdb9c-af7a-44ac-8005-3e4076914a0a"
293-
},
294-
"outputs": [],
295-
"source": [
296-
"!wget https://github.com/peak/s5cmd/releases/download/v2.0.0/s5cmd_2.0.0_Linux-64bit.tar.gz && tar zxf s5cmd_2.0.0_Linux-64bit.tar.gz\n",
297-
"\n",
298-
"!./s5cmd --help"
299-
]
266+
"id": "gdrjpXYM--IH"
267+
}
300268
},
301269
{
302270
"cell_type": "code",
303-
"execution_count": 6,
271+
"execution_count": 3,
304272
"metadata": {
305273
"id": "gZhsxNbuWXy7"
306274
},
@@ -314,7 +282,8 @@
314282
"\n",
315283
"selection_query = \"\"\"\n",
316284
"SELECT\n",
317-
" CONCAT(\"cp \",REPLACE(gcs_url, \"gs://\", \"s3://\"), \" .\") as s5cmd_command,\n",
285+
" CONCAT(\"cp \",REPLACE(gcs_url, \"gs://\", \"s3://\"), \" .\") as s5cmd_gcs_command,\n",
286+
" CONCAT(\"cp \",aws_url, \" .\") as s5cmd_aws_command,\n",
318287
" StudyInstanceUID,\n",
319288
"FROM\n",
320289
" bigquery-public-data.idc_current.dicom_all\n",
@@ -354,24 +323,45 @@
354323
"\n"
355324
]
356325
},
326+
{
327+
"cell_type": "markdown",
328+
"source": [
329+
"Next two cells demonstrate how to generate a manifest and download the files corresponding to your selection from either GCS or AWS buckets."
330+
],
331+
"metadata": {
332+
"id": "M94EtV6jBnXg"
333+
}
334+
},
357335
{
358336
"cell_type": "code",
359337
"execution_count": null,
360338
"metadata": {
361-
"colab": {
362-
"base_uri": "https://localhost:8080/"
363-
},
364-
"id": "DvFRdvyPWpAE",
365-
"outputId": "8c7d2194-78cd-4ff3-93ff-33546cf51197"
339+
"id": "DvFRdvyPWpAE"
366340
},
367341
"outputs": [],
368342
"source": [
369-
"selection_df[\"s5cmd_command\"].to_csv(\"/content/s5cmd_manifest.txt\", header=False, index=False)\n",
343+
"# Download files from GCS\n",
344+
"selection_df[\"s5cmd_gcs_command\"].to_csv(\"/content/s5cmd_gcs_manifest.txt\", header=False, index=False)\n",
370345
"\n",
371-
"!rm -rf downloaded_cohort_files && mkdir downloaded_cohort_files\n",
372-
"!cd downloaded_cohort_files && /content/s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run /content/s5cmd_manifest.txt"
346+
"!rm -rf gcs_downloaded_cohort_files && mkdir gcs_downloaded_cohort_files\n",
347+
"!cd gcs_downloaded_cohort_files && /content/s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run /content/s5cmd_gcs_manifest.txt"
373348
]
374349
},
350+
{
351+
"cell_type": "code",
352+
"source": [
353+
"# Download files from AWS\n",
354+
"selection_df[\"s5cmd_aws_command\"].to_csv(\"/content/s5cmd_aws_manifest.txt\", header=False, index=False)\n",
355+
"\n",
356+
"!rm -rf aws_downloaded_cohort_files && mkdir aws_downloaded_cohort_files\n",
357+
"!cd aws_downloaded_cohort_files && /content/s5cmd --no-sign-request --endpoint-url https://s3.amazonaws.com run /content/s5cmd_aws_manifest.txt"
358+
],
359+
"metadata": {
360+
"id": "e6BjWfTTC5kS"
361+
},
362+
"execution_count": null,
363+
"outputs": []
364+
},
375365
{
376366
"cell_type": "markdown",
377367
"metadata": {
@@ -394,12 +384,7 @@
394384
"cell_type": "code",
395385
"execution_count": null,
396386
"metadata": {
397-
"colab": {
398-
"base_uri": "https://localhost:8080/",
399-
"height": 206
400-
},
401-
"id": "bW5ULTmfpa7g",
402-
"outputId": "90d45223-f381-4d3f-e2f5-6eec2b653951"
387+
"id": "bW5ULTmfpa7g"
403388
},
404389
"outputs": [],
405390
"source": [
@@ -460,11 +445,7 @@
460445
"cell_type": "code",
461446
"execution_count": null,
462447
"metadata": {
463-
"colab": {
464-
"base_uri": "https://localhost:8080/"
465-
},
466-
"id": "rmEGwPyUqrYr",
467-
"outputId": "548d7f8c-4edc-4504-a671-bfb59c5c9331"
448+
"id": "rmEGwPyUqrYr"
468449
},
469450
"outputs": [],
470451
"source": [
@@ -479,7 +460,7 @@
479460
"source": [
480461
"You can observe that there are two rows for the collection `qin_lung_ct`, which highlights important point that `collection_id` should be treated as a label grouping together both the items released by the original contributors of what initially formed the collection, but also the analysis results of the data in the original collection that might be contributed later. \n",
481462
"\n",
482-
"![collection](https://www.dropbox.com/s/ot7dbz5umligji1/collection.png?raw=1)\n",
463+
"![collection](https://www.dropbox.com/s/s6t44nqb2s4ye9l/tcia_collection.jpg?raw=1)\n",
483464
"\n",
484465
"In the example above, [`qin_lung_ct` collection ](https://doi.org/10.7937/K9/TCIA.2015.NPGZYZBZ) was complemented by the [segmentations of the lung nodules](https://doi.org/10.7937/K9/TCIA.2015.PF0M9REI), with both images and segmentations becoming part of the same collection, but having distinct DOIs and attribution requirements."
485466
]
@@ -512,11 +493,7 @@
512493
"cell_type": "code",
513494
"execution_count": null,
514495
"metadata": {
515-
"colab": {
516-
"base_uri": "https://localhost:8080/"
517-
},
518-
"id": "JVIqa3GWUV64",
519-
"outputId": "0df93427-00b0-43c1-d9b9-1f12676cb49a"
496+
"id": "JVIqa3GWUV64"
520497
},
521498
"outputs": [],
522499
"source": [
@@ -527,7 +504,6 @@
527504
]
528505
},
529506
{
530-
"attachments": {},
531507
"cell_type": "markdown",
532508
"metadata": {
533509
"id": "-zRHvX1ZK-Dn"
@@ -539,9 +515,10 @@
539515
]
540516
},
541517
{
542-
"attachments": {},
543518
"cell_type": "markdown",
544-
"metadata": {},
519+
"metadata": {
520+
"id": "pvcX4ZhS-OEu"
521+
},
545522
"source": [
546523
"We can use itkwidgets to view the full 3D model or view slices as well as choose whether or not we'd like to display segmentation label maps with our image data.\n",
547524
"\n",
@@ -552,11 +529,7 @@
552529
"cell_type": "code",
553530
"execution_count": null,
554531
"metadata": {
555-
"colab": {
556-
"base_uri": "https://localhost:8080/"
557-
},
558-
"id": "Bx9snM-MK-v9",
559-
"outputId": "988ea7e4-8a6c-47fa-ebaf-67d98565bb82"
532+
"id": "Bx9snM-MK-v9"
560533
},
561534
"outputs": [],
562535
"source": [
@@ -567,7 +540,7 @@
567540
},
568541
{
569542
"cell_type": "code",
570-
"execution_count": 12,
543+
"execution_count": null,
571544
"metadata": {
572545
"id": "o9hS4wamM3jO"
573546
},
@@ -580,11 +553,7 @@
580553
"cell_type": "code",
581554
"execution_count": null,
582555
"metadata": {
583-
"colab": {
584-
"base_uri": "https://localhost:8080/"
585-
},
586-
"id": "QEPR3kszK_o-",
587-
"outputId": "8791fea9-0869-4cf6-c49f-6c4f6c53eda7"
556+
"id": "QEPR3kszK_o-"
588557
},
589558
"outputs": [],
590559
"source": [
@@ -604,7 +573,7 @@
604573
},
605574
{
606575
"cell_type": "code",
607-
"execution_count": 14,
576+
"execution_count": null,
608577
"metadata": {
609578
"id": "Db18CEtoK_Md"
610579
},
@@ -616,7 +585,7 @@
616585
},
617586
{
618587
"cell_type": "code",
619-
"execution_count": 15,
588+
"execution_count": null,
620589
"metadata": {
621590
"id": "FksVuvkgLbpG"
622591
},
@@ -641,7 +610,7 @@
641610
},
642611
{
643612
"cell_type": "code",
644-
"execution_count": 16,
613+
"execution_count": null,
645614
"metadata": {
646615
"id": "dtNEiziLLlWv"
647616
},
@@ -654,7 +623,7 @@
654623
},
655624
{
656625
"cell_type": "code",
657-
"execution_count": 17,
626+
"execution_count": null,
658627
"metadata": {
659628
"id": "Rvf1sEizLwyR"
660629
},
@@ -682,7 +651,7 @@
682651
},
683652
{
684653
"cell_type": "code",
685-
"execution_count": 18,
654+
"execution_count": null,
686655
"metadata": {
687656
"colab": {
688657
"base_uri": "https://localhost:8080/",
@@ -1087,7 +1056,7 @@
10871056
},
10881057
{
10891058
"cell_type": "code",
1090-
"execution_count": 19,
1059+
"execution_count": null,
10911060
"metadata": {
10921061
"id": "sUi81pCUL5_0"
10931062
},
@@ -1110,7 +1079,7 @@
11101079
},
11111080
{
11121081
"cell_type": "code",
1113-
"execution_count": 20,
1082+
"execution_count": null,
11141083
"metadata": {
11151084
"id": "XSxs_YqeL6XW"
11161085
},
@@ -1172,7 +1141,8 @@
11721141
],
11731142
"metadata": {
11741143
"colab": {
1175-
"provenance": []
1144+
"provenance": [],
1145+
"include_colab_link": true
11761146
},
11771147
"gpuClass": "standard",
11781148
"kernelspec": {

0 commit comments

Comments
 (0)