add link to metadata docs

Author: fedorov

notebooks/getting_started/part2_searching_basics.ipynb

@@ -29,10 +29,12 @@
         "\n",
         "In this notebook you will be introduced into how IDC organizes the metadata accompanying images available in IDC, and how that metadata can be used to define subsets of data.\n",
         "\n",
+        "This documentation page can be used as a complement if you would like to learn more about how IDC metadata is organized: https://learn.canceridc.dev/data/organization-of-data/files-and-metadata.\n",
+        "\n",
         "---\n",
         "Initial version: Nov 2022\n",
         "\n",
-        "Updated: \n"
+        "Updated: Oct 2023\n"
       ]
     },
     {
@@ -89,7 +91,7 @@
       "source": [
         "## Why do I need to search?\n",
         "\n",
-        "Think of IDC as a library. Image files are books, and we have ~45 TB of those. When you go to a library, you want to check out just the books that you want to read. In order to find a book in a large library you need a catalog. \n",
+        "Think of IDC as a library. Image files are books, and we have ~45 TB of those. When you go to a library, you want to check out just the books that you want to read. In order to find a book in a large library you need a catalog.\n",
         "\n",
         "Just as in the library, IDC maintains a catalog that indexes a variety of metadata fields describing the files we curate. That metadata catalog is accessible in a large database table that you should be using to search and subset the images. Each row in that table corresponds to a file, and includes the location of the file alongside the metadata attributes describing that file.\n"
       ]
@@ -102,21 +104,21 @@
       "source": [
         "## How do I search?\n",
         "\n",
-        "When you search, or _query_ IDC catalog, you specify what criteria should the metadata describing the selected files satisfy. \n",
+        "When you search, or _query_ IDC catalog, you specify what criteria should the metadata describing the selected files satisfy.\n",
         "\n",
-        "Queries can be as simple as \n",
+        "Queries can be as simple as\n",
         "\n",
-        "* \"_everything in collection X_\", \n",
+        "* \"_everything in collection X_\",\n",
         "\n",
-        "or as complex as \n",
+        "or as complex as\n",
         "\n",
         "* \"_files corresponding to CT images of female patients that are accompanied by annotations of lung tumors that are larger than 1500 mm^3 in volume_\".\n",
         "\n",
         "Although it would be very nice to just state what you need in free form, in practice queries need to be written in a formal way.\n",
         "\n",
-        "IDC organizes all of the metadata into large tables, where each row corresponds to one image file (as of IDC data release v12, we index ~42 millions of files) and each column represents a metadata attribute present in one or more files in IDC (currently, we index hundreds of such attributes). \n",
+        "IDC organizes all of the metadata into large tables, where each row corresponds to one image file (as of IDC data release v12, we index ~42 millions of files) and each column represents a metadata attribute present in one or more files in IDC (currently, we index hundreds of such attributes).\n",
         "\n",
-        "IDC metadata tables are maintained in [GCP BigQuery](https://cloud.google.com/bigquery),  with only a tiny subset of the attributes indexed in the catalog available via the [IDC Portal exploration page](https://imaging.datacommons.cancer.gov/explore/). IDC metadata can be queried using Standard Query Language (SQL), and does not require learning any IDC-specific API. \n",
+        "IDC metadata tables are maintained in [GCP BigQuery](https://cloud.google.com/bigquery),  with only a tiny subset of the attributes indexed in the catalog available via the [IDC Portal exploration page](https://imaging.datacommons.cancer.gov/explore/). IDC metadata can be queried using Standard Query Language (SQL), and does not require learning any IDC-specific API.\n",
         "\n",
         "In the following steps of the tutorial we will use just a few of the attributes (SQL table columns) to get started. You will be able to use the same principles and SQL queries to extend your search criteria to include any of the other attributes indexed by IDC."
       ]
@@ -134,9 +136,9 @@
         "As the very first query, let's get the list of all the image collections available in IDC. Here is that query:\n",
         "\n",
         "```sql\n",
-        "SELECT \n",
-        "  DISTINCT(collection_id) \n",
-        "FROM \n",
+        "SELECT\n",
+        "  DISTINCT(collection_id)\n",
+        "FROM\n",
         "  bigquery-public-data.idc_current.dicom_all\n",
         "```\n",
         "\n",
@@ -168,9 +170,9 @@
         "2. `idc_current`is a _dataset_ within the `bigquery-public-data` project. Think of BigQuery datasets as containers that are used to organize and control access to the tables within the project.\n",
         "3. `dicom_all` is one of the tables within the `idc_current` dataset. As you spend more time learning about IDC, you will hopefully leverage other tables available in that dataset.\n",
         "\n",
-        "If you now look back at the [BigQuery console](https://console.cloud.google.com/bigquery) and expand the list of datasets under the `bigquery-public-data` project, you will see that in addition to the `idc_current` dataset there are also datasets `idc_v14`, `idc_v13`, etc all the way to `idc_v1`. Those datasets correspond to the IDC data release versions, with `idc_current` being an alias for the latest (at the moment of writing this, v14 is the latest release) version of IDC data. \n",
+        "If you now look back at the [BigQuery console](https://console.cloud.google.com/bigquery) and expand the list of datasets under the `bigquery-public-data` project, you will see that in addition to the `idc_current` dataset there are also datasets `idc_v14`, `idc_v13`, etc all the way to `idc_v1`. Those datasets correspond to the IDC data release versions, with `idc_current` being an alias for the latest (at the moment of writing this, v14 is the latest release) version of IDC data.\n",
         "\n",
-        "We will not spend time discussing how IDC versioning works, but it is important to know that \n",
+        "We will not spend time discussing how IDC versioning works, but it is important to know that\n",
         "\n",
         "1. IDC data is versioned;\n",
         "2. queries against the `idc_current` dataset are equivalent to the queries against the latest version (currently, `idc_v14`) of IDC data;\n",
@@ -207,9 +209,9 @@
         "bq_client = bigquery.Client(my_ProjectID)\n",
         "\n",
         "selection_query = \"\"\"\n",
-        "SELECT \n",
-        "  DISTINCT(collection_id) \n",
-        "FROM \n",
+        "SELECT\n",
+        "  DISTINCT(collection_id)\n",
+        "FROM\n",
         "  bigquery-public-data.idc_current.dicom_all\n",
         "\"\"\"\n",
         "\n",
@@ -362,7 +364,7 @@
         "\n",
         "# Execution of this cell will fail unless you wrote the query below!\n",
         "selection_query = \"\"\"\n",
-        "SELECT \n",
+        "SELECT\n",
         "  DISTINCT(collection_id)\n",
         "FROM\n",
         "  bigquery-public-data.idc_current.dicom_all\n",
@@ -389,11 +391,11 @@
       "source": [
         "## DICOM data model: Patients, studies, series and instances\n",
         "\n",
-        "Up to now we searched the data at the granularity of the collections. In practice, we often want to know how many patients meet our search criteria, or what are the specific images that we need to download. \n",
+        "Up to now we searched the data at the granularity of the collections. In practice, we often want to know how many patients meet our search criteria, or what are the specific images that we need to download.\n",
         "\n",
-        "IDC is using DICOM for data representation, and in the DICOM data model, patients (identified by `PatientID`) undergo imaging exams (or _studies_, in DICOM nomenclature). \n",
+        "IDC is using DICOM for data representation, and in the DICOM data model, patients (identified by `PatientID`) undergo imaging exams (or _studies_, in DICOM nomenclature).\n",
         "\n",
-        "Each patient will have one or more studies, with each study identified uniquely by the attribute `StudyInstanceUID`. During each of the imaging studies one or more imaging _series_ will be collected. As an example, a Computed Tomography (CT) imaging study may include a volume sweep before and after administration of the contrast agent. Imaging series are uniqiely identified by `SeriesInstanceUID`. \n",
+        "Each patient will have one or more studies, with each study identified uniquely by the attribute `StudyInstanceUID`. During each of the imaging studies one or more imaging _series_ will be collected. As an example, a Computed Tomography (CT) imaging study may include a volume sweep before and after administration of the contrast agent. Imaging series are uniqiely identified by `SeriesInstanceUID`.\n",
         "\n",
         "Finally, each imaging series contains one or more _instances_, where each instance corresponds to a file. Most often, one instance corresponds to a single slice from a cross-sectional image. Individual instances are identified by unique `SOPInstanceUID` values.\n",
         "\n",
@@ -419,7 +421,7 @@
         "bq_client = bigquery.Client(my_ProjectID)\n",
         "\n",
         "selection_query = \"\"\"\n",
-        "SELECT \n",
+        "SELECT\n",
         "  COUNT(DISTINCT(PatientID)) as patient_cnt\n",
         "FROM\n",
         "  bigquery-public-data.idc_current.dicom_all\n",
@@ -493,7 +495,7 @@
         "\n",
         "In many cases, image analysis is done at the granularity of the individual DICOM series. In some cases DICOM series corresponds to a single instance (e.g., for X-ray modalities), but in most cases imaging modalities are cross-sectional, containing multiple slices, with each slice stored in a separate instance (file), which can be reconstructed into a 3D volume.\n",
         "\n",
-        "From the examples and queries above, you should have developed some understanding about the modalities and few other collection-level characteristics for the data included in IDC. As an example, we know that IDC data contains MR images of Liver. \n",
+        "From the examples and queries above, you should have developed some understanding about the modalities and few other collection-level characteristics for the data included in IDC. As an example, we know that IDC data contains MR images of Liver.\n",
         "\n",
         "In the following query we select the UID of a sample MR series from the images covering Liver cancer."
       ]
@@ -521,8 +523,8 @@
         "WHERE\n",
         "  Modality = \"MR\" AND tcia_tumorLocation = \"Liver\"\n",
         "\n",
-        "# note the use of this new operator that makes the query \n",
-        "# return just the first one of the matching rows \n",
+        "# note the use of this new operator that makes the query\n",
+        "# return just the first one of the matching rows\n",
         "LIMIT\n",
         "  1\n",
         "\"\"\"\n",
@@ -539,7 +541,7 @@
         "id": "ab2lCsz5dyxA"
       },
       "source": [
-        "The result of this query is the _unique identifier_ for a DICOM series that meets the selection criteria. "
+        "The result of this query is the _unique identifier_ for a DICOM series that meets the selection criteria."
       ]
     },
     {
@@ -607,7 +609,7 @@
         "id": "5cbTYu-4hxH_"
       },
       "source": [
-        "This query introduces a couple of more advanced concepts: \n",
+        "This query introduces a couple of more advanced concepts:\n",
         "* we use `WITH` operator to define an intermediate query that writes the result into `temp_result` table, which is then queried\n",
         "* we capture all of the distinct values of `Modality` into an _array_ `modalities`, since we want to check for presence of both MR and SEG modalities in the study.\n",
         "\n",