change the example and remove keyword search example

Author: Ziqun Ye

llm_application/Use_of_Cohere_embed_models_for_Semantic_Search_in_OCI_OpenSearch.ipynb

@@ -42,7 +42,7 @@
     "from langchain.text_splitter import MarkdownHeaderTextSplitter\n",
     "\n",
     "with fsspec.open(\n",
-    "    \"https://raw.githubusercontent.com/oracle-samples/oci-data-science-ai-samples/155f76109d24860ceeb72ed6b742ced33a46ce22/README.md\",\n",
+    "    \"https://raw.githubusercontent.com/oracle-samples/oci-data-science-ai-samples/main/distributed_training/Tensorboard.md\",\n",
     "    \"r\"\n",
     ") as f:\n",
     "    report = f.read()\n",
@@ -68,10 +68,14 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Number of documents: 15\n",
+      "Number of documents: 4\n",
       "First document:\n",
-      "The Oracle Cloud Infrastructure (OCI) Data Science service has created this repo to make demos, tutorials, and code examples that highlight various features of the [OCI Data Science service](https://www.oracle.com/data-science/cloud-infrastructure-data-science.html) and AI services. We welcome your feedback and would like to know what content is useful and what content is missing. Open an [issue](https://github.com/oracle/oci-data-science-ai-samples/issues) to do this. We know that a lot of you are creating great content and we would like to help you share it. See the [contributions](CONTRIBUTING.md) document.  \n",
-      "Oracle Cloud Infrastructure (OCI) Data Science Services provide a powerful suite of tools for data scientists, enabling faster machine learning model development and deployment. With features like the Accelerated Data Science (ADS) SDK, distributed training, batch processing and machine learning pipelines, OCI Data Science Services offer the scalability and flexibility needed to tackle complex data science and machine learning challenges. Whether you're a beginner or an experienced machine learning practitioner or data scientist, OCI Data Science Services provide the resources you need to build, train, and deploy your models with ease.\n"
+      "TensorBoard helps visualizing your experiments. You bring up a ``TensorBoard`` session on your workstation and point to\n",
+      "the directory that contains the TensorBoard logs.  \n",
+      "`OCI` = Oracle Cloud Infrastructure\n",
+      "`DT` = Distributed Training\n",
+      "`ADS` = Oracle Accelerated Data Science Library\n",
+      "`OCIR` = Oracle Cloud Infrastructure Registry\n"
      ]
     }
    ],
@@ -91,20 +95,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/z7ye/miniconda3/envs/ads_testing/lib/python3.8/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
-    "import ads\n",
     "from ads.llm import GenerativeAIEmbeddings\n",
     " \n",
     "oci_embedings = GenerativeAIEmbeddings(\n",
@@ -123,7 +117,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Number of embeddings: 15\n",
+      "Number of embeddings: 4\n",
       "Embedding dimensions: 1024\n"
      ]
     }
@@ -185,9 +179,20 @@
    "cell_type": "code",
    "execution_count": 7,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'acknowledged': True, 'shards_acknowledged': True, 'index': 'tensorboard'}"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
-    "INDEX_NAME = \"cosine-similarity\"\n",
+    "INDEX_NAME = \"tensorboard\"\n",
     "VECTOR_1_NAME = \"embedding_vector\"\n",
     "VECTOR_2_NAME = \"text\"\n",
     " \n",
@@ -254,7 +259,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Step 5: Semantic Search\n",
+    "\n",
     "A new query coming in, first calcualte the embedding vector and then conduct a semantic search.\n",
     "\n",
     "- `k`: the number of neighbors the search will return\n",
@@ -263,70 +268,50 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 15,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Oracle Cloud Infrastructure (OCI) [Data Science Jobs](https://docs.oracle.com/en-us/iaas/data-science/using/jobs-about.htm) is a powerful tool that allows you to define and run repeatable machine learning tasks on a fully managed infrastructure. With Jobs, you have the flexibility to apply custom tasks to meet your specific use cases, such as data preparation, model training, hyperparameter optimization, batch inference, large model training and more.  \n",
-      "On-demand jobs and batch processing are especially important for businesses that need to process large volumes of data on a regular basis, as they enable companies to automate data processing workflows, reduce the need for manual intervention, and save costs associated with running compute resources for extended periods of time. With the ability to define and schedule jobs to run at specific times, businesses can automate their data processing workflows and reduce the need for manual intervention. This helps to improve efficiency, reduce errors, and save valuable time and resources. Additionally, by using a fully managed infrastructure, businesses can ensure that their data processing workflows are secure and compliant with industry regulations. Overall, OCI Data Science Jobs is a powerful tool that can help businesses to scale their machine learning workflows and improve their data processing capabilities.\n"
+      "It is required that ``tensorboard`` is installed in a dedicated conda environment or virtual environment. Prepare an\n",
+      "environment yaml file for creating conda environment with following command -  \n",
+      "**tensorboard-dep.yaml**:  \n",
+      "```yaml\n",
+      "dependencies:\n",
+      "- python=3.8\n",
+      "- pip\n",
+      "- pip:\n",
+      "- ocifs\n",
+      "- tensorboard\n",
+      "name: tensorboard\n",
+      "```  \n",
+      "Create the conda environment from the yaml file generated in the preceeding step  \n",
+      "```bash\n",
+      "conda env create -f tensorboard-dep.yaml\n",
+      "```  \n",
+      "This will create a conda environment called tensorboard. Activate the conda environment by running -  \n",
+      "```bash\n",
+      "conda activate tensorboard\n",
+      "```  \n",
+      "**Using TensorBoard Logs:**  \n",
+      "To launch a TensorBoard session on your local workstation, run -  \n",
+      "```bash\n",
+      "export OCIFS_IAM_KEY=api_key\n",
+      "tensorboard --logdir oci://my-bucket@my-namespace/path/to/logs\n",
+      "```  \n",
+      "`OCIFS_IAM_KEY=api_key` - If you are using resource principal, set `resource_principal`  \n",
+      "This will bring up TensorBoard app on your workstation. Access TensorBoard at ``http://localhost:6006/``  \n",
+      "**Note**: The logs take some initial time (few minutes) to reflect on the tensorboard dashboard.\n"
      ]
     }
    ],
    "source": [
-    "query_vector = oci_embedings.embed_query(text=\"oci job\")\n",
+    "query_vector = oci_embedings.embed_query(text=\"how to set up tensorboard in oci?\")\n",
     "query = {\n",
     "    \"size\": 2,\n",
-    "    \"query\": {\"knn\": {VECTOR_1_NAME: {\"vector\": query_vector, \"k\": 3}}},\n",
-    "}\n",
-    " \n",
-    "response = es.search(body=query, index=INDEX_NAME)  # the same as before\n",
-    "print(response[\"hits\"][\"hits\"][0]['_source']['text'])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "#### Comparing Semantic Search with Keywords search\n",
-    "\n",
-    "here is the result of keyword search. you can see that since it does not understand the meaning of oci job, it gave irrelevant results."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Check out the following resources for more information about the OCI Data Science and AI services:  \n",
-      "* [ADS class documentation](https://accelerated-data-science.readthedocs.io/en/latest/modules.html)\n",
-      "* [ADS user guide](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
-      "* [AI & Data Science blog](https://blogs.oracle.com/ai-and-datascience/)\n",
-      "* [OCI Data Science service guide](https://docs.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
-      "* [OCI Data Science service release notes](https://docs.cloud.oracle.com/en-us/iaas/releasenotes/services/data-science/)\n",
-      "* [YouTube playlist](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
-      "* [OCI Data Labeling Service guide](https://docs.oracle.com/en-us/iaas/data-labeling/data-labeling/using/home.htm)\n",
-      "* [OCI DLS DP API](https://docs.oracle.com/en-us/iaas/api/#/en/datalabeling-dp/20211001/)\n",
-      "* [OCI DLS CP API](https://docs.oracle.com/en-us/iaas/api/#/en/datalabeling/20211001/)\n"
-     ]
-    }
-   ],
-   "source": [
-    "query = {\n",
-    "  \"query\": {\n",
-    "    \"match\": {\n",
-    "      \"text\": {\n",
-    "        \"query\": \"oci job\",\n",
-    "        \"analyzer\": \"standard\"\n",
-    "      }\n",
-    "    }\n",
-    "  }\n",
+    "    \"query\": {\"knn\": {VECTOR_1_NAME: {\"vector\": query_vector, \"k\": 2}}},\n",
     "}\n",
     " \n",
     "response = es.search(body=query, index=INDEX_NAME)  # the same as before\n",