Merge pull request #364 from lu-ohai/ODSC-50481/deploy_langchain_notebook

Added example to deploy langchain application

Author: qiuosier

notebook_examples/deploy-langchain-as-oci-data-science-model-deployment.ipynb

@@ -0,0 +1,386 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "@notebook{deploy-langchain-as-oci-data-science-model-deployment.ipynb,\n",
+    "    title: Deploy LangChain Application as OCI Data Science Model Deployment,\n",
+    "    summary: Deploy LangChain applications as OCI data science model deployment,\n",
+    "    developed_on: pytorch21_p39_gpu_v1,\n",
+    "    keywords: langchain, deploy model, register model, LLM,\n",
+    "    license: Universal Permissive License v 1.0\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font color=gray>Oracle Data Science service sample notebook.\n",
+    "\n",
+    "Copyright (c) 2023 Oracle, Inc.  All rights reserved.\n",
+    "Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.\n",
+    "</font>\n",
+    "\n",
+    "***\n",
+    "# <font color=red>Deploy LangChain Application as OCI Data Science Model Deployment</font>\n",
+    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=teal> Oracle Cloud Infrastructure Data Science Service Team </font></p>\n",
+    "\n",
+    "***\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "The notebook demonstrates how to deploy LangChain application as OCI Data Science Model Deployment using Oracle Accelerated Data Science (ADS) SDK.\n",
+    "\n",
+    "The `ChainDeployment` class in ADS allows you to rapidly get a LangChain application into production. The `.prepare()` method serializes the LangChain application as `chain.yaml` file and generates `score.py` file which can further be uploaded to OCI model catalog. The uploaded model can be subsequently deployed into production.\n",
+    "\n",
+    "Compatible conda pack: [pytorch21_p39_gpu_v1](https://docs.oracle.com/en-us/iaas/data-science/using/conda-gml-fam.htm) for CPU on Python 3.9 (version 1.0)\n",
+    "\n",
+    "### Prequisites\n",
+    "\n",
+    "This notebook requires authorization to work with the OCI Data Science Service. Details can be found [here](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html#). For the purposes of this notebook what is important to to know is that resource principals will be used absent api_key authentication.\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Contents\n",
+    "\n",
+    "* <a href='#intro'>Introduction</a>\n",
+    "* <a href='#create'>Create a LangChain Application</a>\n",
+    "* <a href='#deploy'>Deploy LangChain Application as OCI Model Deployment</a>\n",
+    "    * <a href='#deploy_chaindeployment'>Create a ChainDeployment</a>\n",
+    "    * <a href='#deploy_prepare'>Prepare</a>\n",
+    "    * <a href='#deploy_verify'>Verify</a>\n",
+    "    * <a href='#deploy_save'>Save</a>\n",
+    "    * <a href='#deploy_deploy'>Deploy</a>\n",
+    "    * <a href='#deploy_predict'>Predict</a>\n",
+    "* <a href='#clean_up'>Clean Up</a>\n",
+    "* <a href='#ref'>References</a>    \n",
+    "\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import ads\n",
+    "import os\n",
+    "import tempfile\n",
+    "\n",
+    "from ads.llm.deploy import ChainDeployment\n",
+    "from langchain.llms import Cohere\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from shutil import rmtree"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='intro'></a>\n",
+    "# Introduction\n",
+    "\n",
+    "In this notebook, you will create a custom LangChain application that links prompt and Cohere model and deploy it on OCI model deployment. It is designed to demonstrate how to use the `ChainDeployment` class in Oracle ADS SDK.\n",
+    "\n",
+    "The `.prepare()` method will serialize the LangChain application as `chain.yaml` file. It will also generate a `score.py` file that will load the LangChain yaml and call the `predict()` method. The `.save()` and `.deploy()` methods will upload the artifacts to OCI model catalog and deploy the uploaded model, respectively."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Authenticate\n",
+    "\n",
+    "Authentication to the OCI Data Science service is required. Here we default to resource principals."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ads.set_auth(auth=\"resource_principal\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='create'></a>\n",
+    "# Create a LangChain Application\n",
+    "\n",
+    "The next cell creates a LangChain application that links prompt and Cohere model. The LangChain application will utilize Cohere model to generate a joke based on the subject that user provides. Remember to replace the `<cohere_api_key>` with the actual api key as Cohere model needs it. You can acquire this key by registering on [Cohere](https://dashboard.cohere.com/welcome/register)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"COHERE_API_KEY\"] = \"<cohere_api_key>\"\n",
+    "\n",
+    "cohere = Cohere() \n",
+    "prompt = PromptTemplate.from_template(\"Tell me a joke about {subject}\")\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=cohere, verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now you have a LangChain object `llm_chain`. Try running it with the prompt `{\"subject\": \"animals\"}` and it should give you the corresponding answer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm_chain.run({\"subject\": \"animals\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='deploy'></a>\n",
+    "# Deploy LangChain Application as OCI Model Deployment\n",
+    "\n",
+    "<a id='deploy_chaindeployment'></a>\n",
+    "## Create a ChainDeployment\n",
+    "\n",
+    "The next cell creates a model artifact directory. This directory is used to store the artifacts that are needed to deploy the model. It also creates the `ChainDeployment` object. The `ChainDeployment` requires the LangChain object `llm_chain` as `chain` parameter."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "artifact_dir = tempfile.mkdtemp()\n",
+    "\n",
+    "chain_deployment = ChainDeployment(\n",
+    "    chain=llm_chain,\n",
+    "    artifact_dir=artifact_dir\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='deploy_prepare'></a>\n",
+    "## Prepare\n",
+    "\n",
+    "The prepare step is performed by the `.prepare()` method of the `ChainDeployment` class. It creates a number of customized files that are used to run the model once it is deployed. These include:\n",
+    "\n",
+    "* `chain.yaml`: A YAML file that is serialized from the LangChain application and can be deserialized in `load_model` in `score.py`.\n",
+    "* `runtime.yaml`: This file contains information that is needed to set up the runtime environment on the deployment server.\n",
+    "* `score.py`: This script contains the `load_model()` and `predict()` functions. The `load_model()` function understands the format the model file was saved in, and loads it into memory. The `.predict()` method is used to make inferences in a deployed model.\n",
+    "\n",
+    "To create the model artifacts, you use the `.prepare()` method\n",
+    "\n",
+    "* `inference_conda_env` variable defines the slug of the conda environment that is used to train the model\n",
+    "\n",
+    "Note that you can only pass in slug for service conda environment. For custom conda environment, you have to pass in the full path along with the `inference_python_version`. \n",
+    "\n",
+    "Here, replace `<custom_conda_environment_uri>` with your conda environment uri that has the latest ADS 2.9.1 and replace `<python_version>` with your conda environment python version. For how to customize and publish conda environment, take reference to [Publishing a Conda Environment to an Object Storage Bucket](https://docs.oracle.com/en-us/iaas/data-science/using/conda_publishs_object.htm)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_deployment.prepare(\n",
+    "    inference_conda_env=\"<custom_conda_environment_uri>\",\n",
+    "    inference_python_version=\"<python_version>\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='deploy_verify'></a>\n",
+    "## Verify\n",
+    "\n",
+    "The `.verify()` method takes a set of test parameters and performs the prediction by calling the `predict` function in `score.py`. It also runs the `load_model` function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_deployment.verify({\"subject\": \"animals\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='deploy_save'></a>\n",
+    "## Save\n",
+    "\n",
+    "Call `.save()` to pack and upload the artifacts under `artifact_dir` to OCI data science model catalog. Once the artifacts are successfully uploaded, you should be able to see the id of the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_deployment.save(display_name=\"LangChain Model\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='deploy_deploy'></a>\n",
+    "## Deploy\n",
+    "\n",
+    "Deploy the LangChain model from previous step by calling `.deploy()`. For more information regarding the allowed parameters, see [here](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/model_serialization/genericmodel.html#deploy). Remember to replace the `<cohere_api_key>` with the actual cohere api key in the `environment_variables`. It usually takes a couple of minutes to deploy the model and you should see the model deployment in the output once the process completes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_deployment.deploy(\n",
+    "    display_name=\"LangChain Model Deployment\",\n",
+    "    environment_variables={\"COHERE_API_KEY\":\"<cohere_api_key>\"}, \n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='deploy_predict'></a>\n",
+    "## Predict\n",
+    "\n",
+    "After the deployment is active, you can call the `predict()` on the `ChainDeployment` object to send request to the deployed endpoint. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_deployment.predict(data={\"subject\": \"animals\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='clean_up'></a>\n",
+    "# Clean Up\n",
+    "\n",
+    "This notebook created a model deployment and a model. This section deletes those resources. \n",
+    "\n",
+    "The model deployment must be deleted before the model can be deleted. You can use the `.delete_deployment()` method on the `ChainDeployment` object to do this."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "delete = chain_deployment.delete_deployment(wait_for_completion=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use the `.delete()` method to delete the model:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_deployment.delete()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The next cell removes the model artifacts that were stored on your local drive:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "rmtree(artifact_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='ref'></a>\n",
+    "# References\n",
+    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
+    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
+    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
+    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)\n",
+    "- [Understanding Conda Environments](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/use-notebook-sessions.htm#conda_understand_environments)\n",
+    "- [Use Resource Manager to Configure Your Tenancy for Data Science](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/orm-configure-tenancy.htm)\n",
+    "- [runtime.yaml](https://docs.content.oci.oracleiaas.com/en-us/iaas/data-science/using/model_runtime_yaml.htm#model_runtime_yaml)\n",
+    "- [score.py](https://docs.content.oci.oracleiaas.com/en-us/iaas/data-science/using/model_score_py.htm#model_score_py)\n",
+    "- [Model artifact](https://docs.content.oci.oracleiaas.com/en-us/iaas/data-science/using/models_saving_catalog.htm#create-models)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python [conda env:pytorch110_p38_cpu_v1]",
+   "language": "python",
+   "name": "conda-env-pytorch110_p38_cpu_v1-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.13"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}