{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Demo Notebook for deploying CLIPTextModel to OpenSearch\n", "\n", "#### [Download notebook](https://github.com/opensearch-project/opensearch-py-ml/blob/main/docs/source/examples/demo_deploy_cliptextmodel.ipynb)\n", "\n", "Related Docs:\n", "* [OpenSearch ML Framework](https://opensearch.org/docs/latest/ml-commons-plugin/ml-framework/)\n", "* [Huggingface - CLIP](https://huggingface.co/docs/transformers/model_doc/clip)\n", "* [Huggingface - export to torchscript](https://huggingface.co/docs/transformers/torchscript)\n", "\n", "This notebook provides a walkthrough for users to trace, register, and deploy a CLIPTextModel from a local file. CLIPTextModel can be used with the [Neural Search](https://opensearch.org/docs/latest/search-plugins/neural-search/) plugin to generate embeddings of documents and ingest time and of user queries at search time. \n", "\n", "Step 0: Import packages and set up client\n", "\n", "Step 1: Trace CLIPTextModel and export to TorchScript\n", "\n", "Step 2: Prep files for registration\n", "\n", "Step 3: Register model to OpenSearch\n", "\n", "Step 4: Deploy model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 0: Import packages and set up client" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from transformers import CLIPProcessor, CLIPTextModel\n", "import torch\n", "\n", "import opensearch_py_ml as oml\n", "from opensearch_py_ml.ml_commons import MLCommonClient\n", "from opensearchpy import OpenSearch\n", "\n", "import warnings\n", "warnings.filterwarnings(\"ignore\", message=\"Unverified HTTPS request\")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# Connect to OpenSearch cluster\n", "host = 'localhost'\n", "port = 9200\n", "auth = ('admin', '< admin password >') # For testing only. Don't store credentials in code.\n", "\n", "def get_os_client(host = host, port = port, auth = auth):\n", " '''\n", " Get OpenSearch client\n", " :param cluster_url: cluster URL like https://ml-te-netwo-1s12ba42br23v-ff1736fa7db98ff2.elb.us-west-2.amazonaws.com:443\n", " :return: OpenSearch client\n", " '''\n", " client = OpenSearch(\n", " hosts = [{'host': host, 'port': port}],\n", " http_compress = True, # enables gzip compression for request bodies\n", " http_auth = auth,\n", " use_ssl = False,\n", " verify_certs = False,\n", " ssl_assert_hostname = False,\n", " ssl_show_warn = False,\n", " )\n", " return client" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "client = get_os_client()\n", "ml_client = MLCommonClient(client)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Trace CLIPTextModel and export to TorchScript\n", "\n", "To use a model in OpenSearch, you’ll need to export the model into a portable format. As of Version 2.5, OpenSearch only supports the TorchScript and ONNX formats.\n", "\n", "Exporting a model to TorchScript requires two things:\n", "\n", "* model instantiation with the torchscript flag\n", "* a forward pass with dummy inputs\n", "\n", "The dummy inputs are used for a model's forward pass. While the inputs’ values are propagated through the layers, PyTorch keeps track of the different operations executed on each tensor. These recorded operations are then used to create the trace of the model.\n", "\n", "As of OpenSearch 2.6, the ML Framework supports text-embedding models only. CLIP is multi-modal, but we will use CLIPTextModel only here." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "model_name = \"openai/clip-vit-base-patch32\" #See https://huggingface.co/models for other options \n", "text_to_encode = \"example search query\" #See https://huggingface.co/docs/transformers/torchscript for more info on dummy inputs\n", "\n", "# Instantiate CLIPTextModel and CLIPProcessor with pretrained weights\n", "model = CLIPTextModel.from_pretrained(model_name, torchscript=True, return_dict=False)\n", "processor = CLIPProcessor.from_pretrained(model_name)\n", "\n", "# Use processor to generate tensors and create dummy input\n", "text_inputs =processor(text=text_to_encode, return_tensors=\"pt\",max_length=77, padding=\"max_length\", truncation=True)\n", "dummy_input = [text_inputs['input_ids'], text_inputs['attention_mask']]\n", "\n", "# Trace model and convert to torchscript object\n", "traced_model = torch.jit.trace(model, dummy_input)\n", "\n", "# Save model in portable format\n", "torch.jit.save(traced_model, \"traced_model_example.pt\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 2: Prep files for registration\n", "\n", "OpenSearch requires two files zipped together for registration:\n", "* Model in TorchScript format\n", "* tokenizor.json file\n", "\n", "The tokenizor for the model used in this example can be found [here](https://huggingface.co/openai/clip-vit-base-patch32/tree/main)\n", "\n", "Additionally, a config.json file with the following details must be passed with the .zip. More info on [model config](https://opensearch.org/docs/latest/ml-commons-plugin/api/#request-fields) " ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# config.json sample contents\n", "\"\"\"\n", "{\n", " \"name\": \"clip-vit-base-patch32\",\n", " \"version\": '1.0.0',\n", " \"model_format\": \"TORCH_SCRIPT\",\n", " \"model_config\": {\n", " \"model_type\": \"clip\",\n", " \"embedding_dimension\": 512,\n", " \"framework_type\": \"huggingface_transformers\"\n", " }\n", "}\n", "\"\"\";" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3: Register model to OpenSearch\n", "\n", "* Model name in config.json should match .pt torchscript file name\n", "* Record the model ID from the output of the next cell\n" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of chunks 19\n", "Sha1 value of the model file: 62f4786ef2d546180dbbaf8fe6b5be218243c8b806e6623840b1fe9d11bcad4a\n", "Model meta data was created successfully. Model Id: -uy7rooBhmcN7ynH0lgK\n", "uploading chunk 1 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 2 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 3 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 4 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 5 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 6 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 7 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 8 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 9 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 10 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 11 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 12 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 13 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 14 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 15 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 16 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 17 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 18 of 19\n", "Model id: {'status': 'Uploaded'}\n", "uploading chunk 19 of 19\n", "Model id: {'status': 'Uploaded'}\n", "Model registered successfully\n" ] } ], "source": [ "model_path = \"/traced_model_example.zip\"\n", "model_config_path = \"/config.json\"\n", "\n", "model_id_file_system = ml_client.register_model(model_path, model_config_path, isVerbose=True, deploy_model = False)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 4: Deploy model\n" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "task_id: --y8rooBhmcN7ynHYFg6\n", "Model deployed successfully\n" ] }, { "data": { "text/plain": [ "{'model_id': '-uy7rooBhmcN7ynH0lgK',\n", " 'task_type': 'DEPLOY_MODEL',\n", " 'function_name': 'TEXT_EMBEDDING',\n", " 'state': 'COMPLETED',\n", " 'worker_node': ['4K6CeIPPTkKiwZMplvJ6CQ'],\n", " 'create_time': 1695148695608,\n", " 'last_update_time': 1695148703362,\n", " 'is_async': True}" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_id = '-uy7rooBhmcN7ynH0lgK' #your model ID from previous step\n", "ml_client.deploy_model(model_id)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'name': 'traced_model_example3',\n", " 'model_group_id': '-ey7rooBhmcN7ynH0Vji',\n", " 'algorithm': 'TEXT_EMBEDDING',\n", " 'model_version': '1',\n", " 'model_format': 'TORCH_SCRIPT',\n", " 'model_state': 'DEPLOYED',\n", " 'model_content_size_in_bytes': 186945250,\n", " 'model_content_hash_value': '62f4786ef2d546180dbbaf8fe6b5be218243c8b806e6623840b1fe9d11bcad4a',\n", " 'model_config': {'model_type': 'clip',\n", " 'embedding_dimension': 512,\n", " 'framework_type': 'HUGGINGFACE_TRANSFORMERS'},\n", " 'created_time': 1695148659209,\n", " 'last_updated_time': 1695148703362,\n", " 'last_deployed_time': 1695148703362,\n", " 'total_chunks': 19,\n", " 'planning_worker_node_count': 1,\n", " 'current_worker_node_count': 1,\n", " 'planning_worker_nodes': ['4K6CeIPPTkKiwZMplvJ6CQ'],\n", " 'deploy_to_all_nodes': True}" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Check model status\n", "ml_client.get_model_info(model_id)" ] } ], "metadata": { "kernelspec": { "display_name": "opensearch_ml", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.0" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }