{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "tM8z02AidjxV" }, "source": [ "
\n", "

\n", " \"phoenix\n", "
\n", " Docs\n", " |\n", " GitHub\n", " |\n", " Community\n", "

\n", "
\n", "

Root-Cause Analysis for a Drifting Sentiment Classification Model

\n", "\n", "Imagine you're in charge of maintaining a model that takes as input online reviews of your U.S.-based product and classifies the sentiment of each review as positive, negative, or neutral. Your model initially performs well in production, but its performance gradually degrades over time.\n", "\n", "Phoenix helps you surface the reason for this regression by analyzing the embeddings representing the text of each review. Your model was trained on English reviews, but as you'll discover, it's encountering Spanish reviews in production that it can't correctly classify.\n", "\n", "In this tutorial, you will:\n", "* Download curated datasets of embeddings and predictions\n", "* Define a schema to describe the format of your data\n", "* Launch Phoenix to visually explore your embeddings\n", "* Investigate problematic clusters to identify the root cause of your model performance issue\n", "\n", "⚠️ This notebook runs slowly without a GPU. If you don't have access to a GPU, you can still use Phoenix by skipping the cells preceded by the 💬 emoji.\n", "\n", "Let's get started!\n", "\n", "## Install Dependencies and Import Libraries" ] }, { "cell_type": "markdown", "metadata": { "id": "Cz-v_EUQdjxY" }, "source": [ "Install Phoenix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -Uq \"arize[AutoEmbeddings]\" \"arize-phoenix[embeddings]\"" ] }, { "cell_type": "markdown", "metadata": { "id": "rT0wW2QvdjxZ" }, "source": [ "Import dependencies." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "from arize.pandas.embeddings import EmbeddingGenerator, UseCases\n", "\n", "import phoenix as px" ] }, { "cell_type": "markdown", "metadata": { "id": "0ERXbP9DdjxZ" }, "source": [ "## Download the Data\n", "\n", "Download training and production data from a model that classifies the sentiment of product reviews as positive, negative, or neutral." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_df = pd.read_parquet(\n", " \"http://storage.googleapis.com/arize-phoenix-assets/datasets/unstructured/nlp/sentiment-classification-language-drift/sentiment_classification_language_drift_training.parquet\",\n", ").reset_index(drop=True)\n", "prod_df = pd.read_parquet(\n", " \"http://storage.googleapis.com/arize-phoenix-assets/datasets/unstructured/nlp/sentiment-classification-language-drift/sentiment_classification_language_drift_production.parquet\",\n", ").reset_index(drop=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "K2DsVzsGdjxa" }, "source": [ "View a few training data points." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "gVkf3hm3djxa" }, "source": [ "The columns of the dataframe are:\n", "- **prediction_ts:** the Unix timestamps of your predictions\n", "- **review_age**, **reviewer_gender**, **product_category**, **language:** the features of your model\n", "- **text:** the text of each product review\n", "- **text_vector:** the embedding vectors representing each review\n", "- **pred_label:** the label your model predicted\n", "- **label:** the ground-truth label for each review" ] }, { "cell_type": "markdown", "metadata": { "id": "leqkAmdG0bnl" }, "source": [ "## Compute Embeddings\n", "\n", "💬 Compute embeddings using a [DistilBERT model fine-tuned on a dataset of product reviews](https://huggingface.co/arize-ai/distilbert_reviews_with_language_drift)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_name = \"arize-ai/distilbert_reviews_with_language_drift\"\n", "generator = EmbeddingGenerator.from_use_case(\n", " use_case=UseCases.NLP.SEQUENCE_CLASSIFICATION,\n", " model_name=model_name,\n", ")\n", "train_df[\"text_vector\"] = generator.generate_embeddings(text_col=train_df[\"text\"])\n", "prod_df[\"text_vector\"] = generator.generate_embeddings(text_col=prod_df[\"text\"])" ] }, { "cell_type": "markdown", "metadata": { "id": "adtRRvQI0Zy3" }, "source": [ "## Launch Phoenix\n", "\n", "Define a schema to tell Phoenix what the columns of your dataframe represent (predictions, actuals, embeddings, etc.). See the [docs](https://arize.com/docs/phoenix/) for guides on how to define your own schema and API reference on `phoenix.Schema` and `phoenix.EmbeddingColumnNames`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "schema = px.Schema(\n", " prediction_id_column_name=\"prediction_id\",\n", " timestamp_column_name=\"prediction_ts\",\n", " prediction_label_column_name=\"pred_label\",\n", " actual_label_column_name=\"label\",\n", " embedding_feature_column_names={\n", " \"text_embedding\": px.EmbeddingColumnNames(\n", " vector_column_name=\"text_vector\", raw_data_column_name=\"text\"\n", " ),\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "sd8KJTA0djxa" }, "source": [ "Create Phoenix datasets that wrap your dataframes with schemas that describe them." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "prod_ds = px.Inferences(dataframe=prod_df, schema=schema, name=\"production\")\n", "train_ds = px.Inferences(dataframe=train_df, schema=schema, name=\"training\")" ] }, { "cell_type": "markdown", "metadata": { "id": "q0m9uxqIdjxb" }, "source": [ "Launch Phoenix. Follow the instructions in the cell output to open the Phoenix UI in your notebook or in a separate browser tab." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "(session := px.launch_app(primary=prod_ds, reference=train_ds)).view()" ] }, { "cell_type": "markdown", "metadata": { "id": "hrILszGmVtsk" }, "source": [ "## Find the Root-Cause of Your Model Performance Issue\n", "\n", "Click on \"text_embedding\" in the \"Embeddings\" section.\n", "\n", "![click on text embedding](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/sentiment-classification-tutorial/click_on_text_embedding.png)\n", "\n", "\n", "In the Euclidean distance graph at the top of the page, click a point on the graph where the Euclidean distance is high.\n", "\n", "![select period of high drift](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/sentiment-classification-tutorial/select_period_of_high_drift.png)\n", "\n", "Click on the top cluster in the panel on the left.\n", "\n", "![select top cluster](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/sentiment-classification-tutorial/select_top_cluster.png)\n", "\n", "Use the panel at the bottom to examine the data points in this cluster.\n", "\n", "![inspect points in cluster](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/sentiment-classification-tutorial/inspect_points_in_cluster.png)\n", "\n", "What do you notice about the text in this cluster? Select other clusters and compare the text. Do you notice a difference?\n", "\n", "It turns out that your model is seeing Spanish product reviews in production, but the training data is all in English. Congrats! You've identified the root-cause of the issue. As an actionable next step, you should enhance your model to support other languages, e.g., by fine-tuning on Spanish product reviews." ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }