{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\"DSPy7" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# DSPy Tutorial: Building a Code Plagiarism Detector\n", "\n", "If you've ever felt intimidated by DSPy, don't worryβ€”it might look complex at first glance, but it's actually quite approachable. This tutorial will walk you through the process of building a project, providing a clear, step-by-step approach to understanding and implementing DSPy concepts.\n", "\n", "## TLDR πŸš€\n", "\n", "We're going to build a system for code plagiarism detection. Our goal is to compare two input code files, determine if plagiarism has occurred, and provide an explanation for the result. \n", "\n", "This project will showcase:\n", "\n", "- Multiple inputs and outputs\n", "- Double validation techniques\n", "\n", "I strongly recommend reading the [DSPy Cheatsheet](https://dspy-docs.vercel.app/docs/cheatsheet) it will help you with quick start.\n", "\n", "## How to Start?\n", "\n", "A highly effective practice I've found to be game-changing when starting any DSPy project is to answer [these 8 key questions](https://dspy-docs.vercel.app/docs/building-blocks/solving_your_task). This exercise helps you develop a clear vision for your project before diving into the code.\n", "\n", "Here's an example of how your answers might look:\n", "1. **Define your task**\n", " - Expected input: Two input code files (strings containing plain code) to be compared.\n", " - Expected output:\n", " - Plagiarism detection result (Yes/No)\n", " - Explanation/justification of the result\n", " - Quality and Cost Specifications: Cost is not a concern; quality is the main priority. We want to try different models.\n", "\n", "2. **Define your pipeline**\n", " - We don't need any external tools or document retrieval. It will be a simple chain-of-thought step, as we want to evaluate LLM capabilities for plagiarism detection.\n", "\n", "3. **Explore a few examples**\n", " - We explored LLM capabilities for plagiarism detection using a few examples with ChatGPT and Claude, yielding promising results.\n", "\n", "4. **Define your data**\n", " - We are working with a dataset from the publication: [Source Code Plagiarism Detection in Academia with Information Retrieval: Dataset and the Observation](https://github.com/oscarkarnalim/sourcecodeplagiarismdataset/blob/master/IR-Plag-Dataset.zip)\n", " - We selected a subset and manually labeled the dataset with our output labels. This dataset should be used for training and testing, while the rest of the original dataset should be used for evaluation.\n", " - Dataset: [train.csv](/data/train.tsv) (65 samples)\n", " - When you don't have labeled dataset, it is good idea to try hand-labeling a few examples to get a sense of the task. It will help you to understand the task better and also increase the quality of program.\n", "\n", "5. **Define your metric**\n", " - We are dealing with a **classification problem**, so we will use accuracy as our main metric. \n", " - Our metric will be simple: if pred_label == true_label then 1 else 0.\n", " - As second evaluation we will be evaluating the quality of the explanation via secondary LLM.\n", "\n", "6. **Collect preliminary \"zero-shot\" evaluations**\n", " - Done in code.\n", "\n", "7. **Compile with a DSPy optimizer**\n", " - We don't want to update weights of the LLM, so we are looking at optimizers such as:\n", " - BootstrapFewShot\n", " - BootstrapFewShotWithRandomSearch\n", " - MIPRO\n", " - ...\n", "\n", "8. **Iterate**\n", " - Regroup and attack again!\n", "\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import os\n", "import re\n", "\n", "import dspy\n", "import pandas as pd\n", "from dotenv import load_dotenv\n", "from dspy.evaluate import Evaluate\n", "from dspy.teleprompt import (\n", " BootstrapFewShot,\n", " BootstrapFewShotWithRandomSearch,\n", " KNNFewShot,\n", " MIPROv2,\n", ")\n", "\n", "# load your environment variables from .env file\n", "load_dotenv()\n", "\n", "# azure-openai model deployment\n", "AZURE_OPENAI_KEY = os.getenv(\"AZURE_OPENAI_KEY\")\n", "AZURE_OPENAI_ENDPOINT = os.getenv(\"AZURE_OPENAI_ENDPOINT\")\n", "AZURE_OPENAI_DEPLOYMENT = os.getenv(\"AZURE_OPENAI_DEPLOYMENT\")\n", "AZURE_OPENAI_VERSION = os.getenv(\"AZURE_OPENAI_VERSION\")\n", "\n", "# openai model deployment\n", "\n", "OPENAI_MODEL = os.getenv(\"OPENAI_MODEL\")\n", "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\")\n", "\n", "# ollama deployment\n", "OLLAMA_URL = os.getenv(\"OLLAMA_URL\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 0. Load dataset\n", "\n", "Our first task is to load our dataset. Each entry in our dataset will consist of the following components:\n", "\n", "* `sample_1`: The first code sample to be analyzed\n", "* `sample_2`: The second code sample to be compared against the first\n", "* `plagiarized`: A boolean value (True if plagiarism is detected, False otherwise)\n", "* `reason`: A detailed explanation of the plagiarism detection result\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Lcasesample_1sample_2plagiarizedreason
001public class T1 { public static void main(Str.../* * To change this license header, choose Li...FalseThe two code samples, while producing similar ...
101public class T1 { public static void main(Str.../** * * @author 65FBEF05E01FAC390CB3FA073FB3...FalseThe code samples demonstrate different approac...
201public class T1 { public static void main(Str.../** * * @author CB6AB3315634A1E4D11B091BA48B...FalseThe two code samples produce the same output b...
311public class T1 { public static void main(Str...* * To change this license header, choose Lice...TrueThe two code samples are nearly identical in t...
421public class T1 { public static void main(Str.../* * To change this license header, choose Li...TrueThe two code samples contain identical main me...
\n", "
" ], "text/plain": [ " L case sample_1 \\\n", "0 0 1 public class T1 { public static void main(Str... \n", "1 0 1 public class T1 { public static void main(Str... \n", "2 0 1 public class T1 { public static void main(Str... \n", "3 1 1 public class T1 { public static void main(Str... \n", "4 2 1 public class T1 { public static void main(Str... \n", "\n", " sample_2 plagiarized \\\n", "0 /* * To change this license header, choose Li... False \n", "1 /** * * @author 65FBEF05E01FAC390CB3FA073FB3... False \n", "2 /** * * @author CB6AB3315634A1E4D11B091BA48B... False \n", "3 * * To change this license header, choose Lice... True \n", "4 /* * To change this license header, choose Li... True \n", "\n", " reason \n", "0 The two code samples, while producing similar ... \n", "1 The code samples demonstrate different approac... \n", "2 The two code samples produce the same output b... \n", "3 The two code samples are nearly identical in t... \n", "4 The two code samples contain identical main me... " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_csv(\"https://raw.githubusercontent.com/williambrach/LLM-plagiarism-check/main/data/train.tsv\", sep=\"\\t\")\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1. Prepare dataset\n", "\n", "DSPy utilizes special objects called [`Example`](https://dspy-docs.vercel.app/docs/deep-dive/data-handling/examples#creating-an-example) to structure and process data. Our next task is to convert our raw dataset into a collection of these `Example` objects.\n", "\n", "## Creating Custom Example Objects\n", "\n", "For our plagiarism detection system, we'll create `Example` objects with the following attributes:\n", "- `code_sample_1`: The first code sample to analyze\n", "- `code_sample_2`: The second code sample to compare\n", "- `plagiarized`: Boolean indicating whether plagiarism was detected\n", "- `explanation`: Detailed reasoning for the plagiarism decision\n", "\n", "## Specifying Inputs\n", "\n", "It's crucial to inform DSPy which attributes serve as inputs for our model. We accomplish this using the `.with_inputs()` method. In our case, we'll specify:\n", "\n", "```python\n", ".with_inputs(\"code_sample_1\", \"code_sample_2\")\n", "```\n" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "def create_example(row: pd.Series) -> dspy.Example:\n", " return dspy.Example(\n", " code_sample_1=row[\"sample_1\"],\n", " code_sample_2=row[\"sample_2\"],\n", " plagiarized=\"Yes\" if row[\"plagiarized\"] else \"No\",\n", " explanation=row[\"reason\"],\n", " ).with_inputs(\"code_sample_1\", \"code_sample_2\")\n", "\n", "\n", "examples = []\n", "for _, row in df.iterrows():\n", " example = create_example(row)\n", " examples.append(example)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2. DSPy setup\n", "\n", "DSPy is designed to be compatible with a variety of language models and their respective clients. For this tutorial, we will primarily utilize GPT-4 through the [Azure client](https://github.com/stanfordnlp/dspy/blob/main/dsp/modules/azure_openai.py). However, to demonstrate DSPy's flexibility, we will also provide configuration examples for other popular options:\n", "\n", "1. [OpenAI client](https://dspy-docs.vercel.app/docs/deep-dive/language_model_clients/remote_models/OpenAI)\n", "2. Local models via [Ollama](https://github.com/stanfordnlp/dspy/blob/main/dsp/modules/ollama.py)\n", "\n", "## Custom Client Implementation\n", "\n", "If your preferred language model client is not natively supported by DSPy, you have the option to implement a custom client. For detailed instructions on creating a custom client, please refer to this [comprehensive guide - custom-lm-client](https://dspy-docs.vercel.app/docs/deep-dive/language_model_clients/custom-lm-client).\n" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [], "source": [ "# client for AzureOpenAI\n", "lm = dspy.AzureOpenAI(\n", " api_base=AZURE_OPENAI_ENDPOINT,\n", " api_version=AZURE_OPENAI_VERSION,\n", " deployment_id=AZURE_OPENAI_DEPLOYMENT,\n", " api_key=AZURE_OPENAI_KEY,\n", ")\n", "\n", "# client for OpenAI\n", "# lm = dspy.OpenAI(\n", "# api_key=OPENAI_API_KEY,\n", "# model=OPENAI_MODEL,\n", "# )\n", "\n", "# client for Ollama\n", "# model_name = \"llama3.1\"\n", "# lm = dspy.OllamaLocal(base_url=OLLAMA_URL, model=model_name)\n", "\n", "dspy.settings.configure(lm=lm)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Setting up signature and module\n", "\n", "\n", "## Core DSPy Components\n", "\n", "[dspy.Signature](https://dspy-docs.vercel.app/docs/building-blocks/signatures) and [dspy.Module](https://dspy-docs.vercel.app/docs/building-blocks/modules) are fundamental building blocks for DSPy programs:\n", "\n", "- **Signature**: A declarative specification of the input/output behavior of a DSPy module.\n", "- **Module**: A building block for programs that leverage Language Models (LMs).\n", "\n", "## Types of DSPy Modules\n", "\n", "DSPy offers various module types, each serving different purposes:\n", "\n", "1. [dspy.Predict](https://dspy-docs.vercel.app/api/modules/Predict)\n", " - Basic predictor\n", " - Maintains the original signature\n", " - Handles key forms of learning (storing instructions, demonstrations, and LM updates)\n", " - Most similar to direct LM usage\n", "\n", "2. [dspy.ChainOfThought](https://dspy-docs.vercel.app/api/modules/ChainOfThought)\n", " - Enhances the LM to think step-by-step before producing the final response\n", " - Modifies the signature to incorporate intermediate reasoning steps\n", "\n", "3. [Additional Advanced Modules](https://dspy-docs.vercel.app/api/category/modules)\n", " - DSPy library offers a range of more specialized modules for complex tasks\n", "\n", "### Recommendation for starting\n", "\n", "For those new to DSPy, it's advisable to start with `dspy.Predict`. Its simplicity makes it ideal for understanding the basics of DSPy operation. Once you've successfully implemented your program using `dspy.Predict`, you can explore more advanced modules like `dspy.ChainOfThought` to potentially enhance your model's performance.\n", "\n", "For an overview of other prompting techniques beyond zero-shot learning, refer to the [Prompting Guide](https://www.promptingguide.ai/techniques). This resource covers various methods that can enhance your DSPy applications as you progress." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "class PlagiarismSignature(dspy.Signature):\n", " # Clarify something about the nature of the task (expressed below as a docstring)! \n", " \"\"\"Detect if two code samples are plagiarized. In plagiarized field answer only : Yes if the code samples are plagiarized, No otherwise. In explanation field add the reason why the code samples are/ are not plagiarized.\"\"\"\n", "\n", " # Supply hints on the nature of an input field, expressed as a desc keyword argument for dspy.InputField.\n", " code_sample_1 = dspy.InputField(desc=\"The first code sample to compare\")\n", " code_sample_2 = dspy.InputField(desc=\"The second code sample to compare\")\n", "\n", " # Supply constraints on an output field, expressed as a desc keyword argument for dspy.OutputField.\n", " explanation = dspy.OutputField(\n", " desc=\"Explanation or reason why the code samples are/ are not plagiarized\"\n", " )\n", " plagiarized = dspy.OutputField(\n", " desc=\"Yes/No indicating if code samples are plagiarized\"\n", " )\n", "\n", "\n", "class PlagiarismCoT(dspy.Module):\n", " def __init__(self) -> None:\n", " super().__init__()\n", "\n", " # self.prog = dspy.ChainOfThought(PlagiarismSignature)\n", " self.prog = dspy.Predict(PlagiarismSignature)\n", "\n", " def forward(self, code_sample_1: str, code_sample_2: str) -> PlagiarismSignature:\n", "\n", " # here you can do any processing you want, calling your function, etc.\n", " # modifying your code, inputs etc.\n", " # similar to pytorch forward function\n", "\n", " # returned signature object \n", " prediction = self.prog(code_sample_1=code_sample_1, code_sample_2=code_sample_2)\n", " return prediction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.1 Test your module with zero-shot program" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "plagiarized : Yes\n", "explanation : Explanation: The two code samples are identical in every aspect, including the class name, method name, and the repeated print statements. There are no differences in the structure, logic, or content of the code.\n" ] } ], "source": [ "output = PlagiarismCoT()(examples[0].code_sample_1, examples[0].code_sample_1)\n", "print(f\"plagiarized : {output.plagiarized}\")\n", "print(f\"explanation : {output.explanation}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Create metric for your programs\n", "\n", "Creating evaluation metric is a crucial component of the DSPy pipeline. This step is often the most complex function in your DSPy application.\n", "\n", "### Metric Structure\n", "\n", "In DSPy, metrics (or evaluation functions) must adhere to the following structure:\n", "\n", "1. **Arguments**: \n", " - Two required arguments, both of type `dspy.Example`:\n", " - The ground truth example from the dataset\n", " - The predicted example from the model\n", " - An optional third argument called `trace`\n", "\n", "2. **Return Value**: \n", " - A numerical score (float, int, or bool)\n", " - For binary metrics, return `True` for correct predictions and `False` for incorrect ones\n", "\n", "For more detailed information on creating metrics in DSPy, refer to the [official documentation on metrics](https://dspy-docs.vercel.app/docs/building-blocks/metrics)." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "def validate_answer(\n", " example: dspy.Example, pred: PlagiarismSignature, trace: object = None\n", ") -> bool:\n", " \"\"\"\n", " Validate the predicted plagiarism answer against the example answer.\n", "\n", " This function compares the predicted plagiarism answer with the example answer,\n", " focusing on a simple \"yes\" or \"no\" response. It extracts the core answer from\n", " the prediction, handling potential variations in formatting and capitalization.\n", "\n", " Parameters:\n", " - example (dspy.Example): The example object containing the correct answer.\n", " - pred (PlagiarismSignature): The prediction object containing the model's answer.\n", " - trace (object, optional): Unused parameter, kept for compatibility.\n", "\n", " Returns:\n", " - bool: True if the predicted answer matches the example answer, False otherwise.\n", "\n", " The function returns False if either the predicted or example answer is None,\n", " or if any exception occurs during the validation process.\n", " \"\"\"\n", " try:\n", " if pred.plagiarized is None:\n", " return False\n", "\n", " # Extract the first line of the predicted answer, convert to lowercase\n", " pred_plag = pred.plagiarized.strip().lower().split(\"\\n\")[0]\n", "\n", " # Define a regex pattern to match \"yes\" or \"no\"\n", " yes_no_pattern = r\"\\b(yes|no)\\b\"\n", "\n", " # Search for the pattern in the predicted answer\n", " match = re.search(yes_no_pattern, pred_plag)\n", "\n", " # If a match is found, use it; otherwise, use the entire predicted answer\n", " extracted_answer = match.group(1) if match else pred.plagiarized.strip().lower()\n", "\n", " if example.plagiarized is None:\n", " return False\n", "\n", " score = (\n", " True if extracted_answer == example.plagiarized.strip().lower() else False\n", " )\n", " except Exception:\n", " score = False\n", " return score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.1 Test your metric with evaluate\n", "\n", "Now we can evaluate zero-shot program with our `validate_answer` function. This will be bottom baseline for our program." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Average Metric: 39 / 63 (61.9): 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 63/63 [00:00<00:00, 1136.76it/s]\n" ] } ], "source": [ "evaluate = Evaluate(\n", " devset=examples,\n", " metric=validate_answer,\n", " num_threads=4,\n", " display_progress=True,\n", " display_table=0,\n", ")\n", "\n", "# zero-shot evaluation on examples\n", "score = evaluate(PlagiarismCoT())" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Zero-shot score: 61.9\n" ] } ], "source": [ "# score for zero-shot evaluation with `validate_answer` function\n", "print(f\"Zero-shot score: {score}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.2 When you can't use simple programmatic metrics and you need more brain power\n", "\n", "In our metric, we are only evaluating the `plagiarized` field. We are not evaluating the `explanation` field. If we wanted to evaluate the `explanation`, which is a string (reasoning), we could use a text similarity metric (cosine similarity, for example) or another LLM to evaluate it.\n", "\n", "The advantage of using another LLM is that we can evaluate the explanation in a more human-like way, but be careful with this approach. It could be very expensive and add more time to your program's runtime.\n", "\n", "More examples of implementing this type of metric can be found in the [tweets example](https://github.com/stanfordnlp/dspy/blob/main/examples/tweets/tweet_metric.py)." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "# Define the signature for automatic assessments.\n", "class Assess(dspy.Signature):\n", " \"\"\"Assess the similarity of a of inputs. Answer only Yes if inputs are similar or No if not.\"\"\"\n", "\n", " original_explanation = dspy.InputField()\n", " predicted_reasoning = dspy.InputField()\n", " similar = dspy.OutputField(desc=\"Yes if inputs are similar or No if not\")\n", "\n", "\n", "gpt_judge = dspy.AzureOpenAI(\n", " api_base=AZURE_OPENAI_ENDPOINT,\n", " api_version=AZURE_OPENAI_VERSION,\n", " deployment_id=AZURE_OPENAI_DEPLOYMENT,\n", " api_key=AZURE_OPENAI_KEY,\n", ")\n", "\n", "\n", "def validate_answer_with_explanation(\n", " example: dspy.Example, pred: PlagiarismSignature, trace: object = None\n", ") -> bool:\n", "\n", " # Extract the true explanation from the example\n", " true_explanation = example.explanation\n", "\n", " # Extract the predicted explanation from the prediction\n", " pred_explanation = pred.explanation\n", "\n", " # Use a language model (gpt_judge) to assess the similarity of explanations\n", " with dspy.context(lm=gpt_judge):\n", " # Create an Assess object to compare the explanations\n", " similar = dspy.Predict(Assess)(\n", " original_explanation=true_explanation, predicted_reasoning=pred_explanation\n", " )\n", "\n", " # Check if the explanations are deemed similar (converting to lowercase for case-insensitive comparison)\n", " similar_score = similar.similar.lower() == \"yes\"\n", "\n", " # Validate the plagiarism answer using the existing validate_answer function\n", " plagiarized_score = validate_answer(example, pred, trace)\n", "\n", " # Return True only if both the explanation is similar and the plagiarism answer is correct\n", " return similar_score and plagiarized_score" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Average Metric: 37 / 63 (58.7): 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 63/63 [00:00<00:00, 1220.76it/s]\n" ] } ], "source": [ "evaluate_with_explanation = Evaluate(\n", " devset=examples,\n", " metric=validate_answer_with_explanation,\n", " num_threads=4,\n", " display_progress=True,\n", " display_table=0,\n", ")\n", "\n", "# zero-shot evaluation on examples\n", "score = evaluate_with_explanation(PlagiarismCoT())" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Zero-shot score for validate_answer_with_explanation : 58.73\n" ] } ], "source": [ "# score for zero-shot evaluation with `validate_answer_with_explanation` function\n", "print(f\"Zero-shot score for validate_answer_with_explanation : {score}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5. DSPy Magic - Optimizers\n", "\n", "## TLDR on Optimizers\n", "\n", "Optimizers in DSPy are powerful tools that handle prompt engineering for you. Think of them as \"functions\" that automate and improve the process of crafting effective prompts. There are several types of optimizers available in DSPy, but in this tutorial, we'll focus on the first two:\n", "\n", "1. **Automatic Few-Shot Learning**: \n", " - Adds examples to your prompts.\n", " - Key optimizers to explore: `BootstrapFewShot` and `BootstrapFewShotWithRandomSearch`\n", "\n", "2. **Automatic Instruction Optimization**: \n", " - Produces optimal instructions for the prompt.\n", " - In the case of MIPRO, also optimizes the set of few-shot examples.\n", " - Key optimizer to explore: [`MIPROv2`](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/mipro_optimizer_v2.py)\n", "\n", "3. Automatic Finetuning:\n", " - Used to fine-tune the underlying Language Model(s).\n", "\n", "4. Program Transformations:\n", " - Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program.\n", "\n", "If you're new to optimizers and unsure where to start, `BootstrapFewShotWithRandomSearch` is a safe and effective choice for beginners.\n", "\n", "## Read the Documentation!\n", "\n", "For more detailed information about optimizers, please refer to the [official DSPy documentation on optimizers](https://dspy-docs.vercel.app/docs/building-blocks/optimizers)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.1 Example of BootstrapFewShot\n", "\n", "[BooststrapFewShot docs](https://dspy-docs.vercel.app/docs/deep-dive/teleprompter/bootstrap-fewshot), [BootstrapFewShot class](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/bootstrap.py) " ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " 6%|β–‹ | 4/63 [00:00<00:00, 3197.49it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Bootstrapped 4 full traces after 5 examples in round 0.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n", "Average Metric: 35 / 63 (55.6): 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 63/63 [00:00<00:00, 2608.37it/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[('prog', Predict(PlagiarismSignature(code_sample_1, code_sample_2 -> explanation, plagiarized\n", " instructions='Detect if two code samples are plagiarized. In plagiarized field answer only : Yes if the code samples are plagiarized, No otherwise. In explenation field add the reason why the code samples are/ are not plagiarized.'\n", " code_sample_1 = Field(annotation=str required=True json_schema_extra={'desc': 'The first code sample to compare', '__dspy_field_type': 'input', 'prefix': 'Code Sample 1:'})\n", " code_sample_2 = Field(annotation=str required=True json_schema_extra={'desc': 'The second code sample to compare', '__dspy_field_type': 'input', 'prefix': 'Code Sample 2:'})\n", " explanation = Field(annotation=str required=True json_schema_extra={'desc': 'Explanation or reason why the code samples are/ are not plagiarized', '__dspy_field_type': 'output', 'prefix': 'Explanation:'})\n", " plagiarized = Field(annotation=str required=True json_schema_extra={'desc': 'Yes/No indicating if code samples are plagiarized', '__dspy_field_type': 'output', 'prefix': 'Plagiarized:'})\n", ")))]\n" ] } ], "source": [ "config = {\"max_bootstrapped_demos\": 4, \"max_labeled_demos\": 4}\n", "\n", "optimizer = BootstrapFewShot(metric=validate_answer, **config)\n", "optimized_program = optimizer.compile(PlagiarismCoT(), trainset=examples)\n", "score = evaluate(optimized_program)\n", "optimized_program.save(\"BootstrapFewShot_program.json\")" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BootstrapFewShot score for validate_answer : 55.56\n" ] } ], "source": [ "# score for BootstrapFewShot evaluation with `validate_answer` function\n", "print(f\"BootstrapFewShot score : {score}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.2 Example of BootstrapFewShotWithRandomSearch\n", "\n", "[BootstrapFewShotWithRandomSearch class](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/random_search.py) " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "config = {\n", " \"max_bootstrapped_demos\": 4,\n", " \"max_labeled_demos\": 4,\n", " \"num_candidate_programs\": 5,\n", " \"num_threads\": 4,\n", "}\n", "\n", "optimizer = BootstrapFewShotWithRandomSearch(metric=validate_answer, **config)\n", "optimized_program = optimizer.compile(PlagiarismCoT(), trainset=examples)\n", "score = evaluate(optimized_program)\n", "optimized_program.save(\"BootstrapFewShotWithRandomSearch_program.json\")" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BootstrapFewShotWithRandomSearch score for validate_answer : 76.19\n" ] } ], "source": [ "# score for BootstrapFewShotWithRandomSearch evaluation with `validate_answer` function\n", "print(f\"BootstrapFewShotWithRandomSearch score : {score}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.3 Example of KNNFewShot\n", "\n", "[KNNFewShot class](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/knn_fewshot.py) " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "k = 3\n", "optimizer = KNNFewShot(k, examples)\n", "optimized_program = optimizer.compile(PlagiarismCoT(), trainset=examples)\n", "score = evaluate(optimized_program)\n", "optimized_program.save(\"KNNFewShot_program.json\")" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "KNNFewShot score : 85.71\n" ] } ], "source": [ "# score for KNNFewShot evaluation with `validate_answer` function\n", "print(f\"KNNFewShot score : {score}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.4 Example of MIPROv2\n", "\n", "[MIPROv2 paper](https://arxiv.org/abs/2406.11695), [MIPROv2 class](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/mipro_optimizer_v2.py) " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "n = 5 # The number of instructions and fewshot examples that we will generate and optimize over\n", "batches = 20 # The number of optimization trials to be run (we will test out a new combination of instructions and fewshot examples in each trial)\n", "temperature = 1 # The temperature configured for generating new instructions\n", "eval_kwargs = {\"num_threads\": 4, \"display_progress\": False, \"display_table\": 0}\n", "optimizer = MIPROv2(\n", " prompt_model=lm,\n", " task_model=lm,\n", " metric=validate_answer,\n", " num_candidates=n,\n", " init_temperature=temperature,\n", " verbose=True,\n", ")\n", "optimized_program = optimizer.compile(\n", " PlagiarismCoT(),\n", " trainset=examples,\n", " num_batches=batches,\n", " max_bootstrapped_demos=16,\n", " max_labeled_demos=16,\n", " requires_permission_to_run=False,\n", " eval_kwargs=eval_kwargs,\n", ")\n", "score = evaluate(optimized_program)\n", "optimized_program.save(\"MIPROv2_program.json\")" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MIPROv2 score : 87.85\n" ] } ], "source": [ "# score for MIPROv2 evaluation with `validate_answer` function\n", "print(f\"MIPROv2 score : {score}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6. Loading a Saved Program\n", "\n", "Loading a program from a saved file is a straightforward process in DSPy. Here's how you can do it:" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [], "source": [ "loaded_program = PlagiarismCoT()\n", "loaded_program.load(path=\"BootstrapFewShotWithRandomSearch_program.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6.1 Deploying a DSPy Program\n", "\n", "Deploying a DSPy program in a production environment can be challenging, as DSPy is primarily designed for research and experimentation at the moment. However, there are two effective methods to successfully deploy a DSPy program in production:\n", "\n", "### 6.1.1 Using DSPy as a Library\n", "\n", "This method involves saving, loading, and using your DSPy program directly:\n", "\n", "1. Save your DSPy program to a file using the `.save()` method.\n", "2. Load your DSPy program from the file using the `.load()` method.\n", "3. Use the loaded program to make predictions in your production environment.\n", "\n", "### 6.1.2 Extracting Prompts and Calling Your LLM Directly\n", "\n", "This approach involves:\n", "\n", "1. Extracting the optimized prompts from your DSPy program.\n", "2. Implementing these prompts in your production code.\n", "3. Calling your Language Model (LLM) directly with the extracted prompts.\n", "\n", "Each method has its advantages and may be more suitable depending on your specific use case and production environment constraints." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# dummy call of your program\n", "res = loaded_program(examples[0].code_sample_1, examples[0].code_sample_2)\n", "\n", "# find last call in your llm\n", "last_prompt = lm.inspect_history(n=1)\n", "with open(\"inspect_history.txt\", \"w\") as f:\n", " f.write(str(last_prompt))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Refining Your Prompt\n", "\n", "You can now streamline your prompt by removing unnecessary parameters and utilizing Python's string formatting to incorporate your inputs. Here's an enhanced version of the prompt:\n", "\n", "```python\n", "prompt = f\"\"\"\n", "Detect if two code samples are plagiarized. Answer only 'Yes' in the plagiarized field if the code samples are plagiarized, 'No' otherwise. Provide the reasoning in the explanation field.\n", "\n", "Format:\n", "Code Sample 1: [First code sample to compare]\n", "Code Sample 2: [Second code sample to compare]\n", "\n", "Reasoning: Let's analyze step by step to determine if the code samples are plagiarized...\n", "\n", "Explanation: [Detailed explanation of why the code samples are or are not plagiarized]\n", "\n", "Plagiarized: [Yes/No]\n", "\n", "Example:\n", "Code Sample 1: \n", "import java.util.Scanner;\n", "public class T3 {{\n", " public static void main(String[] args) {{\n", " Scanner input = new Scanner(System.in);\n", " System.out.print(\"Enter weight in pounds: \");\n", " double weight = input.nextDouble();\n", " System.out.print(\"Enter feet: \");\n", " double feet = input.nextDouble();\n", " System.out.print(\"Enter inches: \");\n", " double inches = input.nextDouble();\n", " double height = feet * 12 + inches;\n", " double bmi = weight * 0.45359237 / ((height * 0.0254) * (height * 0.0254));\n", " System.out.println(\"BMI is \" + bmi);\n", " if (bmi < 18.5) System.out.println(\"Underweight\");\n", " else if (bmi < 25) System.out.println(\"Normal\");\n", " else if (bmi < 30) System.out.println(\"Overweight\");\n", " else System.out.println(\"Obese\");\n", " }}\n", "}}\n", "\n", "Code Sample 2:\n", "import java.util.*;\n", "public class L2 {{\n", " public static void main(String[] args) {{\n", " Scanner sc = new Scanner(System.in);\n", " System.out.print(\"Enter weight in pounds: \");\n", " double berat = sc.nextDouble();\n", " System.out.print(\"Enter feet: \");\n", " double feet = sc.nextDouble();\n", " System.out.print(\"Enter inches: \");\n", " double inci = sc.nextDouble();\n", " double tinggi = feet * 12 + inci;\n", " double bmi = berat * 0.45359237 / ((tinggi * 0.0254) * (tinggi * 0.0254));\n", " System.out.println(\"BMI is \" + bmi);\n", " if (bmi < 18.5) {{ System.out.println(\"Underweight\"); }}\n", " else if (bmi < 25) {{ System.out.println(\"Normal\"); }}\n", " else if (bmi < 30) {{ System.out.println(\"Overweight\"); }}\n", " else {{ System.out.println(\"Obese\"); }}\n", " }}\n", "}}\n", "\n", "Reasoning: Let's analyze the code samples step by step to determine if they are plagiarized...\n", "\n", "Explanation: The two code samples are nearly identical in structure, logic, and even specific constants used, with only minor differences in variable names and formatting. This level of similarity is extremely unlikely to occur independently and strongly indicates that one sample was copied from the other or both were derived from a common source.\n", "\n", "Plagiarized: Yes\n", "\n", "Now, analyze the following code samples:\n", "\n", "Code Sample 1: {code_sample_1}\n", "\n", "Code Sample 2: {code_sample_2}\n", "\n", "Reasoning: Let's analyze step by step to determine if the code samples are plagiarized...\n", "\"\"\"\n", "```\n", "**Just be aware that you need to do your own parsing of the output of the LLM.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Thanks! πŸ™Œ\n", "\n", "In case of any questions, feel free to reach out to me: \n", "William Brach - [@williambrach](https://x.com/williambrach) - william.brach@stuba.sk" ] } ], "metadata": { "kernelspec": { "display_name": "dspy", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 2 }