cookbook/stepback-qa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "83ef724e",
   "metadata": {},
   "source": [
    "# Step-Back Prompting (Question-Answering)\n",
    "\n",
    "One prompting technique called \"Step-Back\" prompting can improve performance on complex questions by first asking a \"step back\" question. This can be combined with regular question-answering applications by then doing retrieval on both the original and step-back question.\n",
    "\n",
    "Read the paper [here](https://arxiv.org/abs/2310.06117)\n",
    "\n",
    "See an excellent blog post on this by Cobus Greyling [here](https://cobusgreyling.medium.com/a-new-prompt-engineering-technique-has-been-introduced-called-step-back-prompting-b00e8954cacb)\n",
    "\n",
    "In this cookbook we will replicate this technique. We modify the prompts used slightly to work better with chat models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "67b5cdac",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate\n",
    "from langchain_core.runnables import RunnableLambda\n",
    "from langchain_openai import ChatOpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "7e017c44",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Few Shot Examples\n",
    "examples = [\n",
    "    {\n",
    "        \"input\": \"Could the members of The Police perform lawful arrests?\",\n",
    "        \"output\": \"what can the members of The Police do?\",\n",
    "    },\n",
    "    {\n",
    "        \"input\": \"Jan Sindel’s was born in what country?\",\n",
    "        \"output\": \"what is Jan Sindel’s personal history?\",\n",
    "    },\n",
    "]\n",
    "# We now transform these to example messages\n",
    "example_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"human\", \"{input}\"),\n",
    "        (\"ai\", \"{output}\"),\n",
    "    ]\n",
    ")\n",
    "few_shot_prompt = FewShotChatMessagePromptTemplate(\n",
    "    example_prompt=example_prompt,\n",
    "    examples=examples,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "206415ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"\"\"You are an expert at world knowledge. Your task is to step back and paraphrase a question to a more generic step-back question, which is easier to answer. Here are a few examples:\"\"\",\n",
    "        ),\n",
    "        # Few shot examples\n",
    "        few_shot_prompt,\n",
    "        # New question\n",
    "        (\"user\", \"{question}\"),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "d643a85c",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_gen = prompt | ChatOpenAI(temperature=0) | StrOutputParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "5ba21b2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"was chatgpt around while trump was president?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "id": "5992c8ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'when was ChatGPT developed?'"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question_gen.invoke({\"question\": question})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "id": "32667424",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.utilities import DuckDuckGoSearchAPIWrapper\n",
    "\n",
    "search = DuckDuckGoSearchAPIWrapper(max_results=4)\n",
    "\n",
    "\n",
    "def retriever(query):\n",
    "    return search.run(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "ffc28c91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This includes content about former President Donald Trump. According to further tests, ChatGPT successfully wrote poems admiring all recent U.S. presidents, but failed when we entered a query for ... On Wednesday, a Twitter user posted screenshots of him asking OpenAI\\'s chatbot, ChatGPT, to write a positive poem about former President Donald Trump, to which the chatbot declined, citing it ... While impressive in many respects, ChatGPT also has some major flaws. ... [President\\'s Name],\" refused to write a poem about ex-President Trump, but wrote one about President Biden ... During the Trump administration, Altman gained new attention as a vocal critic of the president. It was against that backdrop that he was rumored to be considering a run for California governor.'"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever(question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "00c77443",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Will Douglas Heaven March 3, 2023 Stephanie Arnett/MITTR | Envato When OpenAI launched ChatGPT, with zero fanfare, in late November 2022, the San Francisco-based artificial-intelligence company... ChatGPT, which stands for Chat Generative Pre-trained Transformer, is a large language model -based chatbot developed by OpenAI and launched on November 30, 2022, which enables users to refine and steer a conversation towards a desired length, format, style, level of detail, and language. ChatGPT is an artificial intelligence (AI) chatbot built on top of OpenAI's foundational large language models (LLMs) like GPT-4 and its predecessors. This chatbot has redefined the standards of... June 4, 2023 ⋅ 4 min read 124 SHARES 13K At the end of 2022, OpenAI introduced the world to ChatGPT. Since its launch, ChatGPT hasn't shown significant signs of slowing down in developing new...\""
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever(question_gen.invoke({\"question\": question}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "id": "b257bc06",
   "metadata": {},
   "outputs": [],
   "source": [
    "# response_prompt_template = \"\"\"You are an expert of world knowledge. I am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them if they are not relevant.\n",
    "\n",
    "# {normal_context}\n",
    "# {step_back_context}\n",
    "\n",
    "# Original Question: {question}\n",
    "# Answer:\"\"\"\n",
    "# response_prompt = ChatPromptTemplate.from_template(response_prompt_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "id": "f48c65b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "\n",
    "response_prompt = hub.pull(\"langchain-ai/stepback-answer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "id": "97a6d5ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = (\n",
    "    {\n",
    "        # Retrieve context using the normal question\n",
    "        \"normal_context\": RunnableLambda(lambda x: x[\"question\"]) | retriever,\n",
    "        # Retrieve context using the step-back question\n",
    "        \"step_back_context\": question_gen | retriever,\n",
    "        # Pass on the question\n",
    "        \"question\": lambda x: x[\"question\"],\n",
    "    }\n",
    "    | response_prompt\n",
    "    | ChatOpenAI(temperature=0)\n",
    "    | StrOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "id": "ce554cb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"No, ChatGPT was not around while Donald Trump was president. ChatGPT was launched on November 30, 2022, which is after Donald Trump's presidency. The context provided mentions that during the Trump administration, Altman, the CEO of OpenAI, gained attention as a vocal critic of the president. This suggests that ChatGPT was not developed or available during that time.\""
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": question})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9fb8dd2",
   "metadata": {},
   "source": [
    "## Baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "id": "00db8a15",
   "metadata": {},
   "outputs": [],
   "source": [
    "response_prompt_template = \"\"\"You are an expert of world knowledge. I am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them if they are not relevant.\n",
    "\n",
    "{normal_context}\n",
    "\n",
    "Original Question: {question}\n",
    "Answer:\"\"\"\n",
    "response_prompt = ChatPromptTemplate.from_template(response_prompt_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "id": "06335ebb",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = (\n",
    "    {\n",
    "        # Retrieve context using the normal question (only the first 3 results)\n",
    "        \"normal_context\": RunnableLambda(lambda x: x[\"question\"]) | retriever,\n",
    "        # Pass on the question\n",
    "        \"question\": lambda x: x[\"question\"],\n",
    "    }\n",
    "    | response_prompt\n",
    "    | ChatOpenAI(temperature=0)\n",
    "    | StrOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "id": "15e0e741",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Yes, ChatGPT was around while Donald Trump was president. However, it is important to note that the specific context you provided mentions that ChatGPT refused to write a positive poem about former President Donald Trump. This suggests that while ChatGPT was available during Trump's presidency, it may have had limitations or biases in its responses regarding him.\""
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": question})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7b9e5d6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}