Add TriggerPhraseLogitsProcessor for vllm and transformers (#8)

* Add TriggerPhraseLogitsProcessor for vllm

* Add <think> utility for genLengthLogitsProcessor for vllm

* Update vllm notebooks

* Update transformers notebooks

* Fix ignoring immediate trigger bug

Author: aerdem4

example_notebooks/transformers/cite_prompt_logits_processor.ipynb

@@ -2,17 +2,25 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "28ed6952",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "/home/aerdem/projects/nvidia/logits-processor-zoo\n"
+     ]
+    }
+   ],
    "source": [
     "%cd ../.."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "id": "a85f8503",
    "metadata": {},
    "outputs": [],
@@ -54,6 +62,13 @@
    "id": "cbf4c2d5",
    "metadata": {},
    "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n"
+     ]
+    },
     {
      "name": "stdout",
      "output_type": "stream",
@@ -65,11 +80,16 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "The user seems to think that the product is worth its high price. They describe it as \"very soft\" and \"colorful,\" suggesting that these features contribute positively to their overall impression of the item. The phrase \"deserves its price\" indicates satisfaction with the value for money. Additionally, they mention that the product is \"stylish,\" which adds another positive aspect to their opinion.\n",
+      "The user seems to have mixed feelings about the price of the product. They find it \"expensive,\" which might be due to several factors:\n",
       "\n",
-      "Overall, the user appears to have a favorable view of the product despite its higher cost, appreciating both its quality and design.\n",
-      "-----END-----\n",
+      "1. **Quality and Quality Control**: If the product is described as \"very soft\" and \"colorful,\" it suggests that it may be of high quality or unique in some way.\n",
+      "\n",
+      "2. **Design and Design Elements**: Describing the design as \"stylish\" indicates that the product has an appealing aesthetic appeal, which can sometimes come at a higher cost.\n",
+      "\n",
+      "3. **Brand Reputation**: The fact that the product is described as \"deserving its price\" implies that it represents good value for money, suggesting that it offers more than just basic functionality; perhaps it also provides additional features or materials that justify the higher price point.\n",
       "\n",
+      "Overall, while the user appreciates the product's appearance and style, they seem to feel that the price is too high relative to these positive attributes. This could mean there might not be enough information provided to determine if the price reflects the true value of the product. It would help to know more details about the product's features, materials used, and any other relevant specifications to better understand their perspective on pricing.\n",
+      "-----END-----\n",
       "\n",
       "Prompt: \n",
       "    Retrieved information:\n",
@@ -81,9 +101,8 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "A Pokémon is a fictional creature that exists within the Pokémon media franchise. These creatures have unique abilities or characteristics that allow them to interact with their environment and other Pokémon. They can be trained by players through various means such as battling, collecting, or participating in events. Pokémon are often depicted as having magical or supernatural powers, though they may also possess more mundane attributes depending on the specific type of Pokémon.\n",
+      "A Pokémon is a fictional creature that exists within the Pokémon media franchise. These creatures have unique abilities or characteristics that allow them to interact with their environment and other characters in various ways. Pokémon can be found in different types of habitats such as forests, mountains, rivers, and cities, and they often serve as companions or allies for trainers who embark on adventures. Each Pokémon has its own distinct appearance, moveset, and personality traits, making them beloved figures in popular culture.\n",
       "-----END-----\n",
-      "\n",
       "\n"
      ]
     }
@@ -117,10 +136,9 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "The user's opinion about the product's price is that it is worth the cost, as they describe the product as \"very soft, colorful, expensive but deserves its price.\" This indicates that the user believes the product is of high quality and is not overpriced for its features, despite its higher price point.\n",
+      "The user seems to have mixed feelings about the price of the product. They find it expensive, but they also appreciate its softness, colorfulness, and style, which suggests that the product is well-made and worth the cost.\n",
       "-----END-----\n",
       "\n",
-      "\n",
       "Prompt: \n",
       "    Retrieved information:\n",
       "    Pokémon is a Japanese media franchise consisting of video games, animated series and films, a trading card game, and other related media. \n",
@@ -131,9 +149,8 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "A Pokémon is a fictional creature that exists within the Pokémon media franchise, including video games, animated series, films, trading cards, and other related media. These creatures have unique abilities or characteristics, often derived from the environment they inhabit, and can be trained by players or collectors for various purposes, such as battling opponents, participating in competitions, or simply collecting them as collectibles.\n",
+      "A Pokémon is a fictional creature in the Pokémon franchise, which is a Japanese media franchise consisting of video games, animated series, films, a trading card game, and other related media. The franchise takes place in a shared universe where humans coexist with Pokémon, a large variety of species endowed with special powers. The franchise's target audience is children aged 5 to 12, but it is known to attract people of all ages.\n",
       "-----END-----\n",
-      "\n",
       "\n"
      ]
     }
@@ -170,9 +187,10 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "The user seems to think that while the product might be more expensive than other options available in the market (described as \"very soft\" and \"colorful\"), it still warrants this higher cost due to its quality and style. The phrase \"deserves its price\" indicates that they believe the item provides good value for money. Therefore, their overall impression of the product’s pricing seems positive; despite being costly, they consider it worth purchasing because of its appealing features like colorfulness and style.\n",
-      "-----END-----\n",
+      "The reviewer seems to have mixed feelings towards the pricing of the product. They describe it as \"expensive\" and \"deserves its price,\" which suggests that they find it worth paying for its quality or unique features. The use of words like \"stylish\" further emphasizes their positive impression.\n",
       "\n",
+      "So in summary, while they appreciate the design and style of the product, they also acknowledge that it might be quite pricey. Therefore, their overall sentiment can be described as **mixed**, with appreciation for both aspects (quality and style).\n",
+      "-----END-----\n",
       "\n",
       "Prompt: \n",
       "    Retrieved information:\n",
@@ -184,9 +202,12 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "A Pokémon is a fictional creature that exists within the Pokémon media franchise. These creatures have unique abilities or characteristics that allow them to interact with their environment and other Pokémon. They can be trained by players through various means such as battling against each other or participating in events. Pokémon are often depicted as having magical or supernatural powers and are commonly associated with adventure stories and gameplay experiences.\n",
-      "-----END-----\n",
+      "A Pokémon is an imaginary creature that exists within the fictional world of the Pokémon franchise. These creatures have unique abilities or characteristics that allow them to interact with their environment or engage in battles against each other.\n",
+      "\n",
+      "Pokémon can be found throughout various regions on Earth (or \"Gym Leaders\") where trainers can catch them using Poké Balls or capture them through battle encounters. Each type of Pokémon has specific attributes such as strength, speed, or ability to use certain moves.\n",
       "\n",
+      "The concept behind Pokémon revolves around the idea that these magical beings exist alongside human society, providing entertainment through gameplay experiences like battling or collecting cards representing different types of Pokémon. This blend of fantasy elements combined with interactive gaming mechanics makes the Pokémon franchise appealing across diverse age groups due to its accessibility for young audiences while also attracting older fans who appreciate the depth and complexity of the character designs and storylines.\n",
+      "-----END-----\n",
       "\n"
      ]
     }
@@ -201,7 +222,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -215,7 +236,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.7"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,

example_notebooks/transformers/force_last_phrase_logits_processor.ipynb

@@ -2,17 +2,25 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "28ed6952",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "/home/aerdem/projects/nvidia/logits-processor-zoo\n"
+     ]
+    }
+   ],
    "source": [
     "%cd ../.."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "id": "a85f8503",
    "metadata": {},
    "outputs": [],
@@ -51,6 +59,13 @@
    "id": "cbf4c2d5",
    "metadata": {},
    "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n"
+     ]
+    },
     {
      "name": "stdout",
      "output_type": "stream",
@@ -67,15 +82,12 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "Bulbasaur is a fictional Pokémon species that appears in the Pokémon franchise. It was designed by Atsuko Nishida and first appeared in Pocket Monsters: Red and Green (Pokémon Red and Blue outside Japan). As a Grass and Poison-type Pokémon, Bulbasaur is considered to be one of the most iconic characters in the franchise.\n",
+      "Bulbasaur is a fictional Pokémon species that appears in the Pokémon franchise. It was designed by Atsuko Nishida and first appeared in the original Pokémon games, specifically in Pokémon Red and Blue outside of Japan. As a Grass and Poison-type Pokémon, Bulbasaur serves as the starting Pokémon for players to catch and train.\n",
       "\n",
-      "Since its debut, Bulbasaur has been featured in numerous sequels, spin-off games, related merchandise, and animated and printed adaptations of the franchise. In the Pokémon anime, it serves as a central character, being one of Ash Ketchum's main Pokémon for the first season. Later on, another Pokémon named May obtains a different Bulbasaur as her own.\n",
+      "Throughout the series, Bulbasaur has been featured in numerous sequels, spin-offs, related merchandise, and animated and printed adaptations of the franchise. It plays an important role in the Pokémon anime, where it is one of Ash Ketchum's main Pokémon during the first season. Later on, another Pokémon named May obtains a different version of Bulbasaur as her own.\n",
       "\n",
-      "In addition to its appearances in the anime, Bulbasaur also features prominently in various manga series, including those written by Satoshi Hoshino. Furthermore, it is owned by the protagonist Red in the Pokémon Adventures game.\n",
-      "\n",
-      "Overall, Bulbasaur is an important part of the Pokémon universe and continues to be popular among fans of the franchise.\n",
+      "In addition to its appearances in the anime, Bulbasaur also features prominently in various manga stories and is owned by the protagonist Red in the Pokémon Adventures game. The Pokémon has become a beloved character within the franchise due to its enduring popularity across multiple platforms and generations of fans.\n",
       "-----END-----\n",
-      "\n",
       "\n"
      ]
     }
@@ -114,17 +126,14 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "Bulbasaur is a fictional Pokémon species that appears in the Pokémon franchise. It was designed by Atsuko Nishida and first appeared in Pocket Monsters: Red and Green (Pokémon Red and Blue outside Japan). As a Grass and Poison-type Pokémon, Bulbasaur is considered to be one of the most iconic characters in the franchise.\n",
+      "Bulbasaur is a fictional Pokémon species that appears in the Pokémon franchise. It was designed by Atsuko Nishida and first appeared in the original Pokémon games, specifically in Pokémon Red and Blue outside of Japan. As a Grass and Poison-type Pokémon, Bulbasaur serves as the starting Pokémon for players to catch and train.\n",
       "\n",
-      "Since its debut, Bulbasaur has been featured in numerous sequels, spin-off games, related merchandise, and animated and printed adaptations of the franchise. In the Pokémon anime, it serves as a central character, being one of Ash Ketchum's main Pokémon for the first season. Later on, another Pokémon named May obtains a different Bulbasaur as her own.\n",
+      "Throughout the series, Bulbasaur has been featured in numerous sequels, spin-offs, related merchandise, and animated and printed adaptations of the franchise. It plays an important role in the Pokémon anime, where it is one of Ash Ketchum's main Pokémon during the first season. Later on, another Pokémon named May obtains a different version of Bulbasaur as her own.\n",
       "\n",
-      "In addition to its appearances in the anime, Bulbasaur also features prominently in various manga series, including those written by Satoshi Hoshino. Furthermore, it is owned by the protagonist Red in the Pokémon Adventures game.\n",
-      "\n",
-      "Overall, Bulbasaur is an important part of the Pokémon universe and continues to be popular among fans of the franchise.\n",
+      "In addition to its appearances in the anime, Bulbasaur also features prominently in various manga stories and is owned by the protagonist Red in the Pokémon Adventures game. The Pokémon has become a beloved character within the franchise due to its enduring popularity across multiple platforms and generations of fans.\n",
       "\n",
       "References: Wikipedia article on Bulbasaur.\n",
       "-----END-----\n",
-      "\n",
       "\n"
      ]
     }
@@ -169,17 +178,14 @@
       "    \n",
       "\n",
       "LLM response:\n",
-      "Bulbasaur is a fictional Pokémon species that appears in the Pokémon franchise. It was designed by Atsuko Nishida and first appeared in Pocket Monsters: Red and Green (Pokémon Red and Blue outside Japan). As a Grass and Poison-type Pokémon, Bulbasaur is considered to be one of the most iconic characters in the franchise.\n",
+      "Bulbasaur is a fictional Pokémon species that appears in the Pokémon franchise. It was designed by Atsuko Nishida and first appeared in the original Pokémon games, specifically in Pokémon Red and Blue outside of Japan. As a Grass and Poison-type Pokémon, Bulbasaur serves as the starting Pokémon for players to catch and train.\n",
       "\n",
-      "Since its debut, Bulbasaur has been featured in numerous sequels, spin-off games, related merchandise, and animated and printed adaptations of the franchise. In the Pokémon anime, it serves as a central character, being one of Ash Ketchum's main Pokémon for the first season. Later on, another Pokémon named May obtains a different Bulbasaur as her own.\n",
+      "Throughout the series, Bulbasaur has been featured in numerous sequels, spin-offs, related merchandise, and animated and printed adaptations of the franchise. It plays an important role in the Pokémon anime, where it is one of Ash Ketchum's main Pokémon during the first season. Later on, another Pokémon named May obtains a different version of Bulbasaur as her own.\n",
       "\n",
-      "In addition to its appearances in the anime, Bulbasaur also features prominently in various manga series, including those written by Satoshi Hoshino. Furthermore, it is owned by the protagonist Red in the Pokémon Adventures game.\n",
+      "In addition to its appearances in the anime, Bulbasaur also features prominently in various manga stories and is owned by the protagonist Red in the Pokémon Adventures game. The Pokémon has become a beloved character within the franchise due to its enduring popularity across multiple platforms and generations of fans.\n",
       "\n",
-      "Overall, Bulbasaur is an important part of the Pokémon universe and continues to be popular among fans of the franchise.\n",
-      "\n",
-      "Thanks for trying our RAG application! If you have more questions about Bulbasaur or any other topic, feel free to ask. We're here to help!\n",
+      "Thanks for trying our RAG application! If you have more questions about Bulbasaur or any other topic, feel free to ask!\n",
       "-----END-----\n",
-      "\n",
       "\n"
      ]
     }
@@ -196,7 +202,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -210,7 +216,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.7"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,