Skip to content

Commit df7f158

Browse files
aerdem4aerdem4
authored
Add conditional boost to CiteFromPrompt (#20)
Signed-off-by: aerdem4 <ahmeterd4@gmail.com>
1 parent 00520ae commit df7f158

File tree

8 files changed

+112
-118
lines changed

8 files changed

+112
-118
lines changed

‎example_notebooks/transformers/cite_prompt_logits_processor.ipynb‎

Lines changed: 28 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,15 @@
2323
"execution_count": 2,
2424
"id": "a85f8503",
2525
"metadata": {},
26-
"outputs": [],
26+
"outputs": [
27+
{
28+
"name": "stderr",
29+
"output_type": "stream",
30+
"text": [
31+
"Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.\n"
32+
]
33+
}
34+
],
2735
"source": [
2836
"from example_notebooks.transformers.utils import LLMRunner\n",
2937
"from logits_processor_zoo.transformers import CiteFromPromptLogitsProcessor\n",
@@ -136,7 +144,12 @@
136144
" \n",
137145
"\n",
138146
"LLM response:\n",
139-
"The user seems to have mixed feelings about the price of the product. They find it expensive, but they also appreciate its softness, colorfulness, and style.\n",
147+
"The user's opinion about the product's price is mixed. They describe it as \"expensive,\" which could be interpreted in two ways:\n",
148+
"\n",
149+
"1. The user might consider the price to be high for its quality and features.\n",
150+
"2. Alternatively, they may appreciate the product's price, considering its stylish design or unique qualities.\n",
151+
"\n",
152+
"Without more context, it's difficult to determine the user's opinion definitively.\n",
140153
"-----END-----\n",
141154
"\n",
142155
"Prompt: \n",
@@ -149,7 +162,7 @@
149162
" \n",
150163
"\n",
151164
"LLM response:\n",
152-
"A Pokémon is a fictional creature in the Pokémon franchise, which is a Japanese media franchise consisting of video games, animated series, films, a trading card game, and other related media. The franchise takes place in a shared universe where humans coexist with Pokémon, a large variety of species endowed with special powers. The franchise's target audience is children aged 5 to 12, but it is known to attract people of all ages.\n",
165+
"A Pokémon is a fictional creature in the Pokémon franchise, a Japanese media franchise consisting of video games, animated series and films, a trading card game, and other related media. The franchise takes place in a shared universe in which humans co-exist with creatures known as Pokémon, a large variety of species endowed with special powers. The franchise's target audience is children aged 5 to 12, but it is known to attract people of all ages.\n",
153166
"-----END-----\n",
154167
"\n"
155168
]
@@ -158,7 +171,8 @@
158171
"source": [
159172
"runner.generate_response(\n",
160173
" example_prompts,\n",
161-
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=2.0, boost_eos=False)]\n",
174+
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=2.0, boost_eos=False,\n",
175+
" conditional_boost_factor=2.0)]\n",
162176
")"
163177
]
164178
},
@@ -187,17 +201,9 @@
187201
" \n",
188202
"\n",
189203
"LLM response:\n",
190-
"The reviewer seems to have mixed feelings towards the pricing of the product:\n",
191-
"\n",
192-
"- They describe it as \"very soft\" and \"colorful\", suggesting that they appreciate these qualities.\n",
204+
"The reviewer seems to have mixed feelings about the pricing of the product. They describe it as \"expensive\" and \"deserves its price,\" which suggests that they find the high cost justified by the quality or value of the item. The use of words like \"stylish\" further emphasizes their positive impression of the style and design of the product.\n",
193205
"\n",
194-
"- They also mention that it is \"expensive,\" which might be seen as negative if you're looking for an affordable option or if this was their first time buying something like this.\n",
195-
"\n",
196-
"- However, they state that it \"deserves its price,\" indicating that they believe the high cost reflects on quality or value.\n",
197-
"\n",
198-
"Overall, while they seem satisfied with the overall experience and don't mind paying more for what they perceive as good-quality materials and design, they may feel that the price point could be higher than expected for everyday use or budget-conscious shoppers.\n",
199-
"\n",
200-
"So in summary, they find the item to be well-made and aesthetically pleasing despite feeling that it might not be suitable for everyone due to being too pricey for some people's budgets. The reviewer seems generally positive toward the purchase decision itself rather than just the specific item.\n",
206+
"So in summary, while the reviewer finds the price somewhat high, they believe it is worth the investment due to the overall quality and style of the item. This indicates an average-to-good level of satisfaction with the purchase experience.\n",
201207
"-----END-----\n",
202208
"\n",
203209
"Prompt: \n",
@@ -210,11 +216,13 @@
210216
" \n",
211217
"\n",
212218
"LLM response:\n",
213-
"A Pokémon is an imaginary creature that exists within the fictional world of the Pokémon franchise. These creatures have unique abilities or characteristics that allow them to interact with their environment or engage in battles against each other.\n",
219+
"A Pokémon is an imaginary creature that exists within the fictional world of the Pokémon franchise. These creatures have unique abilities or characteristics that allow them to interact with their environment and engage in various activities.\n",
220+
"\n",
221+
"Pokémon can be found throughout the vast landscapes depicted in the franchise, including forests, mountains, rivers, cities, and even outer space. They come in different sizes, shapes, colors, and types (e.g., water-type, fire-type). Each Pokémon has its own distinct personality and backstory.\n",
214222
"\n",
215-
"Pokémon can be found throughout various regions on Earth (or \"Gym Leaders\") where trainers can catch them using Poké Balls or capture them through battle encounters. Each type of Pokémon has specific attributes such as strength, speed, or ability to use certain moves.\n",
223+
"The concept of Pokémon originated from Japan in the early 1990s when Satoshi Tajiri developed the first generation of the Pokémon Red and Blue games for Nintendo Entertainment System (NES). Since then, numerous generations of Pokémon games have been released across multiple platforms, expanding the world of Pokémon into anime, manga, movies, TV shows, books, trading cards, and more.\n",
216224
"\n",
217-
"The concept behind Pokémon revolves around the idea that these magical beings exist alongside human society, providing entertainment through gameplay experiences like battling or collecting cards representing different types of Pokémon. This blend of fantasy elements combined with interactive gaming mechanics makes the Pokémon franchise appealing across diverse age groups due to its accessibility for young audiences while also attracting older fans who appreciate the depth and complexity of the character designs and storylines.\n",
225+
"In summary, Pokémon are magical beings that exist alongside human characters in the Pokémon universe, each possessing unique traits and abilities that make them fascinating subjects for storytelling and gaming experiences.\n",
218226
"-----END-----\n",
219227
"\n"
220228
]
@@ -223,7 +231,8 @@
223231
"source": [
224232
"runner.generate_response(\n",
225233
" example_prompts,\n",
226-
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=-2.0, boost_eos=False)]\n",
234+
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=-1.0, boost_eos=False,\n",
235+
" conditional_boost_factor=-1.0)]\n",
227236
")"
228237
]
229238
},
@@ -252,7 +261,7 @@
252261
"name": "python",
253262
"nbconvert_exporter": "python",
254263
"pygments_lexer": "ipython3",
255-
"version": "3.10.13"
264+
"version": "3.10.17"
256265
}
257266
},
258267
"nbformat": 4,

‎example_notebooks/vllm/cite_prompt_logits_processor.ipynb‎

Lines changed: 32 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -28,45 +28,22 @@
2828
"name": "stdout",
2929
"output_type": "stream",
3030
"text": [
31-
"WARNING 04-30 15:00:30 cuda.py:22] You are using a deprecated `pynvml` package. Please install `nvidia-ml-py` instead. See https://pypi.org/project/pynvml for more information.\n"
32-
]
33-
},
34-
{
35-
"name": "stderr",
36-
"output_type": "stream",
37-
"text": [
38-
"/home/aerdem/projects/LLM/llmenv/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
39-
" warnings.warn(\n"
40-
]
41-
},
42-
{
43-
"name": "stdout",
44-
"output_type": "stream",
45-
"text": [
46-
"WARNING 04-30 15:00:33 config.py:1563] Casting torch.bfloat16 to torch.float16.\n",
47-
"INFO 04-30 15:00:33 llm_engine.py:184] Initializing an LLM engine (v0.5.5) with config: model='Qwen/Qwen2.5-1.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-1.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/Qwen2.5-1.5B-Instruct, use_v2_block_manager=False, enable_prefix_caching=False)\n"
48-
]
49-
},
50-
{
51-
"name": "stderr",
52-
"output_type": "stream",
53-
"text": [
54-
"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
55-
]
56-
},
57-
{
58-
"name": "stdout",
59-
"output_type": "stream",
60-
"text": [
61-
"INFO 04-30 15:00:34 model_runner.py:879] Starting to load model Qwen/Qwen2.5-1.5B-Instruct...\n",
62-
"INFO 04-30 15:00:34 weight_utils.py:236] Using model weights format ['*.safetensors']\n",
63-
"INFO 04-30 15:00:35 weight_utils.py:280] No model.safetensors.index.json found in remote.\n"
31+
"INFO 05-22 14:12:47 [__init__.py:239] Automatically detected platform cuda.\n",
32+
"WARNING 05-22 14:12:50 [config.py:2972] Casting torch.bfloat16 to torch.float16.\n",
33+
"INFO 05-22 14:12:55 [config.py:717] This model supports multiple tasks: {'reward', 'generate', 'classify', 'score', 'embed'}. Defaulting to 'generate'.\n",
34+
"WARNING 05-22 14:12:55 [cuda.py:93] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used\n",
35+
"INFO 05-22 14:12:55 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.5.post1) with config: model='Qwen/Qwen2.5-1.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-1.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen2.5-1.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={\"splitting_ops\":[],\"compile_sizes\":[],\"cudagraph_capture_sizes\":[],\"max_capture_size\":0}, use_cached_outputs=False, \n",
36+
"INFO 05-22 14:12:56 [cuda.py:292] Using Flash Attention backend.\n",
37+
"INFO 05-22 14:12:57 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0\n",
38+
"INFO 05-22 14:12:57 [model_runner.py:1108] Starting to load model Qwen/Qwen2.5-1.5B-Instruct...\n",
39+
"INFO 05-22 14:12:57 [weight_utils.py:265] Using model weights format ['*.safetensors']\n",
40+
"INFO 05-22 14:12:58 [weight_utils.py:315] No model.safetensors.index.json found in remote.\n"
6441
]
6542
},
6643
{
6744
"data": {
6845
"application/vnd.jupyter.widget-view+json": {
69-
"model_id": "e9c350b056a04694bf4f2eade35244ba",
46+
"model_id": "d29121d7259a47f5923ef4d1b3fa3138",
7047
"version_major": 2,
7148
"version_minor": 0
7249
},
@@ -81,8 +58,14 @@
8158
"name": "stdout",
8259
"output_type": "stream",
8360
"text": [
84-
"INFO 04-30 15:00:36 model_runner.py:890] Loading model weights took 2.8875 GB\n",
85-
"INFO 04-30 15:00:38 gpu_executor.py:121] # GPU blocks: 37541, # CPU blocks: 9362\n"
61+
"INFO 05-22 14:12:58 [loader.py:458] Loading weights took 0.57 seconds\n",
62+
"INFO 05-22 14:12:58 [model_runner.py:1140] Model loading took 2.8876 GiB and 1.613375 seconds\n",
63+
"INFO 05-22 14:13:00 [worker.py:287] Memory profiling takes 1.73 seconds\n",
64+
"INFO 05-22 14:13:00 [worker.py:287] the current vLLM instance can use total_gpu_memory (23.66GiB) x gpu_memory_utilization (0.90) = 21.29GiB\n",
65+
"INFO 05-22 14:13:00 [worker.py:287] model weights take 2.89GiB; non_torch_memory takes 0.06GiB; PyTorch activation peak memory takes 2.02GiB; the rest of the memory reserved for KV Cache is 16.32GiB.\n",
66+
"INFO 05-22 14:13:00 [executor_base.py:112] # cuda blocks: 38207, # CPU blocks: 9362\n",
67+
"INFO 05-22 14:13:00 [executor_base.py:117] Maximum concurrency for 32768 tokens per request: 18.66x\n",
68+
"INFO 05-22 14:13:02 [llm_engine.py:437] init engine (profile, create kv cache, warmup model) took 3.24 seconds\n"
8669
]
8770
}
8871
],
@@ -93,8 +76,7 @@
9376
"\n",
9477
"example_prompts =[\n",
9578
" \"\"\"\n",
96-
" A user review: very soft, colorful, expensive but deserves its price.\n",
97-
" I would like to wear it in my friend's wedding.\n",
79+
" A user review: very soft, colorful, expensive but deserves its price, stylish.\n",
9880
" \n",
9981
" What is the user's opinion about the product's price?\n",
10082
" \"\"\",\n",
@@ -130,12 +112,11 @@
130112
"output_type": "stream",
131113
"text": [
132114
"Prompt: \n",
133-
" A user review: very soft, colorful, expensive but deserves its price.\n",
134-
" I would like to wear it in my friend's wedding.\n",
115+
" A user review: very soft, colorful, expensive but deserves its price, stylish.\n",
135116
" \n",
136117
" What is the user's opinion about the product's price?\n",
137118
" \n",
138-
"The user's opinion about the product's price is that it is expensive, but they believe it is worth the price.\n",
119+
"The user's opinion about the product's price is that it is expensive, but they believe it is worth the price due to its softness, colorfulness, and stylish design.\n",
139120
"-----END-----\n",
140121
"\n",
141122
"Prompt: \n",
@@ -175,12 +156,11 @@
175156
"output_type": "stream",
176157
"text": [
177158
"Prompt: \n",
178-
" A user review: very soft, colorful, expensive but deserves its price.\n",
179-
" I would like to wear it in my friend's wedding.\n",
159+
" A user review: very soft, colorful, expensive but deserves its price, stylish.\n",
180160
" \n",
181161
" What is the user's opinion about the product's price?\n",
182162
" \n",
183-
"The user's opinion about the product's price is that it is expensive, but the user is willing to pay the price to wear it in a friend's wedding.\n",
163+
"The user's opinion about the product's price is that it is expensive but deserves its price, stylish.\n",
184164
"-----END-----\n",
185165
"\n",
186166
"Prompt: \n",
@@ -191,15 +171,16 @@
191171
" \n",
192172
" Can you shortly describe what Pokémon is?\n",
193173
" \n",
194-
" Pokémon is a Japanese media franchise consisting of video games, animated series, and films. The franchise takes place in a shared universe in which humans co-exist with Pokémon, a large variety of species endowed with special powers. The franchise's target audience is children aged 5 to 12, but it is known to attract people of all ages.\n",
174+
"Pokémon is a Japanese media franchise consisting of video games, animated series and films, a trading card game, and other related media. The franchise takes place in a shared universe in which humans co-exist with creatures known as Pokémon, a large variety of species endowed with special powers. The franchise's target audience is children aged 5 to 12, but it is known to attract people of all ages.\n",
195175
"-----END-----\n",
196176
"\n"
197177
]
198178
}
199179
],
200180
"source": [
201181
"runner.generate_response(example_prompts,\n",
202-
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=5.0, boost_eos=False)])"
182+
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=1.0, boost_eos=False,\n",
183+
" conditional_boost_factor=3.0)])"
203184
]
204185
},
205186
{
@@ -221,16 +202,11 @@
221202
"output_type": "stream",
222203
"text": [
223204
"Prompt: \n",
224-
" A user review: very soft, colorful, expensive but deserves its price.\n",
225-
" I would like to wear it in my friend's wedding.\n",
205+
" A user review: very soft, colorful, expensive but deserves its price, stylish.\n",
226206
" \n",
227207
" What is the user's opinion about the product's price?\n",
228208
" \n",
229-
"The user's opinion about the product's price seems to be mixed. They appreciate that the product is \"very soft\" and \"colorful,\" indicating that these features contribute positively to their satisfaction with the item. However, they also mention that the product is \"expensive,\" which suggests that they feel the price is justified based on the quality they perceive.\n",
230-
"\n",
231-
"The phrase \"deserves its price\" implies that the user believes the cost of the product is appropriate for what they have received. This indicates that they find value in the product and feel that they are getting good value for their money.\n",
232-
"\n",
233-
"In summary, while the user appreciates the product's qualities and finds them worth the price, they also acknowledge that the cost is higher than they might have expected for such features. This suggests that they view the product as a good investment for their needs and preferences.\n",
209+
"The user's opinion about the product's price is that it is expensive, but they believe it is worth the cost due to its softness, colorfulness, and style.\n",
234210
"-----END-----\n",
235211
"\n",
236212
"Prompt: \n",
@@ -241,15 +217,16 @@
241217
" \n",
242218
" Can you shortly describe what Pokémon is?\n",
243219
" \n",
244-
"Pokémon is a popular Japanese media franchise that features a world where humans live alongside magical creatures called Pokémon. These Pokémon have unique abilities that allow them to fight alongside humans in various adventures. The franchise includes video games, animated series, films, trading cards, and other forms of media aimed at children aged 5 to 12, though it has also gained popularity among adults.\n",
220+
"Pokémon is a popular Japanese media franchise that features a world where humans live alongside Pokémon, mythical creatures with unique abilities. It targets children aged 5 to 12 but has broad appeal across all age groups.\n",
245221
"-----END-----\n",
246222
"\n"
247223
]
248224
}
249225
],
250226
"source": [
251227
"runner.generate_response(example_prompts,\n",
252-
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=-2.0, boost_eos=False)])"
228+
" [CiteFromPromptLogitsProcessor(runner.tokenizer, boost_factor=-1.0, boost_eos=False,\n",
229+
" conditional_boost_factor=-1.0)])"
253230
]
254231
},
255232
{

0 commit comments

Comments
 (0)