LLM-RE-Benchmark: Can LLMs identify functional requirements information?

A benchmark for evaluating LLMs regarding their ability to recognize and extract functional requirement information in natural language texts.

Repository Content

files	Description
BenchmarkRequirements.json	Dataset on which the LLMs run. Contains text excerpts with gold standard answers and metadata.
DataLoader.py	Loads the text excerpts and reference solutions from BenchmarkRequirements.json.
Evaluate.py	Evaluates LLMs on the dataset. This file must be executed to run LLMs through the benchmark. The results of the evaluation are stored in a csv file. Furthermore, functions from MonitorResults.py are executed to analyze the results.
EvaluateJSON.py	Analyzes the data set stored in the json file. The result can be used for further analysis purposes.
Metrics.py	Contains functions with metrics.
ModelWrapper.py	Ensures that prompts are forwarded to the configured LLMs. In our version, various interfaces are addressed that forward to models located on different hardware. The following Python files are controlled: GPTModelsAPI.py, OllamaClient.py, OllamaModels.py. The last two files are not located in this repository, as they address locally operated LLMs. Both Python files only contain functions that pass the transferred prompt to a specific LLM and return the response in the form of an unmodified string. They must be replaced with own files as needed.
MonitorResults.py	This file contains functions for analyzing the results, producing tabular output via console, and generating various diagrams. This file can also be started directly. In the main function, a csv file containing the results of past experiments can be referenced. Analyses are then performed based on the data.
Prompts.py	Contains the prompt.
GPTModelsAPI.py	Forwards the prompt to a gpt model via OpenAI API and returns the response as an unmodified string.
text_analysis_results.csv	This file contains analysis information about the dataset. It was generated by executing the file EvaluateJSON.py.
results_per_textsegment.csv	Contains the results for each analyzed text excerpt during an experiment.
cumulativ_results_per_textsegment.csv	Contains the results for each analyzed text excerpt from all past experiments and continuously stores all data. Must be reset manually if necessary.

Installation

• Python ≥ 3.12
• LLMs run on different hardware (Access must be set up individually)

Access to LLMs

During the execution of the benchmark, the prompt contained in Prompt.py is extended with the text excerpt from the dataset and forwarded to an LLM. The response generated by the LLM is then analyzed. The ModelWrapper.py file contains the function model_wrapper(prompt: str, model: str). It serves as an interface between the benchmark and different LLMs. When this function is called, the prompt and the addressed model are transmitted. The model's response is returned in the form of a string. When installing the benchmark, the LLMs must be added individually. A function must be added that addresses the target model. In our experiments, we used LLMs that we ran on local hardware using Ollama. The code that addresses our local models is not uploaded to this repository. However, we have uploaded the file “GPTModelsAPI.py.” This contains code to access the gpt model via the OpenAI API. When using this code, the individual OpenAI key must be added.

Configuration and execution of benchmark

Start the benchmark with a specific configuration by running the file “Evaluate.py”. The configuration can be set up via the main function of this Python file. The following settings are possible:

• json_file: Refers to the JSON file containing the dataset.
• thresholds: Contains a list of threshold values for evaluating the similarity between the LLM-generated solution and the reference solution. The dataset is run through once with each specified model using each threshold value.
• models: Contains a list of all LLMs that are to run through the data set during the experiment. Here, too, each LLM runs through the data set once using each listed threshold value. The exact name of the LLM is relevant for its correct selection. In the ModelWrapper.py file, the prompt is forwarded to the respective LLM based on its name.
• runs_per_model: Allows any number of experiment runs with the set settings.

Extensibility

The framework has a modular structure:

• New metrics can be easily added under Metrics.py.
• New LLMs can be added via ModelWrapper.py.
• Additional data can be easily appended to the JSON file. In this case, however, EvaluateJSON.py should be executed first.