Minuteman

Minuteman is a tool to help users with meeting minuting (producing a summary of the meeting). It works with the JitSi meeting platform. The user connects the tool to a meeting, a transcript is generated and meeting minutes are iteratively being created. An example instance is available at minuteman.kmjec.cz. Currently supported languages are: English and we are working on supporting Czech. The system was developed under supervision of doc. RNDr. Ondřej Bojar, Ph.D. at Charles University, read more in our paper: https://arxiv.org/abs/2309.05272

Current development

This project is still actively developed. We are working on multilingual transcriptions/summaries and investigating the possibility to not rely on Fairmeeting (Jitsy) but rather allow any form of audio input while retaining decent speaker diarization.

User documentation

On the app main page, you will be greeted with instruction and a possibility to create a new session. Upon creating the session, you will be taken to a new page with a control bar and two editors. The control bar has an input field for you to enter a Jitsi room name. When you enter a room name from fairmeeting.net and hit Select room, a new meeting member will pop up in the meeting. It will record all the audio from the meeting participants and transcribe it in real time. The resulting transcript will appear live in the left editor.

Once enough transcript data has been accumulated, summary bullet points will start appearing in the right editor. These bullet points are generated by a BART summarization model. If you want to summarize a particular part of the meeting, you can highlight it in the transcript editor and press Ctrl + Alt + S. This will send the highlighted text to the summarization model and create a bullet point in the summary editor.

As the transcription mechanism is not perfect, you may want to edit the transcript. You can do that freely. The summary bullet points that correspond to the edited text will be updated accordingly. If you want to edit the summary bullet points, you can do that too. Once you edit a bullet point, it will be frozen and will not be updated again even if the corresponding part of the transcript has been edited.

When you are done with the meeting, you can press the disconnect button in the control bar.

If you want to invite meeting participants into the session, just send them the link you are connected to. They will be able to correct the transcript together with you. Just warn them not to connect the tool again; that would mean two clients would be recording the meeting in parallel, resulting in a double transcription.

Running the application

For running Minuteman, we provide two docker compose files. The configuration provided in docker-compose-dev.yml is targeted at running the tool locally without a GPU and the models in TorchServe. The configuration in docker-compose.yml is meant for running in a production environment, including the summarization models, and thus requires the GPU. In the compose files, configuration for the application components is specified using environment variables. Each optionally configurable variable is highlighted with a comment beginning with an OPTION comment, while the variables that are necessary to set before running are highlighted with the TODO comment.

The build process was tested using Docker version 24.0.2 and docker compose version 2.19.1.

We first need to generate the Etherpad API key and download a VAD model by running init_app_files.sh. Then, we fill out the required TODO fields in the respective compose file; specifically, it means setting the FLASK_SECRET_KEY in the Flask container and ADMIN_PASSWORD in the Etherpad container.

Development

For development, the only thing we need to do now is run the following commands:

docker compose -f docker-compose-dev.yml build
docker compose -f docker-compose-dev.yml up

And after downloading the necessary containers and building the dependencies, we can open localhost:7777 for the Minuteman title page. Note that on startup, starting RabbitMQ and Etherpad usually takes a while, so there will be error messages from the containers periodically trying to connect to the queue before RabbitMQ has had enough time to start. After the queue comes online, it will work normally.

Production

For running in production, we need to download the summarization models. To do that, we run download_summ_models.sh in the torch_model_dir directory. We also need to set an additional option, ETHERPAD_URL, to the domain we have pointed to our Etherpad editors for iframes. After that, the commands to run are the same as for the development version, except with a different compose file.

docker compose -f docker-compose.yml build
docker compose -f docker-compose.yml up

We need to note here that some modern browsers disable audio processing for non-localhost domains if the connection is not over HTTPS. Therefore, it is needed to request a certificate for your domain, perhaps from the LetsEncrypt project. Otherwise, the app will not work correctly.

Adding a New Model to TorchServe

Minuteman makes it easy to add new summarization models by using TorchServe. When adding a new summarization model, it is necessary to create a model .mar archive for TorchServe to accept. The most common way of doing this is to download a model from HuggingFace and then creating the archive using torch-model-archiver.

To be able to create the archive, it is needed to install torch-model-archiver. It is available from PyPi. Then, it is necessary to create a handler file similar to the example handler in torchserve/bart_model_serve.py. It is generally only needed to alter the constant with the model name, as this downloads the correct tokenizer on startup.

After that, the selected model can be downloaded from HuggingFace using the transformers library and the torchserve/model_saver.py script. This unfortunately requires PyTorch. To bypass this, it is possible to download the necessary files from selected the HuggingFace repository manually.

To prepare the .mar archive containing the model, we run the following command:

torch-model-archiver
    --model-name [MODEL_NAME]
    --version 1.0
    --serialized-file [pytorch_model.bin]
    --extra-files  "[config.json],[generation_config.json]"
    --handler "[YOUR_HANDLER_FILE].py"

We replace MODEL_NAME with the desired name of the archive and the other variables with the paths of your downloaded model components. After running the command, you will be left with a .mar archive with the model. For usage, we need to move the created archive to torch_model_dir so that it is visible by the TorchServe docker container. In the docker-compose.yml file, we then specify the model and its name in the command to run torchserve like this:

torchserve
    --start
    --model-store torch_model_dir/
    --models [NAME]=[MODEL_NAME].mar ...`