some code problems

[txy77]

When I run bash codex_expmt_folder_runner.sh runs /codex/mw21_1p_train/python, it seems that I get the error.

[kingb12]

Hi @txy77 !

I'm not at a computer now so I'll have to double check soon, but you'll want to make sure you run this from the directory which contains setup.cfg:

pip install -e .

This should install everything in src/refpydst as a module refpydst and fix the import issues.

Hope this helps!
Brendan

[txy77]

Thanks for your help! I’ll have a try! | | 汤昕宇 | | ***@***.*** |

…

---- Replied Message ---- | From | Brendan ***@***.***> | | Date | 07/17/2023 14:55 | | To | ***@***.***> | | Cc | ***@***.***> , ***@***.***> | | Subject | Re: [jlab-nlp/RefPyDST] some code problems (Issue #1) | Hi @txy77 ! I'm not at a computer now so I'll have to double check soon, but you'll want to make sure you run this from the directory which contains setup.cfg: pip install -e . This should install everything in src/refpydst as a module refpydst and fix the import issues. Hope this helps! Brendan — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[txy77]

Hey guy! I'm sorry to bother you again, but I wanted to ask how to handle this error.

[kingb12]

Hi @txy77 ! Sorry for the delayed response. You'll probably need to follow the steps here to download and prepare the data. Specifically:

cd data
python create_data.py --main_dir mwz21 --mwz_ver 2.1 --target_path mwz2.1  # for MultiWOZ 2.1
python create_data.py --main_dir mwz24 --mwz_ver 2.4 --target_path mwz2.4  # for MultiWOZ 2.4
bash preprocess.sh
python sample.py --input_fn mwz2.1/train_dials.json --target_fn mw21_5p_train_seed0.json --ratio 0.05 --seed 0
bash download_mw23.sh
python build_coref_only_dataset.py

If you have already done that, I would make sure you set the environment variable for the data directory, like so:

export REFPYDST_DATA_DIR="/absolute/path/to/data"

Or from within the ./data directory:

export REFPYDST_DATA_DIR="`pwd`"

Hope this helps!

[txy77]

Thank you for your reply. I’ll have a try! | | 汤昕宇 | | ***@***.*** |

…

---- Replied Message ---- | From | Brendan ***@***.***> | | Date | 07/23/2023 13:28 | | To | ***@***.***> | | Cc | ***@***.***> , ***@***.***> | | Subject | Re: [jlab-nlp/RefPyDST] some code problems (Issue #1) | Hi @txy77 ! Sorry for the delayed response. You'll probably need to follow the steps here to download and prepare the data. Specifically: cd data python create_data.py --main_dir mwz21 --mwz_ver 2.1 --target_path mwz2.1 # for MultiWOZ 2.1 python create_data.py --main_dir mwz24 --mwz_ver 2.4 --target_path mwz2.4 # for MultiWOZ 2.4 bash preprocess.sh python sample.py --input_fn mwz2.1/train_dials.json --target_fn mw21_5p_train_seed0.json --ratio 0.05 --seed 0 bash download_mw23.sh python build_coref_only_dataset.py If you have already done that, I would make sure you set the environment variable for the data directory, like so: export REFPYDST_DATA_DIR="/absolute/path/to/data" Or from within the ./data directory: export REFPYDST_DATA_DIR="`pwd`" Hope this helps! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[txy77]

There will still be some issues with this. What is the 'retriever_dir' needed on this side?

[kingb12]

Hi! retriever_dir an argument needed to run the Codex runs, pointing to the directory containing the relevant trained retriever (e.g. the directory contents you'd get from saving a sentence-transformer after fine-tuning, as is done here.

It looks like the argument is defined, but the path does not exist. I had written some helper code that might be getting in the way here. In the run I pre-defined, retriever_dir is given as a relative path, which is relative to REFPYDST_OUTPUTS_DIR. To resolve you do one of these:

• Change the run argument in .json file to the absolute path to the fine-tuned retriever OR
• Set REFPYDST_OUTPUTS_DIR environment variable to point to the base output directory you used, so that when they are joined as done here, you get the same path.

Sorry you ran into this, this file-path parsing pattern was useful in my experiments but I probably should have simplified for the code release

[txy77]

I am very interested in your work on "Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking." I would like to know the following details:

1. When evaluating JGA, did you use the FuzzyWuzzy library, and is a similarity score of 90 considered a threshold for considering a slot value match?
2. In the zero-shot experiments, did you use "use_gold," which means evaluating the predicted dialogue state against the dialogue state from the previous turn?
   

[kingb12]

Hi!

1. Sort of, the answer is slightly complicated but covered in Appendix A and a README in src/refpydst/normalization. In short:

   1. We take a raw predicted value for a slot like hotel-name: acron guest house, which may reflect typos in the dialogue.

   2. We use the original database released w/ MultiWOZ and fuzzy matching to try to find a matching entity name in the DB. This is done using the class defined in ontology.py, and would produce something like hotel-name: acorn guest house

   3. In the real-world, finding the DB entity is all you need to do, but MultiWOZ gold annotations use non-canonical surface forms of their own (not matching the DB entity name). So we take the canonical name and map back to the most frequently observed surface form in whatever labelled data is available for that experimental setting (none for zero-shot), as well as ontology.json (following its use in prior works for fair comparison). This whole process produces a 'normalized parse' from a 'raw parse' for a DST prediction. So if the annotations that we have available for training generally use hotel-name: the acorn guest house, we would return that instead of its true DB name.

   4. The final evaluation uses exact value match on the normalized parse as in prior work
      We chose to do this as many prior works use hand-crafted typo fixes and value replacements that imply knowledge of common typos in the dialogue data, and we wanted something more robust that was precise about the system/label knowledge it used. In Table 4 we also apply our normalization approach to the strongest baseline for fair comparison

2. No, we evaluated under the same approach as the few-shot experiments, such that values in Table 1 and Table 3 are directly comparable, except for stated changes in % training data and model.