Flow Metadata Collector

This repository enables you to collect metadata datasets of mixnet flows at the Sphinx packet level in the way we did for our paper "Shift Your Shape: Correlating and Defending Mixnet Flows Based on Their Shapes". Please see our primary repository, shift-your-shape_correlating-and-defending-mixnet-flows-based-on-their-shapes, for more information.

Usage

Follow the Bash scripts and Jupyter Notebooks provided in this repository in ascending order. If you do so, you end up with two new folders per experimental setting for which you collect data: the raw dataset and its ready version. The latter one is suitable for use with the classifiers and instrumentation scripts we provide in shift-your-shape_correlating-and-defending-mixnet-flows-based-on-their-shapes.

Before attempting to adjust this repository to your setting, please read our paper first in order to understand the terminology we use and the setup we create.

This flow metadata collector assumes to create the data collection infrastructure on public cloud provider Hetzner. The scripts will call out to Hetzner's CLI tool hcloud for various instance provisioning steps. If you intend to use a different cloud provider, please make sure to replace these Hetzner-specific parts with equivalents for your cloud vendor. However, the step of instance provisioning is the extent to which these scripts are cloud-provider-specific. All other steps on the way to obtaining usable datasets for Nym in version nym-binaries-v2023.5-rolo are regular Bash or Python commands.

We assume that you run:

• Scripts *_local_* on your local machine (e.g., work laptop),
• Scripts *_on-gwreq_* manually on each of the provisioned gwreq-* instances,
• Scripts *_on-endpoint_* manually on each of the provisioned endpt-* instances,
• Scripts and Jupyter Notebooks *_on-jupyter_* manually on the jupyter instance provisioned with script A_local_spawn-jupyter-cloud-server.sh.

Thus, in its current configuration and unless changed by you, this data collector assumes 1 local machine, 3 gwreq-*, 12 endpt-* instances, and 1 jupyter instance.

This repository serves simultaneously as an archive for relevant runs of these scripts for our research project as well as a tool for others to collect datasets in a similar manner. Thus, before use, you have to adjust at least the following parts to your own setting:

• Change the number of gwreq-* and endpt-* instances in 01_local_create-3-gwreq-12-endpoint.sh if you intend to deviate from the 3 gwreq-* and 12 endpt-* instances that this repository creates by default,
• Adjust the various variables at the top of each Bash script and Jupyter Notebook to match the paths in your setup and targeted experiment (typically signified as MODIFY BEGIN and MODIFY END comment blocks),
• Adjust the various Nym owner details (e.g., mixcorr_private_gateway_owner in 02_local_create-gwreq.sh) such that the created gwreq-* instances are properly accepted by your endpt-* nodes,
• Tailor the large number of checks conducted by script 09_on-jupyter_check-raw-dataset.sh on each raw dataset to the specifics of that respective dataset.

For inspiration and reference, we publish the relevant scripts of this repository and their accompanying logs for the three datasets we collected on the live Nym mixnet for our research project. Read through them if you are unsure of how a specific piece of these scripts and Jupyter Notebooks works:

• Raw default: shift-your-shape_exp01-default-nym-binaries-v2023.5-rolo_2024-04-09,
• Ready default: shift-your-shape_ready_exp01-default-nym-binaries-v2023.5-rolo_2024-04-09,
• Raw high_delay: shift-your-shape_exp02-higher-mix-delay-nym-binaries-v2023.5-rolo_2024-04-15,
• Ready high_delay: shift-your-shape_ready_exp02-higher-mix-delay-nym-binaries-v2023.5-rolo_2024-04-15,
• Raw large_pkts: shift-your-shape_exp03-larger-packet-size-nym-binaries-v2023.5-rolo_2024-04-29,
• Ready large_pkts: shift-your-shape_ready_exp03-larger-packet-size-nym-binaries-v2023.5-rolo_2024-04-29.

Licensing

This repository is licensed under GPLv3.

However, the following individual files are licensed under Apache-2.0:

• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/bug-track-ack-insert-crash_acknowledgement_control_action_controller.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_init_helpers.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_topology_control_mod.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_topology_gateway.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_acknowledgement_listener.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_input_message_listener.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_retransmission_request_listener.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_chunking_fragment.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_client_received_buffer.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_real_messages_control_message_handler.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_real_messages_control_real_traffic_stream.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_reply_controller_mod.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_init_helpers.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_topology_control_mod.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_topology_gateway.rs.patch
• ./exp01-default-nym-binaries-v2023.5-rolo/socks5-client/track-sphinxflows_real_messages_control_message_handler.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/bug-track-ack-insert-crash_acknowledgement_control_action_controller.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_init_helpers.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_topology_control_mod.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_topology_gateway.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_acknowledgement_listener.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_input_message_listener.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_retransmission_request_listener.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_chunking_fragment.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_client_received_buffer.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_real_messages_control_message_handler.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_real_messages_control_real_traffic_stream.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_reply_controller_mod.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_init_helpers.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_topology_control_mod.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_topology_gateway.rs.patch
• ./exp02-higher-mix-delay-nym-binaries-v2023.5-rolo/socks5-client/track-sphinxflows_real_messages_control_message_handler.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/bug-track-ack-insert-crash_acknowledgement_control_action_controller.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/larger-packet-size_client_core_config_mod.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/larger-packet-size_params_packet_size.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_init_helpers.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_topology_control_mod.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/private-gateway_topology_gateway.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_acknowledgement_listener.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_input_message_listener.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_acknowledgement_control_retransmission_request_listener.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_chunking_fragment.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_client_received_buffer.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_real_messages_control_message_handler.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_real_messages_control_real_traffic_stream.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/gateway_requester/track-sphinxflows_reply_controller_mod.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/socks5-client/larger-packet-size_client_core_config_mod.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/socks5-client/larger-packet-size_params_packet_size.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_init_helpers.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_topology_control_mod.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/socks5-client/private-gateway_topology_gateway.rs.patch
• ./exp03-larger-packet-size-nym-binaries-v2023.5-rolo/socks5-client/track-sphinxflows_real_messages_control_message_handler.rs.patch