Metal rust shared structs

[yaroslavyaroslav]

This is POC of sharing data structs across Rust and Metal codebase.

Such sharing allows 1on1 mapping of data from CPU side to Metal side through both of them have exact same layout and alignment (bc of they're the very same structs after all).

Moving all but final msm shaders to C++ implementation allows:

1. Perform further shader development with IDE like experience leveraging tools like clangd.
2. Use better test framework and even to include their tests in rust cargo test pipeline.
3. Better organize GPU side code, to make it less error prone and more reusable (thanks to separate compilation unit conception used by C++).

PS: this PR isn't supposed to be ever merged.

[moven0831]

what does the common.rs looks like? with this, can we read, encode, or parse these data in between CPU/GPU more smoothly?

fyi, in current code, we do

read data from GPU as Vec<u32>:

gpu-acceleration/mopro-msm/src/msm/metal_msm/metal_msm.rs

Lines 505 to 507 in d0320d5

	let bucket_x = helper.read_results(&bucket_x_buf, bucket_size);
	let bucket_y = helper.read_results(&bucket_y_buf, bucket_size);
	let bucket_z = helper.read_results(&bucket_z_buf, bucket_size);

encoding data in CPU:

gpu-acceleration/mopro-msm/src/msm/metal_msm/metal_msm.rs

Lines 448 to 455 in d0320d5

	let row_ptr_buf = helper.create_buffer(&csc_col_ptr.to_vec());
	let val_idx_buf = helper.create_buffer(&csc_val_idxs.to_vec());
	let point_x_buf = helper.create_buffer(&point_x.to_vec());
	let point_y_buf = helper.create_buffer(&point_y.to_vec());
	let bucket_x_buf = helper.create_empty_buffer(bucket_size);
	let bucket_y_buf = helper.create_empty_buffer(bucket_size);
	let bucket_z_buf = helper.create_empty_buffer(bucket_size);

parse Vec to bigint or ec types:

gpu-acceleration/mopro-msm/src/msm/metal_msm/metal_msm.rs

Lines 224 to 241 in d0320d5

	let xr_limbs = &g_points_x[limb_start_idx..limb_end_idx];
	let yr_limbs = &g_points_y[limb_start_idx..limb_end_idx];
	let zr_limbs = &g_points_z[limb_start_idx..limb_end_idx];
	let xr_bigint = BigInt::<4>::from_limbs(xr_limbs, self.config.log_limb_size);
	let yr_bigint = BigInt::<4>::from_limbs(yr_limbs, self.config.log_limb_size);
	let zr_bigint = BigInt::<4>::from_limbs(zr_limbs, self.config.log_limb_size);
	// decoded points from Montgomery form to standard form
	let xr_reduced = raw_reduction(xr_bigint);
	let yr_reduced = raw_reduction(yr_bigint);
	let zr_reduced = raw_reduction(zr_bigint);
	let x = BaseField::from_bigint(xr_reduced).unwrap_or_else(|| BaseField::zero());
	let y = BaseField::from_bigint(yr_reduced).unwrap_or_else(|| BaseField::zero());
	let z = BaseField::from_bigint(zr_reduced).unwrap_or_else(|| BaseField::zero());
	let point = G::new(x, y, z);

[yaroslavyaroslav]

The goal of this PR was to test the boundaries of c++<-> metal interoperability.

In short the conclusion if the following: sharing header files is the best that we can get so far, no implementation can be shared between gpu and cpu as for the time being by the compiler.

I pretty much sure that this goal can be achieved by a third party tools though, but this topic is out of the box of this research.

Why goal matters:

• metal still has lack of pure code completion support. So the reasonable strategy could be is to implement the very least in pure metal (say exact kernel logic) and to put all the underlying structs and methods on C++ layer to gain such completion.
• DRY compilance. Now library has to have 2 independent implementations of BigInt structs, which violates DRY rule.