Update MessagePack compatibility and tests by YukiMatsuzawa · Pull Request #747 · systemsomicslab/MsdialWorkbench

YukiMatsuzawa · 2026-07-09T05:12:11Z

Preserve large-list file compatibility while moving the MessagePack stack to the newer API surface.

This updates the core serializer path, related data objects, and regression tests so existing persisted data and MessagePack round-trips keep working after the upgrade.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…improve/update-messagepack

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ed-telegram

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…pdate-messagepack

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ve/update-messagepack

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR upgrades the repository’s MessagePack stack (notably to MessagePack v3.x) while aiming to preserve compatibility for existing persisted data, especially for large-list serialization, and adds regression tests that validate byte-level round-trips.

Changes:

Upgrade MessagePack packages / frameworks and migrate serialization calls to the newer MessagePackSerializer + options API surface.
Refactor restoration-key and annotator-container serialization to work with the updated MessagePack union/formatter approach.
Add/extend regression tests that assert deserialization of known legacy bytes and stable re-serialization outputs.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/MSDIAL5/MsdialCoreTests/Parser/MspDbRestorationKeyTests.cs	Updates restoration-key serialization test to use the new non-generic interface + stream rewind.
tests/MSDIAL5/MsdialCoreTests/DataObj/ProjectDataStorageTests.cs	Makes test timestamps explicit with timezone offset for stable serialization behavior.
tests/MSDIAL5/MsdialCoreTests/DataObj/MoleculeDataBaseTests.cs	Adds missing stream rewind before deserialization in MessagePack round-trip test.
tests/Common/CommonStandardTests/MessagePack/LargeListMessagePackTests.cs	Adds byte-level tests for `LargeListMessagePack` (round-trip + legacy byte fixtures).
tests/Common/CommonStandardTests/Lipidomics/LipidEieioMsmsCharacterizationTests.cs	Adjusts deployed resource path for lipidomics test input file.
tests/Common/CommonStandardTests/DataObj/Property/AdductIonTests.cs	Adds byte-fixture tests for `AdductIon` serialization compatibility and stream rewinds.
tests/Common/CommonStandardTests/Components/MoleculePropertyTests.cs	New byte-fixture test validating `IMoleculeProperty` formatter compatibility.
tests/Common/CommonStandardTests/Components/DriftTimeTests.cs	Adds stream rewind before deserialization.
tests/Common/CommonStandardTests/CommonStandardTests.csproj	Ensures test resource is copied to output under the updated target path.
src/MSFINDER/MsfinderConsoleAppCore/MsfinderConsoleAppCore.csproj	Updates .NET Core target from netcoreapp3.0 to netcoreapp3.1.
src/MSDIAL5/SpectrumViewer/SpectrumViewer.csproj	Updates `System.Runtime.CompilerServices.Unsafe` package version.
src/MSDIAL5/MsdialGuiApp/MsdialGuiApp.csproj	Upgrades MessagePack package reference to v3.1.7.
src/MSDIAL5/MsdialCore/Parser/ReferRestorationKey.cs	Adds union metadata + generic bridge methods to support updated restoration-key serialization.
src/MSDIAL5/MsdialCore/Parser/IReferRestorationKey.cs	Introduces a non-generic `IReferRestorationKey` interface for serialization friendliness.
src/MSDIAL5/MsdialCore/DataObj/ShotgunProteomicsDB.cs	Adjusts MessagePack member attributes (e.g., `disposedValue`) for the new serializer behavior.
src/MSDIAL5/MsdialCore/DataObj/ProteomicsAnnotatorParameterPair.cs	Switches stored restoration-key type to non-generic interface + explicit string keys.
src/MSDIAL5/MsdialCore/DataObj/MsdialDataStorage.cs	Removes `[MessagePackObject]` attribute from a base class as part of serializer refactor.
src/MSDIAL5/MsdialCore/DataObj/MetabolomicsAnnotatorParameterPair.cs	Switches stored restoration-key type to non-generic interface + explicit string keys.
src/MSDIAL5/MsdialCore/DataObj/IAnnotatorParameterPair.cs	Adds a non-generic marker interface to support union-based serialization of annotator pairs.
src/MSDIAL5/MsdialCore/DataObj/EadLipidAnnotatorParameterPair.cs	Adds serialization constructor accepting the new non-generic restoration-key interface.
src/MSDIAL5/MsdialCore/DataObj/DataBaseItem.cs	Adds a serialization-friendly `PairsForSerialization` representation using the new non-generic pair interface.
src/MSDIAL5/MsdialCore/DataObj/ChromatogramPeakFeature.cs	Adjusts MessagePack member inclusion (notably match-results and interface members).
src/MSDIAL5/MsdialCore/DataObj/AnnotatedMSDecResult.cs	Migrates custom formatter to new MessagePack reader/writer APIs with explicit array layout checks.
src/MSDIAL5/MsdialCore/Algorithm/Annotation/IAnnotationQueryFactory.cs	Removes an unused `Accord.Collections` using directive.
src/MSDIAL5/MsdialCore/Algorithm/Annotation/AnnotatorContainer.cs	Updates annotator container key types + constructors for new restoration-key serialization approach.
src/MSDIAL4/StructureFinderStandard/StructureFinderStandard.csproj	Upgrades MessagePack package reference to v3.1.7.
src/MSDIAL4/MathematicsStandard/MathematicsStandard.csproj	Upgrades MessagePackAnalyzer to v3.1.7.
src/MSDIAL4/DatabaseStandard/DatabaseStandard.csproj	Upgrades MessagePackAnalyzer to v3.1.7.
src/Common/CommonStandard/Paramerter/AnalysisParamOfMsdialGcms.cs	Removes redundant/high-index `[Key]` attributes on backing fields (favoring keyed properties).
src/Common/CommonStandard/MessagePack/MessagePackHandler.cs	Migrates handler to `MessagePackSerializer` with compression + standard resolver.
src/Common/CommonStandard/MessagePack/LargeListMessagePack.cs	Reimplements large-list chunking/formatting against the v3 reader/writer API and adds byte-fixture support.
src/Common/CommonStandard/DataObj/Property/AdductIon.cs	Migrates custom `AdductIon` formatter to v3 reader/writer APIs and updates serialization layout.
src/Common/CommonStandard/Components/Interfaces/IMoleculeProperty.cs	Migrates `IMoleculeProperty` formatter to v3 reader/writer APIs with stricter array-length checks.
src/Common/CommonStandard/Components/Interfaces/IChromatogramPeakFeature.cs	Replaces union-based interface serialization with a dedicated interface formatter.
src/Common/CommonStandard/CommonStandard.csproj	Upgrades MessagePack + analyzer packages to v3.1.7.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        public static void Serialize<T>(Stream stream, IReadOnlyList<T> value) {
+            if (value is null) {
+                MessagePackSerializer.Serialize(stream, value, MessagePackSerializerOptions.Standard.WithResolver(StandardResolver.Instance).WithCompression(MessagePackCompression.Lz4BlockArray));
+                return;
+            }

-        private static byte[] GetBufferLZ4()
-        {
-            return new byte[65536];
-            //if (bufferLz is null)
-            //{
-            //    bufferLz = new byte[65536];
-            //}
-            //return bufferLz;
+            var options = MessagePackSerializerOptions.Standard.WithResolver(StandardResolver.Instance).WithCompression(MessagePackCompression.Lz4BlockArray);
+            Memory<T> memory = (value as T[]) ?? value.ToArray();
+            var length = memory.Length;
+            var size = (int)Math.Sqrt(length) + 1;
+            var iteration = (length + size - 1) / size;
+            for (int i = 0; i < iteration; i++) {
+                MessagePackSerializer.Serialize(stream, new SerializingDataContainer<T> { Data = memory.Slice(i * size, Math.Min(size, length - i * size)) }, options);
+            }


Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… HEAD

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… improve/update-messagepack

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…gine' into improve/update-messagepack

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 7 comments.

+            var options = MessagePackSerializerOptions.Standard.WithResolver(StandardResolver.Instance).WithCompression(MessagePackCompression.Lz4BlockArray);
+            Memory<T> memory = (value as T[]) ?? value.ToArray();
+            var length = memory.Length;
+            var size = (int)Math.Sqrt(length) + 1;
+            var iteration = (length + size - 1) / size;
+            for (int i = 0; i < iteration; i++) {
+                MessagePackSerializer.Serialize(stream, new SerializingDataContainer<T> { Data = memory.Slice(i * size, Math.Min(size, length - i * size)) }, options);
+            }


Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 4 comments.

+            var options = MessagePackSerializerOptions.Standard.WithResolver(StandardResolver.Instance).WithCompression(MessagePackCompression.Lz4BlockArray);
+            Memory<T> memory = (value as T[]) ?? value.ToArray();
+            var length = memory.Length;
+            var size = (int)Math.Sqrt(length) + 1;
+            var iteration = (length + size - 1) / size;
+            for (int i = 0; i < iteration; i++) {
+                MessagePackSerializer.Serialize(stream, new SerializingDataContainer<T> { Data = memory.Slice(i * size, Math.Min(size, length - i * size)) }, options);
+            }


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

YukiMatsuzawa and others added 22 commits July 6, 2026 15:15

fix test

d92f04c

Add AdductIon fixed-byte round-trip tests

53f4a16

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add MoleculeProperty messagepack compatibility test

ebd5d0a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add LargeListMessagePack compatibility test

f51339a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge branch 'yukimatsuzawa-messagepack-moleculeproperty-tests' into …

720a70c

…improve/update-messagepack

Merge branch 'master' into improve/update-messagepack

69b79e6

Update MessagePack package versions

8710c09

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Adapt MessagePack serializers to v3

d9fcc6e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Fix AnnotatedMSDecResult MessagePack formatter

a612576

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Checkpoint current MessagePack migration

b4e53dc

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Commit staged AdductIon MessagePack fix

b74804d

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Commit CommonStandard MessagePack updates

4b30e02

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Commit CommonStandard test updates

25d6890

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Checkpoint ongoing MessagePack fixes

9575673

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add deserialization test for LargeListMessagePack

98ad6a4

Merge branch 'improve/update-messagepack' into yukimatsuzawa-refactor…

391dffd

…ed-telegram

Add LargeListMessagePack compatibility tests

8e249e9

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge branch 'yukimatsuzawa-messagepack-upgrade-tests' into improve/u…

a4f99d4

…pdate-messagepack

Restore LargeListMessagePack compatibility

9e61c2c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge improve/update-messagepack

6edd908

Merge branch 'yukimatsuzawa-messagepack-compatibility-fix' into impro…

b3179ef

…ve/update-messagepack

Remove duplicate LargeListMessagePack tests

85693d3

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

YukiMatsuzawa requested a review from Copilot July 9, 2026 07:35

Copilot started reviewing on behalf of YukiMatsuzawa July 9, 2026 07:35 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

YukiMatsuzawa and others added 5 commits July 9, 2026 20:33

Add MessagePack compatibility fixtures

48025da

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge branch 'copilot/yukimatsuzawa-messagepack-old-state-tests' into…

65bf03c

… HEAD

Add legacy MessagePack load coverage

b07f428

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add MessagePack legacy byte fixtures

f20b893

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add MessagePack compatibility fixtures

f8ef053

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

YukiMatsuzawa and others added 5 commits July 10, 2026 15:12

Merge branch 'copilot/yukimatsuzawa-messagepack-old-state-tests' into…

0c05925

… improve/update-messagepack

Fix test

95120df

Add MessagePack compatibility fixtures for legacy bytes

1b8c248

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge branch 'copilot/yukimatsuzawa-copilot-yukimatsuzawa-animated-en…

4f9b342

…gine' into improve/update-messagepack

Expand MessagePack compatibility test coverage

6f129d9

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

YukiMatsuzawa requested a review from Copilot July 13, 2026 06:05

Copilot started reviewing on behalf of YukiMatsuzawa July 13, 2026 06:06 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

YukiMatsuzawa added 4 commits July 13, 2026 15:38

Delete the commented-out legacy declaration

1d6718b

Update LargeListMessagePack

5cb0293

Fix IMoleculeProperty serializer

57927d1

Fix AdductIon deserializer

7e04793

YukiMatsuzawa requested a review from Copilot July 13, 2026 09:54

Copilot started reviewing on behalf of YukiMatsuzawa July 13, 2026 09:55 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

YukiMatsuzawa and others added 3 commits July 13, 2026 19:06

Potential fix for pull request finding

3fc52f7

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

e17708c

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

05f2205

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update MessagePack compatibility and tests#747

Update MessagePack compatibility and tests#747
YukiMatsuzawa wants to merge 39 commits into
masterfrom
improve/update-messagepack

YukiMatsuzawa commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

YukiMatsuzawa commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants