Explicit type specification for unions

[altendky]

Sometimes it would be good to store a type identifier in union fields. This would be especially true for sparsely populated data and unions containing str (described below).

Consider:

import typing

import attr
import desert


@attr.dataclass
class A:
    color: str


@attr.dataclass
class B:
    color: str


@attr.dataclass
class C:
    things: typing.List[typing.Union[A, B]]

We then create some instances:

instances = C(
    things=[
        A(color='red'),
        A(color='green'),
        B(color='blue'),
    ],
)

Theoretically that would serialize to (it doesn't seem to so there will presumably be another issue report):

{"things": [{"color": "red"}, {"color": "green"}, {"color": "blue"}]}

Which leaves us wondering what any of the things are.

On the flip side, if we take an example in the tests where there is a field t.Union[int, str] then adjust it to include a custom class after str such as t.Union[int, str, MyClass] then str will 'always' work and MyClass instances will be serialized to a str such as 'MyClass()' (well, for an attrs class with such a repr() anyways).

tl;dr, marshmallow-union has some pretty significant limitations. I got myself confused and started creating a solution over in marshmallow-polyfield Bachmann1234/marshmallow-polyfield#34 which would introduce an extra layer to the serialization and include a string to designate the type of that element. No more guessing.

{
    "things": [
        {
            "type": "A",
            "value": {
                "color": "red",
            }
        },
        {
            "type": "A",
            "value": {
                "color": "green",
            }
        },
        {
            "type": "B",
            "value": {
                "color": "blue",
            }
        }
    ]
}

Perhaps desert is interested in using marshmallow-polyfield for that feature? Perhaps desert would rather the feature be added to marshmallow-union?

In the mean time I'll try to find time to work my way through reporting and possibly fixing some other issues.

Polyfield looks interesting. I think it'd be worthwhile to make sure we're not duplicating ideas that may have already been well worked out in, say, marshmallow-jsonschema.

[altendky]

To get it on the record and out of my tab list...

OAI/OpenAPI-Specification#57 (comment)

OpenAPI 3.0 uses an extended subset of JSON Schema Specification Wright Draft 00 (aka Draft 5) to describe the data formats. “Extended subset” means that some keywords are supported and some are not, some keywords have slightly different usage than in JSON Schema, and additional keywords are introduced.

https://swagger.io/docs/specification/data-models/keywords/

[altendky]

As I think about looking at my own files I do really like the lack of an extra layer and the explicit type specification of every item. So I'm thinking that perhaps both options would be good. Of course, two ways has plenty of cost so...

I suspect the wrapper-layer approach is only going to be of significant interest to people that do not want types indicated in the serialized data but that also want functional unions. The wrapper layer is the least intrusive to the serialized form. This is a property of a field. Just union fields? Maybe, I haven't thought about any others. The identifer/type relationship is local.

The integrated form is nice when you do it everywhere. It is only one extra line per object and no indentation. This is more of a property of the class to be serialized. The identifier/type relationship is more 'global'. It doesn't have to literally be global but it isn't part of a field or a class.

I think the two may be able to cleanly coexist not only as features in desert but even in single serialized data blobs. I suppose there could be some silliness if you used different names for the two methods.

{
    "type": "MyClass",
    "value": {
        "_type": "color_stuff",
        "red": 0,
        "green": 1,
        "blue": 0.5
    }
}

So... meh. Maybe if all classes to be serialized in the union are internally typed then the union automatically drops the external typing layer? Too implicit? Maybe there are hazards in some scenarios such as serializing with one set of classes then adding a new option that isn't internally typed and the layers are expected but not present? The developer must choose but we give warnings or otherwise provide a means of testing that if their choice isn't obviously sensible?

There's also the matter of subclasses that I put in the WIP list over in polyfield but haven't thought through yet. IIRC, the types registered with the union are expecting to dict-lookup match the types of the actual objects.

No conclusions here I suppose, just some thoughts to revisit later.

Where's the indication that 1 is a float? Do we only want that if the defining schema has a union field for "green" (which it presumably doesn't here)?

[altendky]

Correct, no union for green. It was meant to just be a float.

The indication is in the schema (unless marshmallow does 1.0, I don't know). For people that want the integrated type form for native JSON (or more general) types, we could provide fields that have integrated types and a way to trigger them. Or I suppose the same for wrapped-layer.

For reference, I use decimal.Decimals as strings for all my 'floats'.

If we are assuming the loader's schema is the same as the dumpers, and key ordering is canonicalized or maintained, there could be a List[int] as a top-level value where the ith element of the list gives the index of the Union[...] entry of the ith union-typed value in the data.

For example,

{
   "data": {"things": [{"color": "red"}, {"color": "green"}, {"color": "blue"}]},
   "union_indexes": [0,0,1],
   "schema_hash": "01abef8" # or at least "schema_name"
}

[obi1kenobi]

Hi all, I'm new to this project 👋Very interested in seeing this issue (and #89) get resolved. I've tried both dataclass-json and marshmallow-dataclass and gotten pretty disappointed, so I'm hoping desert can be the production-grade dataclass serialization library I need for the projects I maintain. Thank you for all the hard work you all have put into this library 🙏

I happened to come across a good writeup of the various options for explicit type specification for unions, courtesy of the Serde library that handles serialization in the Rust programming language: https://serde.rs/enum-representations.html

I think supporting a choice of explicit representation (and if possible, compatibility with Serde for cross-language applications) would really set this library apart from the competition, and if that's the direction chosen by the maintainers, I'd love to be part of that journey.

Hey @obi1kenobi, welcome!

That Serde link gives a great rundown of our options. @altendky's polyfield link above looks like what Serde calls "adjacently tagged", and marshmallow-union looks like the "untagged" option.

I'm +1 on providing built-in support for all four options discussed there.

[altendky]

So let's see if my original example can still be used to show the four options here.

Class Definitions

import typing

import attr
import desert


@attr.dataclass
class A:
    color: str


@attr.dataclass
class B:
    color: str


@attr.dataclass
class C:
    things: typing.List[typing.Union[A, B]]

Sample Data

instances = C(
    things=[
        A(color='red'),
        A(color='green'),
        B(color='blue'),
    ],
)

I'll use some 'non-standard' formatting and hopefully it'll end up more readable including considerations of fitting on the screen. I am also only applying it to the members of the attribute list. I think that may be an orthogonal question. 1) How to tag when being explicit and 2) whether to tag even when there isn't any ambiguity.

Externally Tagged

{
    "things": [
        {"A": {"color": "red"}},
        {"A": {"color": "green"}},
        {"B": {"color": "blue"}}
    ]
}

Internally Tagged

{
    "things": [
        {"type": "A", "color": "red"},
        {"type": "A", "color": "green"},
        {"type": "B", "color": "blue"}
    ]
}

Adjacently Tagged

{
    "things": [
        {"t": "A", "c": {"color": "red"}},
        {"t": "A", "c": {"color": "green"}},
        {"t": "B", "c": {"color": "blue"}}
    ]
}

Untagged

{
    "things": [
        {"color": "red"},
        {"color": "green"},
        {"color": "blue"}
    ]
}

If something looks wrong I think it will be worth updating in place here so just let me know and I'll tweak it.

Are the specific strings "type", "t", and "c" significant? For the internally tagged it seems maybe useful to default to something that can't be an attribute (normally anyways) like "@type" or whatever. For adjacent tagging I would have (well did :]) use full words.

As noted in the provided link (thanks), internal tagging doesn't apply to things serialized as JSON arrays rather than objects. If we wanted we could make the first element be the tag I suppose.

Side note, should we talk more about marshalling to Python dict/list/int/float/str/bool/None rather than serializing to JSON object/array/number/true/false/null? If we go big (instead of going home) and try to allow for multiple backends etc this seems maybe more relevant? I dunno, I don't have this big architecture plan in my head yet. Going big might also suggest it would be useful to implement the disambiguation in desert itself rather than delegating to a new implementation specific to each backend. Maybe...

@altendky

Are the specific strings "type", "t", and "c" significant?

I'm guessing they come from the #[serde(tag = "t", content = "c")] declaration just above the enum.

@altendky

Side note, should we talk more about marshalling to Python dict/list/int/float/str/bool/None rather than serializing to JSON object/array/number/true/false/null?

I think that's what we're doing already, since schema.dump(obj) returns a dict (or other appropriate type). Or do you mean something else?