Please include UUID property in set tags for tokens

[FluxNeven]

Recently I discovered that the Cockatrice Client could not distinguish between different printings of tokens via its Printing Selector.
It was quickly discovered that this was because the tokens.xml does not have a 'uuid' property in the 'set' tag in which the image link for a given set printing is kept.

for example, from the token.xml file, we have these two lines in the 'Feather' token card block:

<set picURL="https://cards.scryfall.io/large/front/e/4/e401e2a8-d0a3-4517-ba20-449a1fff7f85.jpg?1682211019" >MOC</set>
<set picURL="https://cards.scryfall.io/large/front/a/9/a9cc7c63-5d13-4fd6-af9d-4a26c2bab8e6.jpg?1591225596" >IKO</set>

If we were to alter those card blocks as follows:

<set uuid="e401e2a8-d0a3-4517-ba20-449a1fff7f85" picURL="https://cards.scryfall.io/large/front/e/4/e401e2a8-d0a3-4517-ba20-449a1fff7f85.jpg?1682211019" >MOC</set>
<set uuid="a9cc7c63-5d13-4fd6-af9d-4a26c2bab8e6" picURL="https://cards.scryfall.io/large/front/a/9/a9cc7c63-5d13-4fd6-af9d-4a26c2bab8e6.jpg?1591225596" >IKO</set>

Then Cockatrice would be able to 'consume' that uuid data directly and be able to display per-printing token card images.

Since mtgjson does provide scryfall token uuids, it should be fairly straightforward to include this property, where it is available, in the tokenx.xml.

Obviously this only works for tokens that have printings, of course. For tokens that don't otherwise exist, it looks as though you have a special image storage that the file links to for those tokens, and thus a scryfall uuid makes no sense to include.

Fortunately, Cockatrice Client's Printing Selector already properly loads the picURL for those tokens which require a non-Scryfall image from the image storage blob without needing a uuid, since they have only one 'printing'.

As such, please update the process by which you generate the tokens.xml to place that uuid=(scryfallID) property in the 'set' tags when it exists, so that it can be used.

[tooomm]

Related #286
Linking Cockatrice/Cockatrice#5961

please update the process by which you generate the tokens.xml to place that uuid=(scryfallID) property in the 'set' tags when it exists

FYI, the token file is maintained manually in this repo

[FluxNeven]

Interesting!

I feel like there might be a useful way to run a bit of code on the token.xml file that checks if the scryfall uuid exists in the picurl, then chops it out and puts it into a 'uuid' property in the set tag without needing to manually change every existing entry one by one. Doing this process manually would certainly be a bit tedious, though it could be done.

If that property is there in the official file, it would be incredibly useful for the Cockatrice folks to have that piece of data in a directly consumable format.

Since the file is maintained manually as it is, I could certainly see being willing to do the alteration to the existing file myself if there was some way to know that future manual additions to the file would continue to have the extra property there when available.

If that is something that would be viable, I would be happy to make the effort to get it into that maintainable state. If you'd prefer to keep the adjustments 'in house', though, I do understand.

[tooomm]

Adding uuid could of course be scripted to add it to the existing entries.

I'm not aware of the implications right now though. @ebbit1q should know better.
Also @SlightlyCircuitous is currently the main contributor to keep the token file semi-manually up to date.

In general, I think we would be open if somebody comes up with e.g. a solid Python script that automates the updating/maintenance of the token file and we can run it in CI (#232).
That would be pretty cool. But note there are also token that are not available in MTGJSON for example as I understand it.

If that property is there in the official file, it would be incredibly useful for the Cockatrice folks to have that piece of data in a directly consumable format.

Btw, we are the "Cockatrice folks". 😉
As you can see by the url structure, both repositories belong to the same organization.
Also see this note in the Cockatrice README.

[FluxNeven]

Well, the main Cockatrice repo bounced me over here when I raised the issue about token art, so I was under the impression these were different 'folks'. My mistake. I much appreciate the patience and information. :)

At the moment, as far as I can tell, the effect would be that the alternate printings of tokens that have alternate printings would appear properly in the Printing Selector if that uuid property was present.

The immediate benefit to this is that players could manually select a specific token printing to tuck into their sideboard so that they could clone that specific art token rather than use the default for the card, if desired.

Thank you for adding the link back here to the main issue track (#232).

This project as a whole is very interesting to me. Given the mention of the existing automated script that updates the 'spoiler.xml', I might see if I can look at that code for inspiration on a clean way to do the tokens. Seems like there would indeed be a lot of similarities. (I'm only an amateur coder, myself, so I'm sort of learning as I go, here :3)

[tooomm]

No issue at all. We try to separate issues and code a bit for different topics.
It could maybe be even more clarly stated in the related section of the main app repository.

At the moment, as far as I can tell, the effect would be that the alternate printings of tokens that have alternate printings would appear properly in the Printing Selector if that uuid property was present.

You could try to update the file for some tokens with your suggestion and see if it has the expected effect out of the box.

The immediate benefit to this is that players could manually select a specific token printing to tuck into their sideboard so that they could clone that specific art token rather than use the default for the card, if desired.

I think it was at one point even discussed to remove the ability to add tokens to a deck.
That was needed years ago to be able to quickly create relevant token out of all available ones, but is obsolete since a long time with tokens showing in the right click menu of their creators (with the help of maintained relations in the tokens.xml file).

This project as a whole is very interesting to me. Given the mention of the existing automated script that updates the 'spoiler.xml', I might see if I can look at that code for inspiration on a clean way to do the tokens. Seems like there would indeed be a lot of similarities. (I'm only an amateur coder, myself, so I'm sort of learning as I go, here :3)

Happy to hear the project sparked your interest! 👍
Exactly, for spoilers we use a script in the https://github.com/Cockatrice/Magic-Spoiler repo and run it automated on a schedule in GitHub Actions CI to update the data that the Cockatrice client checks and pulls on app start (if enabled in settings).

The script however is quite old and would need a rewrite itself at one point. So don't be too inspirited by it. Haha

[FluxNeven]

You could try to update the file for some tokens with your suggestion and see if it has the expected effect out of the box..

I've done exactly that, and that effect I described was the effect it had. Putting the UUID property in the set tags causes the printing selector images to properly popular, and allows the token printings to be added to a deck as cards (I usually add them in the sideboard) so that they can be actively used in-game.

The test I made, initially, was with the Feather token (which has two Printings). It worked as described. I ran another test with the Food token (which has a lot more printings), and it also worked like a charm.

Sadly, this doesn't do anything for the 'automatic create token' action within a game instance. That is likely to need some TLC elsewhere in the code.
Perhaps one way to implement it without needing to code up a whole new selector and slow the auto-gen process could be to design it so that if there is a matching token to the one created loaded in the sideboard of a deck, then that deck creates that printing instead of the default it would try to. For example, if I have a LotR card making a Food token, but I want to use the lovely Bloomburrow Food Token art, I could just stick a Bloomburrow Food Token printing in my sideboard, and the 'Create Food Token' menu option on the card would see the Bloomburrow printing of 'Food Token' in my sideboard and make that instead of the default Foundations one (or the default LotR one, if they also decide to make cards default to producing their own set's token printing, which would be nice behavior to see as well).

At this moment, if the UUIDs are added to the set tag properties, the 'adding tokens to deck' functionality will once again be useful, briefly, as part of that work-around. At least until such time as the actual Cockatrice Client gets the ability to select the created token printing directly during play by some means - a capacity which will definitely need the scryfall uuid to be there regardless, as far as I can tell, so win-win. :)

The script however is quite old and would need a rewrite itself at one point. So don't be too inspirited by it. Haha

Well, it might at least point me the direction of a useful learning experience

This whole journey basically came about because I was trying to find a way to load custom card and token art into the game without overwriting every single printing of a card with the 'Custom' folder art method, which led me to Explorations In XML. It's been an adventure. :)

[BruebachL]

I've made a simple script to merge in UUIDs but it's not 100% accurate yet (something to do with the names and normalization).
I leave this for someone more familiar with the system to polish. I think the normalization logic is backwards right now (as in, we should be adding whitespaces instead of trimming them if we fail to find?). In any case, this script worked flawlessly for the few tokens I've tested and I haven't looked into the misses closely yet.

import json
import xml.etree.ElementTree as ET

print("Loading AllPrintings.json...")
with open("AllPrintings.json", "r", encoding="utf-8") as f:
    all_printings = json.load(f)

def normalize_name(name: str) -> str:
    """Lowercase, strip whitespace, and remove trailing 'token'."""
    name = (name or "").lower().strip()
    if name.endswith(" token"):
        name = name[:-6].strip()
    return name

json_lookup = {}
print("Building JSON lookup...")
for set_code, set_data in all_printings["data"].items():
    for card in set_data.get("tokens", []):
        if card.get("layout") == "token":
            json_lookup[(set_code, normalize_name(card["name"]))] = card["uuid"]

print(f"Loaded {len(json_lookup)} entries from JSON.")

print("Loading tokens.xml...")
tree = ET.parse("tokens.xml")
root = tree.getroot()

cards_section = root.find("cards")
if cards_section is None:
    raise ValueError("No <cards> section found in XML.")

updated_count = 0
missing_count = 0

for card_elem in cards_section.findall("card"):
    name_elem = card_elem.find("name")
    if name_elem is None:
        continue

    card_name = normalize_name(name_elem.text)
    set_elems = card_elem.findall("set")
    if not set_elems:
        continue

    found_any = False

    for set_elem in set_elems:
        set_code = (set_elem.text or "").strip()

        # Try direct match
        uuid = json_lookup.get((set_code, card_name))

        # If not found, try with 'T' prefix
        if not uuid:
            uuid = json_lookup.get(("T" + set_code, card_name))

        if uuid:
            set_elem.set("uuid", uuid)
            updated_count += 1
            print(f"[OK] {set_code} - {name_elem.text} -> {uuid}")
            found_any = True
        else:
            missing_count += 1
            print(f"[MISS] {set_code} - {card_name}")

    if not found_any:
        print(f"[NO UUID FOUND for any set] {name_elem.text}")

tree.write("tokens_updated.xml", encoding="utf-8", xml_declaration=True)

print(f"Done. Updated {updated_count} sets, {missing_count} missing matches.")

[SlightlyCircuitous]

Thank you for the script, BruebachL! It works great and I really appreciate how informative the output is. I made a few changes in order to handle cards with layout values of 'normal' (dungeons, apparently), 'flip' (mostly role tokens), 'double_faced', and 'emblem'. I also adjusted normalize_name() to handle tokens that are nearly exact copies (e.g. from Embalm) since we mark them with '(Token)' at the end in the XML. I got the misses down from 390 to 134, (including fixing a couple tokens that were just mislabeled in Cockatrice) and I think that's nearly as far as we can go. The remaining misses fall into several categories:

Fixable:

Tokens that need to be fixed in Cockatrice (2)
DMU Ragavan should be in DMC
DMC Insect should be in DMU
Potentially fixed in this PR: #321

Tokens that need to be fixed in MTGJSON (1)
Tamiyo, Seasoned Scholar Emblem is labeled in the JSON as simply "Tamiyo, Seasoned Scholar", unlike every other emblem.

Tokens I am not sure where to fix (2)
'Experience Counter' is called 'Experience' in MTGJSON and on Scryfall, unlike 'Posion Counter' which is consistently called exactly that. Can't just strip 'Counter' and find a match. There are two printings in the XML.

Difficult to handle:

'Phyrexian' Tokens (29)
Some cards make tokens that are called out as being specifically called 'Phyrexian X', but those specific tokens don't exist (or didn't until recently). Cockatrice uses particularly Phyrexian-looking tokens for these entries, even if they don't say 'Phyrexian'. As such, old printings of Phyrexian horror, wurm, myr, germ, beast, golem, zombie, minion, and insect tokens are labeled as 'Phyrexian' in Cockatrice, but not in MTGJSON. (More info here: #151)

Sets that have tokens in the XML, but no entries in MTGJSON (accounts for 42 tokens)
8ED, APC, CHK, CSP, DST, INV, LGN, MRD, ODY, ONS, PLS

Tokens that were never printed in paper and aren't recorded anywhere (58)
I have a full list if you are interested, but I won't reproduce it here. Many of these were fabricated by the community with custom art.

[SlightlyCircuitous]

BruebachL's script with my modifications:

def normalize_name(name: str) -> str:
    """Lowercase, strip whitespace, and remove trailing 'token' or '(token)'."""
    
    name = (name or "").lower().strip()
    
    # normalize generic token names to match json (e.g. 'bear token' -> 'bear')
    if name.endswith(" token"):
        name = name[:-6].strip()
        
    #normalize copy token names to match json (e.g. 'adorned pouncer (token)' to 'adorned pouncer')
    elif name.endswith(" (token)"):
        name = name[:-8].strip()
    
    return name

#load the json from local file

print("Loading AllPrintings.json...")
with open('./AllPrintings.json', "r", encoding="utf-8") as f:
    all_printings = json.load(f)
    print("Building JSON lookup...")
    json_lookup = {}

    for set_code, set_data in all_printings["data"].items():
        for card in set_data.get("tokens", []):

            if card.get("layout") in ("token","emblem","normal","flip"):
                json_lookup[(set_code, normalize_name(card["name"]))] = card["uuid"]
            if card.get("layout") == 'double_faced_token':
                json_lookup[(set_code, normalize_name(card["faceName"]))] = card["uuid"]


print(f"Loaded {len(json_lookup)} entries from JSON.")

#build the xml tree from tokens file

print("Loading tokens.xml...")
tree = ET.parse("./tokens.xml")
root = tree.getroot()

cards_section = root.find("cards")
if cards_section is None:
    raise ValueError("No <cards> section found in XML.")
    
#match tokens in the xml to UUIDs in the JSON

updated_count = 0
missing_count = 0

for card_elem in cards_section.findall("card"):
    name_elem = card_elem.find("name")
    if name_elem is None:
        continue

    card_name = normalize_name(name_elem.text)
    set_elems = card_elem.findall("set")
    if not set_elems:
        continue

    found_any = False


    for set_elem in set_elems:
        set_code = (set_elem.text or "").strip()

        # Try direct match
        uuid = json_lookup.get((set_code, card_name))

        # If not found, try with 'T' prefix
        if not uuid:
            uuid = json_lookup.get(("T" + set_code, card_name))

        if uuid:
            set_elem.set("uuid", uuid)
            updated_count += 1
            print(f"[OK] {set_code} - {name_elem.text} -> {uuid}")
            found_any = True
        else:
            missing_count += 1
            print(f"[MISS] {set_code} - {card_name}")

    if not found_any:
        print(f"[NO UUID FOUND for any set] {name_elem.text}")
        
#write a new xml file with UUIDs and output summary

tree.write("tokens_updated.xml", encoding="utf-8", xml_declaration=True, short_empty_elements=False)

print(f"Done. Updated {updated_count} sets, {missing_count} missing matches.")

[SlightlyCircuitous]

Also, maybe I missed it, but I don't think this accounts for different tokens with the same name in the same set e.g. the 4/4 green Elemental and the 1/1 red Elemental tokens in UMA.

[ebbit1q]

to differentiate those I propose that instead of keying by name we key by name, text, pt, and colors

I'd also like to note that for all tokens with scryfall art the uuid is already part of the image url

[BruebachL]

I'd like to avoid relying on the providerId in the url since this seems like coding in a dependency on manual labor.

[ebbit1q]

some tokens have multiple printings in a set, they have unique urls and uuids