Can we handle opinions, authorities, and taxon table differently?

[aazaff]

Simplifying greatly, the current workflow for creating a taxon in PBDB follows this set of operations.

1. A new taxon X is first created by adding an entry to the Authorities table (through an Authorities API route)
2. An opinion can then be entered about that taxon in the Opinions table (through an Opinions API route)
   1. Although there are theoretically many types of opinion that can be selected they boil down to types
      1. X taxon belongs to Y parent taxon
      2. X taxon is actually taxon Z
      3. X taxon does not actually exist
   2. Multiple, conflicting opinions can be entered for a single taxon
      1. higher-quality references or opinions are always preferred. Quality is ranked (in order) as:
         1. Stated with Evidence
         2. Stated without Evidence
         3. Implied
         4. Second Hand
      2. if two references or opinions of the same quality conflict, the most recently published one is preferred
3. The taxa table is then built daily by analysis of the Authorities + Opinions tables to determine the final status of taxon X.

A possible alternative workflow is to only have a Taxa and Other Opinions table. Here is the assumed workflow in this world. (Note: although this feels like it is a violation of REST, I did some research and this workflow is REST-OK in justified circumstances, which this might be).

1. New taxon X is created by adding an entry to the Taxa table (through a Taxa API route)
2. When a new opinion about X is created, it is also submitted through the Taxa API route
   1. If the opinion supersedes the current information in the Taxa (according to the rules for opinion preference) above, then a new version entry is added to the taxon table.
   2. If the opinion would not supersede the current information about the taxon, then an entry is recorded in an Other Opinions table.

The benefit of this approach is that there would no longer need to be nightly-query/rebuilds of the taxa table based on analysis of Opinions and Authorities. There is a one time check of the preference logic upon upload and the taxa table will therefore always reflect the "correct" information.

[aazaff]

I've been thinking about the above and I don't like it because I think it makes things difficult if/when attempting to retroactively edit data - e.g., what if an opinion entry is revised from Stated Without Evidence to Stated With Evidence?

Another aspect that I've been dwelling on is how the Authorities table is not really a table of Authorites, but is rather a table of taxonomic names.

Here is another alternative model that I am considering.

1. Authorities table is now genuinely a table of Authorities. Simplified example below.

CREATE TABLE authorities (
    id integer GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    permid integer NOT NULL,
    authorizer_person_id integer REFERENCES persons("id") NOT NULL, -- Whoever was the authorizer of the enterer
    enterer_person_id integer REFERENCES persons("id") NOT NULL, -- Whoever made the entry or edit
    parenthetical_citation text NOT NULL, -- e.g., Linnaeus 1766, L. R. Cox et al. 1969 (full citation, unstandardized format)
    authors jsonb, -- using same format as in references, can be NULL
    year integer(4) NOT NULL, -- year of authority
    authority_reference_id REFERENCES references ("id"), -- see notes below
    reference_id integer REFERENCES references("id") NOT NULL, -- see notes below
    preceded_by_id integer REFERENCES opinions("id"),
    succeeded_by_id integer REFERENCES opinions("id")
    );

This accommodates the following two scenarios:

1. I am reading Meredith 2026. It cites Lamarck et al. 1818 as the authority for the taxon Abra. I do not have access to the full citation information for Lamarck et al. 1818. The database would thus be updated like this

References table (simplified)

id	reference (jsonb)
18	{"title": "Meredith's article", "year": "2026", "authors": [{"firstName": "Douglas", "lastName": "Meredith"}]}

Authorities Table (simplified)

id	parenthetical_citation	authors	year	authority_reference_id	reference_id
1	Lamarck et al. 1818	NULL (left NULL because we don't know who the et. al are)	1818	NULL	18

Taxa Table (simplified)

id	name	authority_id	reference_id
314	Abra	1	18

2. I am reading Lamarck 1799. It is both the reference and the authority for taxon Nucula. In PBDB this scenario is tracked with a boolean called "ref_is_authority". In this new model, we would no longer track that, but would instead just set the authority_id and authority_reference_id to be the same.

References table (simplified)

id	reference (jsonb)
21	{"title": "Lamarck's article", "year": "1799", "authors": [{"firstName": "Jean-Baptiste", "lastName": "Lamarck"}]}

Authorities Table (simplified)

id	parenthetical_citation	authors	year	authority_reference_id	reference_id
111	Lamarck 1799	"authors": [{"firstName": "Jean-Baptiste", "lastName": "Lamarck"}]}	1799	21	21

Taxa Table (simplified)

id	name	authority_id	reference_id
531	Nucula	111	111

[mclapham]

FYI - there has been discussion about modifying the stated with evidence/etc. list, so that's something we'll want to deal with. At minimum, there isn't a need for three separate "lower-quality" tags, since I don't know if there's a hierarchy among them. Initially, Alroy hoped this would be evidence-based (does the paper provide a phylogeny or discuss why the taxon belongs to its parent, etc.), but in practice that isn't workable. The Treatise, for example, doesn't do any of that, but is considered very authoritative.

We do need some mechanism for flagging that an opinion should override older ones (the de facto use of that drop-down, because the enterer knows that the opinion isn't a great one), but it needs to be phrased diplomatically.

[aazaff]

Along those lines, there are some discrepancies between the actual drop down options and the documentation provided on the current PBDB form. https://paleobiodb.org/classic?action=displayPage&page=classification_quality

In my opinion, this should be updated for both PBDB v1.0 and V2.0.

[mclapham]

Some other things the taxon system does:

1. Tracks type locality for species (and ideally subspecies)
2. Tracks type specimen for species (and ideally subspecies)
3. Tracks type species for genus-group taxa and type genus for family-group taxa

People have expressed interest in tracking type designations in a similar way to the other opinions, so we could have a history of who said what about the type species/genus. There can also be different kinds of type specimens (holotype vs. lectotype) and sometimes multiple specimen designations if a holotype is considered lost and a neotype designated (helpful to record both museum numbers). These things would be a "nice to have" if we could set up the taxon system to accommodate it, but we've survived nearly 30 years without them.

Some other things that may or may not be important when thinking about the data structure (I imagine much of this is on the application logic side of the house, but it may be helpful to set up the data structure in particular ways to accommodate these):

3. All sets of opinions are based on the original name/basionym, but sometimes enterers mistakenly enterer the original name twice under different spellings (once with a subgenus and once without, or because someone has entered it with the wrong original genus, etc.), so the database needs to be able to merge two lists of opinions into a single one. This currently can be done, but there's a bug (the "original name bug") that causes them to be incompletely merged or something. An example of this bug is the taxon Ostrea vesicularis - - note that it's not using the most recent opinion even though newer ones are marked as "with evidence". It's using the most recent opinion for the basionym that the database considers the "original name." In the API output (https://paleobiodb.org/data1.2/taxa/list.txt?base_name=Pycnodonteinae&taxon_status=accepted), Pycnodonta vesicularis, Pycnodonte vesicularis, and Pycnodonte (Phygraea) vesicularis are all output as separate accepted names, even though in the Classic view they are one set of opinons (this is an example where enterers created three versions that got combined through two separate merging events).

4. Conversely, sometimes enterers accidentally combine two distinct taxa into one set of opinions, so the database needs functionality for someone (perhaps super-users) to disentangle them, noting which opinions go with which original taxon. An example of this is the taxon Nautilus mermeti, entangled with Nautilus triangularis. This is also an example of the original name bug.

5. Sometimes papers create secondary homonyms. For example, they recombine Xus aus into Yus, creating Yus aus. However there's already a Yus aus in existence, so the new one is a secondary homonym. When creating this opinion, the enterer needs to be able to choose whether this is a new Yus aus concept (the secondary homonym) or whether the two should be merged (because the existing opinions had data entry mistakes, a la point 3 above). PBDB 1.0 should be able to do this, but the code has a bug so it merges them even if you choose the "create a new one" option.

6. Another thing that happens at the family-group level (superfamily, family, subfamily, tribe, subtribe) is that authors will create a new taxon at one of those ranks, but it is sometimes elevated or demoted to a different rank. E.g., they create subfamily X-inae, which is then elevated to family X-idae by later authors. But sometimes even later people can divide that X-idae into subfamilies, so X-inae has to show up again and there are two X-inae concepts in existence. For opinions assigning a taxon into X-inae, the database needs to know which X-inae the author meant. It can be a hassle when there is a lot of rank shuffling within a group (e.g. tribe raised to subfamily, then new tribes created within it, then subfamily raised to family and tribes raised to subfamily, then maybe new tribes). This isn't that common, except maybe occasionally in insects, so the current workflow may be the least bad way of handling it.