feat(FieldFormatters): add visual editor

[maxpatiiuk]

Fixing an issue from 2012 😊

Fixes #23

Checklist

• Self-review the PR after opening it to make sure the changes look good
  and self-explanatory (or properly documented)
• Add automated tests
• Add relevant issue to release milestone

Testing instructions

Generally visual editor for Field Formatters should act and look very similar to the one for Record formatters since it shares a lot of code with it.

To test:

• Verify that each field formatter type (constant, any character, ...) can be set in the UI, without user-experience issues, and after saving, can be read back correctly

• Verify that a formatter created/edited using a visual editor works in Specify 7 and Specify 6 (in forms and query results)

  • For reference, here is how to configure a field formatter in Specify 6

• P.S: don't try to enter an emoji character like on the screenshot above - Specify 7 back-end does not support emojis in app resources 😢

• Test prs in this list: feat(FieldFormatters): add visual editor #5075 (comment) (go over the main feature not in details)

Documentation on field formatters

[maxpatiiuk]

these cases should not happen - the UI doesn't allow them. but handling them here just in case (and to satisfy TypeScript)

[grantfitzsimmons]

❤️

[grantfitzsimmons]

1. When creating a field format like GiftNumber using the regular expression /^(KUI|KUBI|NHM)$/, I am unable to set it up correctly.

See that I am supposed to put the 'hint' on the left and 'pattern' on the right:

This works as expected when using a regex validator, but when setting it as the pattern it is not recognized properly.

	<format system="true" name="GiftNumber" class="edu.ku.brc.specify.datamodel.Gift" fieldname="giftNumber" title="" default="true">
		<autonumber>edu.ku.brc.af.core.db.AutoNumberGeneric</autonumber>
		<field type="regex" size="4" value="/^(KUI|KUBI|NHM)$/" byyear="true" pattern="KUBI"/>
	</format>

---

2. After modifying other sections of the format, those changes are not made immediately visible for the 'Example Field'. Instead, it uses the previous field format before the edits have been made.

See that it shows "Required Format: KUBI" in the tooltip:

(This tooltip is most often obscured by the browser's "match the requested formatter" error)
https://github.com/specify/specify7/assets/37256050/d34bcb16-85d5-43fc-8c68-56bef2e36cc6

After making it simply numeric:

---

3. What does the 'Auto Numbering' do that the 'Auto-number' checkbox does not?

[maxpatiiuk]

When creating a field format like GiftNumber using the regular expression /^(KUI|KUBI|NHM)$/, I am unable to set it up correctly.

Looks like the expected regex between Sp6 and Sp7 differed.
In Specify 6, the regex is run on each field part separately, so /^ and $/ are necessary
In Specify 7, all regexes are joined together into on regex, thus, having /^ or $/ will break things if there is more than one part in your formatter.

Fixed:

• When reading from .xml, both Specify 7 front-end and back-end will trim any /^ or $/ from the regex pattern to not break the pattern when multiple parts are joined together. Similarly, outer ( and ) are removed if present
• When front-end saves the updated .xml, the syncers will make sure to add /^ and $/, even if not originally specified by the user. Also, if the pattern includes |, leading ( and trailing ) will be added

In practice, that means:

• When using the visual field format editor, it doesn't matter anymore if you put or not put /^ and $/ in a regex field as the value will be normalized automatically
• Backwards compatibility with Specify 6

[maxpatiiuk]

See that I am supposed to put the 'hint' on the left and 'pattern' on the right:

yeah, it's not ideal that 'hint' comes before the 'pattern' as the opposite order seems more intuitive.

The reason I did it this way is because of the column heading.
For all other part types, the 'hint' column is used both for declaring what the format should be, and for proving the validation message
For regex parts, since regex is not "user friendly" enough to put into a regex message, the 'hint' field is used only for validation messages, and the regex pattern itself is a separate field (which also validates that the regex pattern has correct syntax)

Open to ideas on how to make this more intuitive

[maxpatiiuk]

After modifying other sections of the format, those changes are not made immediately visible for the 'Example Field'. Instead, it uses the previous field format before the edits have been made.

Fixed

[maxpatiiuk]

What does the 'Auto Numbering' do that the 'Auto-number' checkbox does not?

Ups, the "Auto Numbering" checkbox at the top is not supposed to be exposed in the UI - removed
(it corresponds to whether .xml had a tag like <autonumber>edu.ku.brc.specify.dbsupport.CollectionAutoNumber</autonumber>, which is an sp6-only construct - sp7 cares about it only in so far as creating sp6-compatiable field formatters)

[grantfitzsimmons]

@maxpatiiuk Can you resolve the merge conflicts?

[lexiclevenger]

Testing instructions

Generally visual editor for Field Formatters should act and look very similar to the one for Record formatters since it shares a lot of code with it.

To test:

• Verify that each field formatter type (constant, any character, ...) can be set in the UI, without user-experience issues, and after saving, can be read back correctly
• Verify that a formatter created/edited using a visual editor works in Specify 7 and Specify 6 (in forms and query results)
  • For reference, here is how to configure a field formatter in Specify 6
• P.S: don't try to enter an emoji character like on the screenshot above - Specify 7 back-end does not support emojis in app resources 😢

---

Super exciting! I have a few recommendations:

1. If no default formatter is selected, the message says "record formatter. I would change this to "field formatter"

1. The default for the "size" field for most field types is 0, and this can be saved as long as there is a value in the "hint" field. The value will be shown in the formatter, but it will make the format invalid if that value is entered in the field. In the Specify 6 field formatter, the "size" value cannot go below 1; I think adding this in 7 would be a good solution.

3. The user can input and save any value at any length in the "hint" field (see screenshot above) for the numeric field type, but this is misleading because "#" is the only character that actually works, and it will only appear as many times as the value in the "size" field. Maybe this field could be read-only since it's not really functional.

I'd like to hear what others have to say about 2 and 3; they don't conflict with the xml editor, and the "Example field" displays their consequences correctly, but I think these solutions would make the visual editor more user-friendly.

[maxpatiiuk]

If no default formatter is selected, the message says "record formatter. I would change this to "field formatter"

Good point. To have the same message be applicable both in the field formatters editor and record formatters editor, I edited it to say:

"Please designate one of the formatters as default"

The default for the "size" field for most field types is 0, and this can be saved as long as there is a value in the "hint" field. The value will be shown in the formatter, but it will make the format invalid if that value is entered in the field. In the Specify 6 field formatter, the "size" value cannot go below 1; I think adding this in 7 would be a good solution.

Made "size" value not go below 1

The user can input and save any value at any length in the "hint" field (see screenshot above) for the numeric field type, but this is misleading because "#" is the only character that actually works, and it will only appear as many times as the value in the "size" field. Maybe this field could be read-only since it's not really functional.

Fixed. "hint" is now readonly for number fields

[lexiclevenger]

Looks good! Another thing I noticed is that clicking on the "edit" button next to a format in schema config often takes you to the wrong one.

Schema.Config_.Gift._.Specify.7.-.Google.Chrome.2024-07-23.11-30-30.mp4

https://fwri1924-field-editor.test.specifysystems.org/specify/schema-config/en/Gift/

[emenslin]

The carrying over is not really a feature anywhere in 7 as far as I know which is why I think it should be removed. From my perspective I feel like it could be confusing to the user to have values carried over, but if people thing otherwise then I don't mind it staying in. From @maxpatiiuk comment

If by fixed you mean (looking at your example video) the size should revert back to 8 after being truncated to 4 be, the that is possible but a bit tricky.

I do not think this should be the case though. By fixed I did mean to remove it entirely but if other people do think it should be a feature then that's okay. The main reason I wanted it to be removed is because I did find it annoying when testing but if we don't think the users will care about it then I don't mind keeping it in. @grantfitzsimmons do you have any thoughts on this?

[CarolineDenis]

@maxpatiiuk, is this PR waiting on final testing or there are still some code changes needed?

[maxpatiiuk]

no major known issues. testing required. and fixing merge conflicts

[CarolineDenis]

TODO:

• Fix merge conflicts in lib/components/FieldFormatters/index.ts

[emenslin]

Here is a list of every PR Since Jul 6, 2024 relating to

formatters
aggregators
catalog number
formatting in any way
formats in the QB
catalog number issues with 0
catalog number issues with COTypes
Anything else you think that might be relevant

#3501
#4804
#4952
#5240
#5242
#5253
#5355
#5400
#5431
#5476
#5485
#5489
#6115
#6206
#6245
#6246
#6252
#6255
#6285
#6300
#6306
#6324
#6373
#6419
#6469
#6483
#6710
#6721
#6994
#7080
#7143
#7179
#7341
#7452

I did my best to try to only get ones that were relevant but there might be some in here that are unnecessary as there was a lot of PRs to go through.

---

Updated to have only the relevent PRs that do not have open known issues

[emenslin]

I didn't get a chance for a full review but I wanted to comment what I've found so far.

Previous issue that came back

• Error message says 'record formatter' instead of either formatter or field formatter (first mentioned here)

---

Catalog number formats issues

• This might not be a big deal; however, the format marked as the default does not match the COT default format.
• Visual editor is not available for this resource for a default catalog number format

• Preview not available also for a default catalog number format

---

Autonumber

• Auto-number is disabled on a numeric field for certain tables, but auto-number by year seems to work fine. On other tables such as CO, accession, gift, loan, etc. auto-number is enabled for both numeric and by year. Could have to do with this comment #5075 (comment) but I am not sure so I wanted to mention it

09-29_12.17.mp4

Link to the DB I was using: https://ojsmnh20250910-field-editor.test.specifysystems.org/specify/

[emenslin]

I started on the list of PRs but I did not finish so this only includes the PRs I had a chance to get through.

• Unhide string comparison operators in QueryBuilder #3501
• Add bulk carry forward feature #4804
• Add series tool in query builder #4952
• Fix Table Aggregation Separator being set to undefined if it matches default "; " #5240
• Disable bulk carry for alphanumeric catalogNumber fields #5242

Bulk carry forward is enabled for alphaNumeric catalog numbers in this PR but not in main
Main:

Issue branch:

• Allow Independent Subviews via API #5253
• Fix format of age tables in models #5355
• Make default catalogNumber uniqueness rule editable #5400
• Allow catalogNumber format to be determined by CollectionObjectType #5431
• Check if parser valid only for cat num #5476
• Support COT catalog number configurations in the Query Builder #5485
• Support COT catalog number format configuration in WorkBench #5489
• Fix missing aggregation separators, add patches Django app #6115
• Age Extension Query with Formatted Age Range #6206
• Remove cast to default catalog number formatter #6245
• Add key to display correct cat num format tooltip #6246
• Ensure Correct COT Selection in QB When Sharing Formatters #6252
• Allow to select formatter in field schema config #6255

[CarolineDenis]

@emenslin

• For Disable bulk carry for alphanumeric catalogNumber fields #5242 In feat(FieldFormatters): add visual editor #5075 (comment) fixed in aaeda89

• For Error message says 'record formatter' instead of either formatter or field formatter (first mentioned feat(FieldFormatters): add visual editor #5075 (review)) fixed in 628c9bb with Max's previous fix.

• Auto-number is disabled on a numeric field for certain tables ==> expected

• Visual editor is not available for this resource for a default catalog number format ==> expected ==> visual editor is not available for the formatter that have the external attribute i.e edu.ku.brc.specify.ui.CatalogNumberUIFieldFormatter see line

  specify7/specifyweb/frontend/js_src/lib/components/FieldFormatters/FieldFormatter.tsx

  Line 36 in 68a6973

  {fieldFormatter.external === undefined &&

• This might not be a big deal; however, the format marked as the default does not match the COT default format. ==> it matches what is set to default in the xml

• Preview not available also for a default catalog number format ==> expected in this case; the preview is not available for that type ==> we should document which types are not supported; see line

specify7/specifyweb/frontend/js_src/lib/components/FieldFormatters/index.ts

Lines 50 to 62 in 68a6973

	if (typeof formatter.external === 'string') {
	return parseJavaClassName(formatter.external) ===
	'CatalogNumberUIFieldFormatter'
	? new CatalogNumberNumeric()
	: undefined;
	} else {
	const parts = filterArray(
	formatter.parts.map((part) =>
	typeof part.type === 'string'
	? new fieldFormatterTypeMapper[part.type](part)
	: undefined
	)
	);

[emenslin]

All the above issues I mentioned are fixed! I tested the rest of the PRs I didn't get a chance to before and also removed some either I decided weren't relevent or there are already open known issues with them, looks good.

• #6285
• #6300
• #6306
• #6324
• #6373
• #6419
• #6469
• #6483
• #6710
• #6721
• #6994
• #7080
• #7143
• #7179
• #7341
• #7452

[grantfitzsimmons]

Looking very good. I am really excited to see this in action now! Scoping for the preview is working as expected, non-conforming is clear enough of a descriptor, and it is fairly intuitive for a previously cryptic XML process. There are some interface improvements we could consider implementing in a future release, but this is a strong beginning.