Burnins: Text alignment and padding based on font metrics

[vincentullmann]

Changelog Description

Make sure burnin's through FFMPEG's drawtext aligned with the bottom in Extract Burnin don't jump around based on what text is set in them.

Align them by a static font height instead of the text height so that a particular higher character being in the text for a bit doesn't make the box jump up or down.

Additional info

Came up internally here.

Testing notes:

1. Set up burnins to include e.g. {comment} at the bottom of the burnin.
2. Submit e.g. with comment "oooo" and "he||0" which may come out as different heights per character, etc.

Example Images

[iLLiCiTiT]

The backdrop does not respect full font height for me. And the text is cut off.

[vincentullmann]

The backdrop does not respect full font height for me. And the text is cut off. <img alt="image" width="241"

Ah.. I see. This is caused by this line this line

padding_y = padding - descent

This was done to shrink the box closer to the original dimensions

I've tested a few things and see three possible approaches:

1. remove the adjustment and accept that the box will retain a small amount of padding, even when padding = 0
   (y_align=font + boxborderw={padding})
   

   text_font_noadjust.mp4

2. introduce some additional logic to clamp the padding based on the texts bounding box, in relation to the fonts ascend/descent.
   

   text_baseline_adjust.mp4

3. ignore the bounding box differences and focus only on the alignment of the text itself
   (y_align=baseline + boxborderw={padding})
   

   text_baseline_noadjust.mp4

Note: Options 1 and 3 are considerably simpler from a code perspective, which may make them more robust and easier to maintain. Option 1 and 2 arguably produces the cleanest visual results.
1 would be my choice, but it's the most different to the previous versions.. so that might be an issue

For reference, the script used to generate these examples is attached below:

import subprocess
from PIL import ImageFont


# SETTINGS
FONT_FILE = "/mnt/pipeline/fonts/Roboto-Regular.ttf"
FONT_SIZE = 32
FONT_ALIGN = "font"  # "text", "basline", "font"
ADJUST_PADDING = True


def draw_text(text: str, position: tuple[int, int], padding: int):

    font = ImageFont.truetype(FONT_FILE, FONT_SIZE)
    args = [
        "drawtext@bottomleft",
        f"fontfile='{FONT_FILE}'",
        f"fontsize={FONT_SIZE}",
        f"text='{text}'",
        f"x={position[0]}",
        f"y_align={FONT_ALIGN}",
        "fontcolor=#FFFFFF@1.0",
        "box=1",
        "boxcolor=#338D3885@0.5",
    ]

    if FONT_ALIGN == "font":
        args.append(f"y=h-{position[1]}-font_a")
    else:
        args.append(f"y=h-{position[1]}")

    # get the height of the area below the baseline
    _, _, _, extra_bottom = font.getbbox(text, anchor="ms")

    # distance from the top the current text to the top of an lowercase letter
    (_, dist_lowercase, _, _) = font.getbbox("a", anchor="la")
    (_, dist_top, _, _) = font.getbbox(text, anchor="la")

    pad_l = padding
    pad_r = padding

    if ADJUST_PADDING:
        pad_t = max(padding + dist_top - dist_lowercase, 0)
        pad_b = max(padding - extra_bottom, 0)
    else:
        pad_t = padding
        pad_b = padding

    boxborderw = f"boxborderw={pad_t}|{pad_l}|{pad_b}|{pad_r}"
    args.append(boxborderw)

    return ":".join(args)


def main(i):

    print(f"===== Main {i=} ======")
    filters = [
        draw_text(f"p={i}", (20, 100), padding=i),

        draw_text("Abc",  (100, 100), padding=i),
        draw_text("uvw",  (200, 100), padding=i),
        draw_text("xyz",  (300, 100), padding=i),
        draw_text("_",    (400, 100), padding=i),
        draw_text("~",    (500, 100), padding=i),
        draw_text("^",    (600, 100), padding=i),
    ]

    args = [
        "ffmpeg",
        "-y",
        "-v", "quiet",
        "-i",
        "/mnt/d/work/checkerboard.png",
        "-vf",
        ",".join(filters),
        "-frames:v", "1",
        "-update", "1",
        f"/mnt/d/work/out_a_{i}.png",
    ]
    subprocess.run(args)


for i in range(30):
    main(i)

# for i in [0, 10, 20, 30]:
#     main(i)
# main(8)

[iLLiCiTiT]

I think that option 1 is the best from all options.

[iLLiCiTiT]

BUT what I think is important is that bottom padding should be decresed by descent if descent was used so we're not extending it too much, if possible.

So if I have 5 padding and descent is 2 then bottom padding would change to 3.

[vincentullmann]

BUT what I think is important is that bottom padding should be decresed by descent if descent was used so we're not extending it too much, if possible.

So if I have 5 padding and descent is 2 then bottom padding would change to 3.

This is sort of what I was trying to achieve with the original adjustment / and option 2).

here is a screenshot of padding=0 and padding=10 from option 2, with some lines + some 10x10px reference cuvbes
The padding around lowercase characters and characters without descent is 10px, but for letters such as the p or `y as well as the uppercase ones it is decreased accordingly.

[iLLiCiTiT]

Ok, then if padding is 0 it probably should still cover full height (even if content does not need it).

If there is y, A, a and Ay, they all should have same height of the box.

[vincentullmann]

Ok, then if padding is 0 it probably should still cover full height (even if content does not need it).

If there is y, A, a and Ay, they all should have same height of the box.

ye, thats option 1 then. I think thats the cleanest and most robust.

edit: not sure why none of the characters in my screenshot touched the bottom border. I guess there must the some other character in that font that extends a little bit further down

[iLLiCiTiT]

Not really, it is combination of both. Bottom padding does not start from the "full height" bottom, but from descentless bottom.

bottom_padding = bottom_padding - font.descent
if bottom_padding< 0:
    bottom_padding = 0

So if padding is 0 nothing changes, if padding is 10 but font descent is 2 then it will change to 8.

EDITED:
Maybe it is wrong, as this should also apply to top padding, but there is no "descent", it is height of "smaller" letters. Which makes it "look nicer" but might be confusing.

Why I think it is important: It is weird that the height is so big with padding compared to side padding.

[iLLiCiTiT]

This is how web styles do handle paddings for fonts (this is padding: 5px).

The bottom is from the base line of font.

[vincentullmann]

Ok, I think I got it

here are some examples with padding=0 and padding=10

and as animation going from 0-30
(it looks weird as a animation.. but I think the individual, frames look good)

text_v3.mp4

Explanation:

using PIL's TextAchor Reference

• green = bounding box of the current text (will vary based on content) (font.getbbox(text))
• pink-orange = fixed values (font.getmetrics())
• blue box = the padding

at padding=10

• left and right are the full padding starting at left / right
• bottom:
  • the distance from baseline -> descender is always padded (2px in this example)
  • remaining padding (padding - distance(d, s)) is added on top (8px in this example)
• top:
  • distance from top -> ascender is always added
    (Note: there is no fixed counterpart to baseline. top is based on the current text content and thus varies. I'm using the height of a uppercase letter A in my test to get a fixed value)
  • remaining padding (padding - distance(t, a)) is added

this addresses:

if padding is 10 but font descent is 2 then it will change to 8.

at padding=0

• left/right = no padding
• top/bottom = "forced" padding to cover ascender and descender --> guarantee fixed box height, independent of the content

this adressses:

Ok, then if padding is 0 it probably should still cover full height (even if content does not need it).
If there is y, A, a and Ay, they all should have same height of the box.

compared to web-text

its a bit different to how css does padding.

• red = 5x5px
• green = 12x12px = distance between padding and baseline

so it appears that the padding is simply applied around left / right / ascend and descent like this:

here is a jsfiddle version where I changed the padding background color to confirm this:
https://jsfiddle.net/dqx0vmtc/7

[iLLiCiTiT]

Very nice, the only remaing issue is bottom height of the text. It is cut off at the bottom.

Looks like all you need to do is to subtract bottom padding.

[iLLiCiTiT]

Now there is one more issue, which might change how the burnins do work. With this the y offset actually works as it offsets the whole content including padding.

I do belive that is correct approach, but we should also do the same in horizontal axis. With that the offsets would work as "margins" and we should define default values to 0 instead of 5.

With defaults (x: 5 y: 5)

With (x: 0 y: 5)

EDITED:
Or, it is not right and it should be aligned?

[vincentullmann]

I think it makes sense to include the padding in the x/y positions.

I used that as an opportunity to combine some of the logic, since otherwise we'd have to repeat the padding calulations in two places.

Here are some examples:
(using a font size of 60 for easier debugging)

• offset = yellow
• padding = red box

padding: 10 / offsets: 30/20

note: padding at the top & bottom is slightly larger than 10px to fully include the ascend and descend (blue)

padding=0 / offset=0

padding=20 / offset=10

and here an example where the padding fully encloses the ascend/descent.
Around a "regular" character such as the uppercase A we get the full 20px of padding
but we can see how the y and Á intersect into the padding as expected

[vincentullmann]

I left this in, since it is used for ffmpeg_burnins._drawtext and I dont know if that function would break if the dict contains more than just the expected x and y keys

[iLLiCiTiT]

For some reason filters are not added to arguments for me in current state.

They are added but font paths is not escaped as expected so it fails on windows path C:\....

[iLLiCiTiT]

The bottom still seems to be cut off a little bit. Not sure if that can be really determined correctly?

NOTE: Settings: x/y offset set to 0 and padding to 5.

[vincentullmann]

@iLLiCiTiT
which font is this?

I had a look at Roboto Regular (left, the font I was using in my tests) vs. LiberationSans-Regular.ttf (right)
and the character y does indeed goes beyond the descent line in the latter... :

to avoid clipping text that goes beyond the reported font metrics I replaced font.getmetrics() with a helper method that measures the "real" ascent and descent using font.getbbox

[iLLiCiTiT]

Learning new things every day. Did one last change reuse ascent from get_metrics for pad_t. Now it looks like it works

Really great job!

[iLLiCiTiT]

@BigRoy can you give it a go? Just to confirm...

[iLLiCiTiT]

I did one more change, that I'm not sure if is related to my font. The most right x offset was off by 2 pixels.

How to test:
I did set x offset to 1 to see how much off it is.

[BigRoy]

Before

tes_maya_reviewMain_v005_h264.mp4

After

tes_maya_reviewMain_v006_h264.mp4

---

It's hard to tell for me whether it's better. I like the fact that the text is more centered within their backgrounds, but I personally prefer the old where all the boxes touch the sides, and the data is tighter to the 'sides'

I think if we can keep it so they are as tightly squeezed to the sides then I think this is solid. But now the font is coming in from the sides too much. I'm a bit worried if instead we change the 'defaults' for the text in settings that it may fix it to look more alike, but at the same time anyone with any override to the settings would likely not benefit from this newer standard - so not sure how to keep this better backwards compatible.

So - is there anything we can do make it stick closer to the original states?

[iLLiCiTiT]

but I personally prefer the old where all the boxes touch the sides, and the data is tighter to the 'sides'

That is "autofixed" with settings conversion. To test it without conversion, set x and y offset to 0.

[BigRoy]

but I personally prefer the old where all the boxes touch the sides, and the data is tighter to the 'sides'

That is "autofixed" with settings conversion. To test it without conversion, set x and y offset to 0.

I see now. What a beauty.

tes_maya_reviewMain_v007_h264.mp4

[iLLiCiTiT]

Pillow does not calculate text width correctly, I will also test timecode and list values, because I do believe we changed only one part.

[iLLiCiTiT]

Ok, all tested... Merging, and let's pray.