Lazy import only required plugin: open 2.3-15.6x & save 2.2-9x faster

[hugovk]

preinit + init

When loading or saveing an image, if Pillow isn't yet initialised, we call a preinit function.

This loads five drivers for five popular formats by importing their plugins: BMP, GIF, JPEG, PPM and PNG.

Then we check each of these plugins in turn to see if one will accept it (which usually involves reading at least part of the image data for a magic prefix), and if so, then load it.

If none of these common five match, we call init, which imports the remaining 42 plugins. We then check each of these for a match.

This has been the case since at least PIL 1.1.1 (released in 2000).

Lazy

This is all a bit wasteful if we only need one or two image formats during a program's lifetime. (Longer running ones may need a few more, but unlikely all 47, and in any case speed is less of a worry for long-running programs.)

This PR adds a mapping of common extensions to plugins. Before preinit and init, we can do a very cheap lookup, and may save us importing many plugins, and save us trying to see load with many plugins.

Of course, we may have an image without an extension, or with the "wrong" extension, but that's fine, I expect it's rare and anyway we'll fallback to the full preinit -> init flow.

Benchmarks

Combined

This scripts times the new code:

• Opening a PNG, and compares it with explicitly calling preinit first.
• Opening a WEBP, and compares it with explicitly calling init first.
• Saving a PNG, and compares it with explicitly calling preinit first.
• Saving a WEBP, and compares it with explicitly calling init first.

import statistics
import subprocess
import sys

OPEN_TEMPLATE = """
import time
from PIL import Image
start = time.perf_counter()
{preload}
im = Image.open("Tests/images/hopper.{ext}")
print(f"{{(time.perf_counter() - start)*1000:.2f}}")
"""

SAVE_TEMPLATE = """
import time
import tempfile
from PIL import Image
im = Image.new("RGB", (100, 100))
start = time.perf_counter()
{preload}
with tempfile.NamedTemporaryFile(suffix=".{ext}") as f:
    im.save(f.name)
print(f"{{(time.perf_counter() - start)*1000:.2f}}")
"""


def run_benchmark(code: str, iterations: int = 20) -> list[float]:
    times = []
    for _ in range(iterations):
        result = subprocess.run(
            [sys.executable, "-c", code], capture_output=True, text=True
        )
        times.append(float(result.stdout.strip()))
    return times


def print_results(name: str, times: list[float]) -> None:
    print(f"  {name}:\tmedian {statistics.median(times):.2f}ms, min {min(times):.2f}ms")


def main():
    print("Benchmarking...\n")

    open_png_opt = run_benchmark(OPEN_TEMPLATE.format(ext="png", preload=""))
    open_png_preinit = run_benchmark(OPEN_TEMPLATE.format(ext="png", preload="Image.preinit()"))
    open_webp_opt = run_benchmark(OPEN_TEMPLATE.format(ext="webp", preload=""))
    open_webp_init = run_benchmark(OPEN_TEMPLATE.format(ext="webp", preload="Image.init()"))

    save_png_opt = run_benchmark(SAVE_TEMPLATE.format(ext="png", preload=""))
    save_png_preinit = run_benchmark(SAVE_TEMPLATE.format(ext="png", preload="Image.preinit()"))
    save_webp_opt = run_benchmark(SAVE_TEMPLATE.format(ext="webp", preload=""))
    save_webp_init = run_benchmark(SAVE_TEMPLATE.format(ext="webp", preload="Image.init()"))

    print("Open:")
    print_results("PNG lazy    (1 plugin)", open_png_opt)
    print_results("PNG preinit (5 plugins)", open_png_preinit)
    print_results("WebP lazy   (1 plugin)", open_webp_opt)
    print_results("WebP init   (47 plugins)", open_webp_init)

    print("\nSave:")
    print_results("PNG lazy    (1 plugin)", save_png_opt)
    print_results("PNG preinit (5 plugins)", save_png_preinit)
    print_results("WebP lazy   (1 plugin)", save_webp_opt)
    print_results("WebP init   (47 plugins)", save_webp_init)

    print("\nSpeedup (lazy vs preload):")
    print(f"  Open PNG:  {statistics.median(open_png_preinit) / statistics.median(open_png_opt):.1f}x")
    print(f"  Open WebP: {statistics.median(open_webp_init) / statistics.median(open_webp_opt):.1f}x")
    print(f"  Save PNG:  {statistics.median(save_png_preinit) / statistics.median(save_png_opt):.1f}x")
    print(f"  Save WebP: {statistics.median(save_webp_init) / statistics.median(save_webp_opt):.1f}x")


if __name__ == "__main__":
    main()

Python 3.10

Open:
  PNG lazy    (1 plugin):       median 3.17ms, min 3.14ms
  PNG preinit (5 plugins):      median 7.20ms, min 7.04ms
  WebP lazy   (1 plugin):       median 1.35ms, min 1.32ms
  WebP init   (47 plugins):     median 21.05ms, min 20.88ms

Save:
  PNG lazy    (1 plugin):       median 3.76ms, min 3.70ms
  PNG preinit (5 plugins):      median 8.44ms, min 7.62ms
  WebP lazy   (1 plugin):       median 2.49ms, min 2.29ms
  WebP init   (47 plugins):     median 22.42ms, min 22.00ms

Speedup (lazy vs preload):
  Open PNG:  2.3x
  Open WebP: 15.6x
  Save PNG:  2.2x
  Save WebP: 9.0x

Python 3.14

Open:
  PNG lazy    (1 plugin):       median 3.13ms, min 3.04ms
  PNG preinit (5 plugins):      median 8.27ms, min 7.98ms
  WebP lazy   (1 plugin):       median 1.56ms, min 1.47ms
  WebP init   (47 plugins):     median 21.98ms, min 20.89ms

Save:
  PNG lazy    (1 plugin):       median 4.51ms, min 3.87ms
  PNG preinit (5 plugins):      median 9.72ms, min 9.06ms
  WebP lazy   (1 plugin):       median 2.83ms, min 2.34ms
  WebP init   (47 plugins):     median 22.41ms, min 21.65ms

Speedup (lazy vs preload):
  Open PNG:  2.6x
  Open WebP: 14.0x
  Save PNG:  2.2x
  Save WebP: 7.9x

Read

These are hyperfine comparisons between main and the new code, and include the overhead of the Python interpreter startup and PIL import Image etc.

import sys
from PIL import Image

im = Image.open(f"Tests/images/hopper.{sys.argv[1]}")

Read png (preinit group)

❯ hyperfine --warmup 5 \
--prepare "git checkout main" "python3 1.py png # main" \
--prepare "git checkout lazy" "python3 1.py png # lazy"
Benchmark 1: python3 1.py png # main
  Time (mean ± σ):      58.9 ms ±   1.4 ms    [User: 48.5 ms, System: 8.8 ms]
  Range (min … max):    57.2 ms …  63.1 ms    28 runs

Benchmark 2: python3 1.py png # lazy
  Time (mean ± σ):      55.4 ms ±   4.6 ms    [User: 44.8 ms, System: 8.4 ms]
  Range (min … max):    52.0 ms …  70.8 ms    29 runs

Summary
  python3 1.py png # lazy ran
    1.06 ± 0.09 times faster than python3 1.py png # main

Read webp (init group)

❯ hyperfine --warmup 5 \
--prepare "git checkout main" "python3 1.py webp # main" \
--prepare "git checkout lazy" "python3 1.py webp # lazy"
Benchmark 1: python3 1.py webp # main
  Time (mean ± σ):      74.0 ms ±   2.8 ms    [User: 61.3 ms, System: 10.9 ms]
  Range (min … max):    71.1 ms …  82.1 ms    24 runs

Benchmark 2: python3 1.py webp # lazy
  Time (mean ± σ):      55.6 ms ±   5.7 ms    [User: 44.1 ms, System: 9.1 ms]
  Range (min … max):    50.8 ms …  70.6 ms    26 runs

Summary
  python3 1.py webp # lazy ran
    1.33 ± 0.15 times faster than python3 1.py webp # main

Save

import sys
from PIL import Image

im = Image.new("RGB", (100, 100), "red")
im.save(f"/tmp/out.{sys.argv[1]}")

Save png (preinit group)

❯ hyperfine --warmup 5 \
--prepare "git checkout main" "python3 2.py png # main" \
--prepare "git checkout lazy" "python3 2.py png # lazy"
Benchmark 1: python3 2.py png # main
  Time (mean ± σ):      68.3 ms ±  12.7 ms    [User: 54.5 ms, System: 10.5 ms]
  Range (min … max):    58.6 ms … 109.1 ms    26 runs

Benchmark 2: python3 2.py png # lazy
  Time (mean ± σ):      60.2 ms ±   7.5 ms    [User: 47.3 ms, System: 9.5 ms]
  Range (min … max):    53.5 ms …  80.1 ms    28 runs

Summary
  python3 2.py png # lazy ran
    1.13 ± 0.25 times faster than python3 2.py png # main

Save webp (init group)

❯ hyperfine --warmup 5 \
--prepare "git checkout main" "python3 2.py webp # main" \
--prepare "git checkout lazy" "python3 2.py webp # lazy"
Benchmark 1: python3 2.py webp # main
  Time (mean ± σ):      75.6 ms ±   2.2 ms    [User: 62.4 ms, System: 11.3 ms]
  Range (min … max):    72.4 ms …  80.8 ms    24 runs

Benchmark 2: python3 2.py webp # lazy
  Time (mean ± σ):      57.6 ms ±   7.4 ms    [User: 44.7 ms, System: 9.2 ms]
  Range (min … max):    51.9 ms …  79.3 ms    23 runs

Summary
  python3 2.py webp # lazy ran
    1.31 ± 0.17 times faster than python3 2.py webp # main