⚡️ Speed up function can_unstructured_elements_be_merged by 97%

[codeflash-ai]

📄 97% (0.97x) speedup for can_unstructured_elements_be_merged in unstructured/partition/html/transformations.py

⏱️ Runtime : 206 milliseconds → 105 milliseconds (best of 67 runs)

📝 Explanation and details

The optimization achieves a 97% speedup by introducing LRU caching for HTML parsing - the primary bottleneck in the original code.

Key Optimization: Cached HTML Parsing

• Added @functools.lru_cache(maxsize=4096) decorator to a new _cached_html_tags() function that wraps BeautifulSoup().find_all()
• Replaced duplicate BeautifulSoup parsing calls in can_unstructured_elements_be_merged() with cached lookups
• The profiler shows HTML parsing (BeautifulSoup + find_all) consumed ~49% of total runtime in the original code

Why This Works:

• HTML element text is often repeated across document processing, especially when merging consecutive elements
• BeautifulSoup parsing is expensive (DOM construction, tag analysis) but deterministic for identical input
• LRU cache eliminates redundant parsing with O(1) hash lookups for previously seen HTML strings
• Cache size of 4096 balances memory usage with hit rate for typical document sizes

Performance Impact by Test Case:

• Massive gains on repeated HTML: test_edge_empty_html_strings shows 4567% speedup (44.7μs → 958ns)
• Consistent improvements across all cases: Even unique HTML sees 100-200% speedups from eliminating duplicate parsing within single function calls
• Large-scale processing: test_large_scale_all_mergeable improves 238% (48.8ms → 14.4ms)

Hot Path Context:
Based on function_references, this function is called from combine_inline_elements() which processes consecutive element pairs during HTML document partitioning. This optimization significantly accelerates document processing pipelines where HTML merging is a frequent operation.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 892 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	✅ 1 Passed
📊 Tests Coverage	100.0%

🌀 Generated Regression Tests and Runtime

# Minimal stubs for ontology, elements, and mappings to support the test
from types import SimpleNamespace

# imports
from unstructured.partition.html.transformations import can_unstructured_elements_be_merged

# --- Minimal Elements Implementation for Testing ---


class Metadata:
    def __init__(self, category_depth, text_as_html):
        self.category_depth = category_depth
        self.text_as_html = text_as_html


class Element:
    def __init__(self, metadata):
        self.metadata = metadata


elements = SimpleNamespace(Element=Element)

# ------------- UNIT TESTS FOR can_unstructured_elements_be_merged -------------

# --- Basic Test Cases ---


def test_basic_mergeable_paragraphs():
    # Two paragraphs, same depth, no children, should merge
    e1 = elements.Element(Metadata(1, '<p class="paragraph">Hello</p>'))
    e2 = elements.Element(Metadata(1, '<p class="paragraph">World</p>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 118μs -> 39.9μs (197% faster)


def test_basic_mergeable_inline_and_text():
    # Inline hyperlink and narrative text, same depth, no children, should merge
    e1 = elements.Element(Metadata(2, '<a class="hyperlink">link</a>'))
    e2 = elements.Element(Metadata(2, '<span class="narrative">narrative</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 123μs -> 49.2μs (152% faster)


def test_basic_non_mergeable_different_depth():
    # Same elements, but different category_depth, should not merge
    e1 = elements.Element(Metadata(1, '<p class="paragraph">Hello</p>'))
    e2 = elements.Element(Metadata(2, '<p class="paragraph">World</p>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 292ns -> 250ns (16.8% faster)


def test_basic_non_mergeable_block_element():
    # One element is image (block), should not merge
    e1 = elements.Element(Metadata(1, '<p class="paragraph">Text</p>'))
    e2 = elements.Element(Metadata(1, '<img class="image" src="foo.jpg"/>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 118μs -> 42.2μs (181% faster)


def test_basic_non_mergeable_with_children():
    # Element has children (nested tag), should not merge
    e1 = elements.Element(Metadata(1, '<div class="quote"><span>child</span></div>'))
    e2 = elements.Element(Metadata(1, '<span class="narrative">Text</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 149μs -> 66.6μs (124% faster)


# --- Edge Test Cases ---


def test_edge_empty_html_strings():
    # Both elements have empty html, should merge (no children, both text by fallback)
    e1 = elements.Element(Metadata(0, ""))
    e2 = elements.Element(Metadata(0, ""))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 44.7μs -> 958ns (4567% faster)


def test_edge_one_empty_one_text():
    # One element empty, one is paragraph
    e1 = elements.Element(Metadata(0, ""))
    e2 = elements.Element(Metadata(0, '<p class="paragraph">Text</p>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 85.0μs -> 23.2μs (267% faster)


def test_edge_unrecognized_css_class():
    # CSS class not in map, fallback to UncategorizedText
    e1 = elements.Element(Metadata(1, '<span class="unknown">foo</span>'))
    e2 = elements.Element(Metadata(1, '<span class="unknown">bar</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 124μs -> 51.5μs (142% faster)


def test_edge_unrecognized_html_tag():
    # Tag not in map, fallback to UncategorizedText
    e1 = elements.Element(Metadata(1, "<foo>foo</foo>"))
    e2 = elements.Element(Metadata(1, "<bar>bar</bar>"))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 114μs -> 47.7μs (140% faster)


def test_edge_image_within_span():
    # span with only <img> inside, should be Image (block), not mergeable
    e1 = elements.Element(Metadata(1, '<span><img src="foo.jpg"/></span>'))
    e2 = elements.Element(Metadata(1, "<span>Text</span>"))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 145μs -> 70.0μs (107% faster)


def test_edge_inline_and_annotation():
    # Inline hyperlink and form field value (annotation), should merge
    e1 = elements.Element(Metadata(2, '<a class="hyperlink">link</a>'))
    e2 = elements.Element(Metadata(2, '<input type="text" value="foo"/>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 114μs -> 39.4μs (190% faster)


def test_edge_checkbox_and_radio():
    # Checkbox and radio are annotation, should merge
    e1 = elements.Element(Metadata(1, '<input type="checkbox"/>'))
    e2 = elements.Element(Metadata(1, '<input type="radio"/>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 103μs -> 38.2μs (170% faster)


def test_edge_br_tag():
    # <br> tag is treated as Paragraph, should merge with other text
    e1 = elements.Element(Metadata(1, "<br/>"))
    e2 = elements.Element(Metadata(1, '<span class="narrative">foo</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 113μs -> 45.8μs (148% faster)


def test_edge_layout_element():
    # One element is image (layout), should not merge
    e1 = elements.Element(Metadata(1, '<img class="image" src="foo.jpg"/>'))
    e2 = elements.Element(Metadata(1, '<span class="narrative">foo</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 119μs -> 47.2μs (154% faster)


def test_edge_nested_inline():
    # Inline element with nested inline (should not merge, has children)
    e1 = elements.Element(
        Metadata(1, '<a class="hyperlink"><span class="narrative">foo</span></a>')
    )
    e2 = elements.Element(Metadata(1, '<span class="narrative">bar</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 161μs -> 73.7μs (119% faster)


# --- Large Scale Test Cases ---


def test_large_scale_all_mergeable():
    # 500 pairs of mergeable paragraphs
    for i in range(500):
        e1 = elements.Element(Metadata(0, f'<p class="paragraph">Text {i}</p>'))
        e2 = elements.Element(Metadata(0, f'<p class="paragraph">Text {i+1}</p>'))
        codeflash_output = can_unstructured_elements_be_merged(
            e1, e2
        )  # 48.8ms -> 14.4ms (238% faster)


def test_large_scale_mixed_mergeable_and_non_mergeable():
    # 100 pairs, half mergeable, half not (alternating)
    for i in range(100):
        if i % 2 == 0:
            e1 = elements.Element(Metadata(1, f'<span class="narrative">Text {i}</span>'))
            e2 = elements.Element(Metadata(1, f'<span class="narrative">Text {i+1}</span>'))
            codeflash_output = can_unstructured_elements_be_merged(e1, e2)
        else:
            e1 = elements.Element(Metadata(1, f'<img class="image" src="foo{i}.jpg"/>'))
            e2 = elements.Element(Metadata(1, f'<span class="narrative">Text {i+1}</span>'))
            codeflash_output = can_unstructured_elements_be_merged(e1, e2)


def test_large_scale_all_non_mergeable_due_to_depth():
    # 200 pairs, all with different depths
    for i in range(200):
        e1 = elements.Element(Metadata(i, f'<span class="narrative">Text {i}</span>'))
        e2 = elements.Element(Metadata(i + 1, f'<span class="narrative">Text {i+1}</span>'))
        codeflash_output = can_unstructured_elements_be_merged(
            e1, e2
        )  # 25.1μs -> 24.9μs (0.844% faster)


def test_large_scale_with_children():
    # 50 pairs with children (should not merge)
    for i in range(50):
        e1 = elements.Element(Metadata(1, f'<div class="quote"><span>child {i}</span></div>'))
        e2 = elements.Element(Metadata(1, f'<span class="narrative">Text {i+1}</span>'))
        codeflash_output = can_unstructured_elements_be_merged(
            e1, e2
        )  # 6.69ms -> 2.31ms (190% faster)


# --- Regression/Mutation Tests ---


def test_regression_mutation_not_mergeable_if_child():
    # If one element has a child, must not merge
    e1 = elements.Element(Metadata(1, '<span class="narrative"><span>child</span></span>'))
    e2 = elements.Element(Metadata(1, '<span class="narrative">Text</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 146μs -> 64.5μs (127% faster)


def test_regression_mutation_not_mergeable_if_not_inline_or_text():
    # If one element is not inline nor text, can't merge
    e1 = elements.Element(Metadata(1, '<img class="image" src="foo.jpg"/>'))
    e2 = elements.Element(Metadata(1, '<input type="checkbox"/>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 109μs -> 39.2μs (179% faster)


def test_regression_mutation_mergeable_uncategorized_text():
    # UncategorizedText should be considered text, so mergeable
    e1 = elements.Element(Metadata(1, '<span class="uncategorized">foo</span>'))
    e2 = elements.Element(Metadata(1, '<span class="uncategorized">bar</span>'))
    codeflash_output = can_unstructured_elements_be_merged(e1, e2)  # 124μs -> 51.6μs (141% faster)


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

# imports
from unstructured.partition.html.transformations import can_unstructured_elements_be_merged


# Minimal elements module
class Metadata:
    def __init__(self, category_depth, text_as_html):
        self.category_depth = category_depth
        self.text_as_html = text_as_html


class Element:
    def __init__(self, metadata):
        self.metadata = metadata


class elements:
    Element = Element


# ========== UNIT TESTS ==========

# --- Basic Test Cases ---


def make_element(text_as_html, category_depth=0):
    """Helper to create an elements.Element with given HTML and depth."""
    return elements.Element(Metadata(category_depth, text_as_html))


def test_merge_simple_inline_and_text():
    # Merge two paragraphs at same depth, both with simple text
    el1 = make_element("<p>Paragraph 1</p>", 1)
    el2 = make_element("<p>Paragraph 2</p>", 1)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 114μs -> 38.3μs (198% faster)


def test_merge_inline_and_hyperlink():
    # Merge a span (narrative) and a hyperlink at same depth
    el1 = make_element('<span class="narrative">Some text</span>', 2)
    el2 = make_element('<span class="hyperlink">Link</span>', 2)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 130μs -> 53.3μs (144% faster)


def test_merge_quote_and_paragraph():
    # Merge a quote and a paragraph at same depth
    el1 = make_element('<blockquote class="quote">A quote</blockquote>', 0)
    el2 = make_element("<p>Some text</p>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 112μs -> 39.5μs (184% faster)


def test_merge_different_depth():
    # Should not merge if category_depth is different
    el1 = make_element("<p>Depth 1</p>", 1)
    el2 = make_element("<p>Depth 2</p>", 2)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 250ns -> 208ns (20.2% faster)


def test_merge_with_children():
    # Should not merge if either element has children (nested tags)
    el1 = make_element("<p>Text <span>child</span></p>", 1)
    el2 = make_element("<p>Text</p>", 1)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 155μs -> 71.2μs (118% faster)


# --- Edge Test Cases ---


def test_merge_uncategorized_text():
    # Merge two divs with plain text (UncategorizedText)
    el1 = make_element("<div>plain text</div>", 0)
    el2 = make_element("<div>more text</div>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 116μs -> 48.2μs (142% faster)


def test_merge_img_and_text():
    # Should not merge if one is an image (detected by img tag inside div)
    el1 = make_element('<div><img src="x.png"/></div>', 0)
    el2 = make_element("<div>text</div>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 147μs -> 70.1μs (110% faster)


def test_merge_with_br_tag():
    # <br> should be treated as Paragraph, can merge with text
    el1 = make_element("<br>", 0)
    el2 = make_element("<p>foo</p>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 98.0μs -> 36.6μs (168% faster)


def test_merge_input_checkbox_and_text():
    # Should not merge checkbox (not text/inline) with text
    el1 = make_element('<input type="checkbox"/>', 0)
    el2 = make_element("<p>foo</p>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 104μs -> 36.9μs (183% faster)


def test_merge_span_with_class_and_without():
    # Merge span with class narrative and plain span
    el1 = make_element('<span class="narrative">text</span>', 0)
    el2 = make_element("<span>plain</span>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 121μs -> 51.0μs (138% faster)


def test_merge_hyperlink_and_quote():
    # Merge hyperlink with quote
    el1 = make_element('<span class="hyperlink">link</span>', 1)
    el2 = make_element('<blockquote class="quote">quote</blockquote>', 1)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 120μs -> 45.0μs (167% faster)


def test_merge_with_nested_elements():
    # Should not merge if one element has nested structure (child tag)
    el1 = make_element("<div><span>child</span></div>", 0)
    el2 = make_element("<div>text</div>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 174μs -> 77.2μs (126% faster)


def test_merge_with_empty_html():
    # Should merge two empty divs
    el1 = make_element("<div></div>", 0)
    el2 = make_element("<div></div>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 114μs -> 46.5μs (146% faster)


def test_merge_with_only_whitespace():
    # Should merge two whitespace-only paragraphs
    el1 = make_element("<p>   </p>", 0)
    el2 = make_element("<p>\n\t</p>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 110μs -> 38.1μs (189% faster)


# --- Large Scale Test Cases ---


def test_merge_many_simple_elements():
    # Merge two elements each with 500 inline spans (should be True)
    html1 = "".join(f'<span class="narrative">t{i}</span>' for i in range(500))
    html2 = "".join(f'<span class="hyperlink">l{i}</span>' for i in range(500))
    el1 = make_element(html1, 0)
    el2 = make_element(html2, 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 28.6ms -> 17.1ms (67.7% faster)


def test_merge_many_with_one_nested():
    # 999 simple, 1 with child (should be False)
    html1 = "".join(f'<span class="narrative">t{i}</span>' for i in range(999))
    html1 += "<span><b>nested</b></span>"
    html2 = "".join(f'<span class="hyperlink">l{i}</span>' for i in range(1000))
    el1 = make_element(html1, 0)
    el2 = make_element(html2, 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 56.8ms -> 33.9ms (67.3% faster)


def test_merge_large_with_different_depth():
    # Large elements, but different category_depth (should be False)
    html1 = "".join(f'<span class="narrative">t{i}</span>' for i in range(500))
    html2 = "".join(f'<span class="hyperlink">l{i}</span>' for i in range(500))
    el1 = make_element(html1, 0)
    el2 = make_element(html2, 1)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 375ns -> 375ns (0.000% faster)


def test_merge_large_uncategorized_text():
    # Merge two large divs with plain text (should be True)
    html1 = "".join(f"<div>text{i}</div>" for i in range(500))
    html2 = "".join(f"<div>foo{i}</div>" for i in range(500))
    el1 = make_element(html1, 1)
    el2 = make_element(html2, 1)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 25.3ms -> 15.8ms (60.1% faster)


def test_merge_large_with_image():
    # One element contains an image, should not merge
    html1 = "".join(f"<div>text{i}</div>" for i in range(499)) + '<div><img src="x.png"/></div>'
    html2 = "".join(f"<div>foo{i}</div>" for i in range(500))
    el1 = make_element(html1, 1)
    el2 = make_element(html2, 1)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 25.3ms -> 15.9ms (59.8% faster)


# --- Additional Edge Cases ---


def test_merge_span_and_paragraph():
    # Merge a span and a paragraph, both allowed as text/inline
    el1 = make_element("<span>span text</span>", 0)
    el2 = make_element("<p>para text</p>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 121μs -> 44.5μs (174% faster)


def test_merge_with_input_radio():
    # Should not merge radio input with text
    el1 = make_element('<input type="radio"/>', 0)
    el2 = make_element("<p>foo</p>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 111μs -> 40.7μs (173% faster)


def test_merge_with_input_text():
    # Should not merge input[type=text] with text
    el1 = make_element('<input type="text"/>', 0)
    el2 = make_element("<p>foo</p>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 107μs -> 38.8μs (178% faster)


def test_merge_with_unknown_tag():
    # Unknown HTML tags should be treated as UncategorizedText and allowed
    el1 = make_element("<custom>foo</custom>", 0)
    el2 = make_element("<custom>bar</custom>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 115μs -> 47.7μs (142% faster)


def test_merge_with_nested_unknown_tag():
    # Unknown tag with child should not merge
    el1 = make_element("<custom><b>child</b></custom>", 0)
    el2 = make_element("<custom>text</custom>", 0)
    codeflash_output = can_unstructured_elements_be_merged(
        el1, el2
    )  # 138μs -> 63.7μs (117% faster)


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

from unstructured.documents.elements import Element, ElementMetadata, Text
from unstructured.partition.html.transformations import can_unstructured_elements_be_merged


def test_can_unstructured_elements_be_merged():
    can_unstructured_elements_be_merged(
        Text(
            "",
            element_id=None,
            coordinates=None,
            coordinate_system=None,
            metadata=None,
            detection_origin=None,
            embeddings=None,
        ),
        Element(
            element_id="",
            coordinates=None,
            coordinate_system=None,
            metadata=ElementMetadata(
                attached_to_filename=None,
                bcc_recipient=None,
                category_depth=0,
                cc_recipient=None,
                coordinates=None,
                data_source=None,
                detection_class_prob=None,
                emphasized_text_contents=None,
                emphasized_text_tags=None,
                file_directory=None,
                filename=None,
                filetype=None,
                header_footer_type="",
                image_base64=None,
                image_mime_type="",
                image_url=None,
                image_path=None,
                is_continuation=None,
                languages=None,
                last_modified="",
                link_start_indexes=None,
                link_texts=None,
                link_urls=[],
                links=[],
                email_message_id="",
                orig_elements=None,
                page_name="",
                page_number=None,
                parent_id=None,
                sent_from=[],
                sent_to=None,
                signature="",
                subject="",
                table_as_cells=None,
                text_as_html="",
                url="",
            ),
            detection_origin="",
        ),
    )

🔎 Concolic Coverage Tests and Runtime

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
codeflash_concolic__a8m4ody/tmp7showwtr/test_concolic_coverage.py::test_can_unstructured_elements_be_merged	3.88μs	4.21μs	-7.94%⚠️

To edit these changes git checkout codeflash/optimize-can_unstructured_elements_be_merged-mjcc1y3t and push.