test(kafka): fix flaky consumer ignore-topic tests with producer.flush()

[dependabot]

Kafka tests sending to multiple topics were non-deterministic after migrating from bitnami/kafka:3.1 to apache/kafka:4.1.0 (KRaft mode). producer.send() is async — messages to 3 different topics/partitions can arrive at the broker at different times, causing the consumer to poll, receive only foo+bar, then time out before test is committed.

Changes

• Add producer.flush() after sends in delayed_send for:
  • test_kafka_consumer_ignore_topic
  • test_kafka_consumer_ignore_topic_ongoing_transaction

This ensures all messages are broker-acknowledged before the consumer polls, guaranteeing all 3 are visible within consumer_timeout_ms.

def delayed_send():
    time.sleep(0.2)
    producer.send(topic="foo", key=b"foo", value=b"bar")
    producer.send("bar", key=b"foo", value=b"bar")
    producer.send("test", key=b"foo", value=b"bar")
    producer.flush()  # wait for broker ack before consumer polls

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[fr4nc1sc0-r4m0n]

LGTM

[fr4nc1sc0-r4m0n]

@copilot fix the testcases

[Copilot]

@copilot fix the testcases

Fixed in commit 7518ac3c. The failing test test_kafka_consumer_ignore_topic_ongoing_transaction was not receiving the test topic message before consumer_timeout_ms elapsed. With messages sent to 3 different topics (foo, bar, test) asynchronously, they may not all be committed to the broker before the consumer polls — especially with Apache Kafka 4.1.0. Added producer.flush() to delayed_send in both test_kafka_consumer_ignore_topic and test_kafka_consumer_ignore_topic_ongoing_transaction to ensure all messages are acknowledged by the broker before the consumer polls.