Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/confluentinc/librdkafka/issues/4649 below:

Reading committed offsets where metadata contains null byte leads to reading random data after null byte. · Issue #4649 · confluentinc/librdkafka · GitHub

I was reproducing with code in Rust using rust-rdkafka but that wrapper doesn't do much while reading committed offsets besides calling librdkafka itself. Because in rust-rdkafka it checks that metadata is a valid UTF-8 string it panics with errors like:

Metadata is not UTF-8: Utf8Error { valid_up_to: 3, error_len: Some(1) }

if it starts to return "random" data.

Verified also by implementing OffsetFetch and OffsetCommit in Rust that it's not an issue on the Kafka side - with pure Rust impl I couldn't reproduce issue with reading invalid data.

Use byte array [10, 20, 0, 30, 40] as the commit metadata and commit for any partition. Then read committed offsets via rd_kafka_committed and in some cases metadata after \0 is just different than what was written.

Examples from other tests I've conducted where for the same metadata written we get random responses:

  4 |   0 |  66 |  32 |  64 |  32 |   2 |  16 |  82 | 108 |  25 |  74 | 120 |  24 |  52 |  20 |  58 |  28 |  76 |  22 |  51 |  25 |  82 |  99 |  47 |  91 |  12 |  22 | 115 |  20 | 116 | 100 |  50 |  89 |  76 |  23 |  43 |  49 | 104 |  34 |   0 |   0 | 

  4 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |  90 |  84 |  85 |  77 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 | 160 |  32 |   0 |   0 |   0 |   0 |   0 |   0 |  90 |  84 |  85 |  77 |   0 |   0 | 

  4 |   0 |  66 |  32 |  64 |  32 |   2 |  16 |  93 | 108 |  25 |  74 | 120 |  24 |  52 |  20 |  58 |  28 |  76 |  22 |  51 |  25 |  82 |  99 |  47 |  91 |  12 |  22 | 115 |  20 | 116 | 100 |  50 |  89 |  76 |  23 |  43 |  49 | 104 |  34 |   0 |   0 | 

  4 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 | 120 | 200 | 240 |  78 |   1 |   0 |   0 |   0 |  15 |   0 |   0 |  64 |   0 |   0 |   0 |   0 |   0 |   0 |

  4 |   0 |  66 |  32 |  64 |  32 |   2 |  16 | 106 |  44 |  25 |  74 | 120 |  24 |  52 |  20 |  58 |  28 |  76 |  22 |  51 |  25 |  82 |  99 |  47 |  91 |  12 |  22 | 115 |  20 | 116 | 100 |  50 |  89 |  76 |  23 |  43 |  49 | 104 |  34 |   0 |   0 | 

  4 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 |   0 | 160 |  62 |  84 | 232 |   1 |   0 |   0 |   0 | 239 | 179 | 223 | 191 | 254 | 255 | 255 | 255 |  17 |  76 |  32 |  64 |   1 |   0 |   0 |   0 |   3 |  25 |

rust-rdkafka used librdkafka 2.3.0 - fede1024/rust-rdkafka@87105bc.

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): <REPLACE with e.g., v0.10.5 or a git sha. NOT "latest" or "current">
• Apache Kafka version: 3.7.0
• librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>

this is all I set:

config: ClientConfig {
    conf_map: {
        "bootstrap.servers": "XXX",
        "group.id": "bar",
    },
    log_level: Error,
}

• Operating system: 32-Ubuntu SMP Mon Jan 9 12:28:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Provide logs (with debug=.. as necessary) from librdkafka

Nothing is logged and everything seems to be working just fine.

• Provide broker log excerpts

Can't do it but no errors / warning on the broker side. Also as said above I've configured it isn't the issue purely on the Kafka side.

• Critical issue

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4