Dear @TsuyoshiUshio, @fbeltrao and @ryancrawcour,
as I see you have upgraded to Confluent.Kafka 1.4.3 within #151 the other day, I would like to humbly point out confluentinc/librdkafka#3109 to you.
Without using the Azure Kafka bindings yet, we and others tripped into different issues when connecting to Event Hubs with librdkafka <1.5.0 through its Python binding confluent-kafka-python when running within a Kubernetes environment on Azure.
Now, as I believe the same things might also apply to the Azure Kafka bindings, I wanted to give you a heads up on this.
Without any offense to you or the networking infrastructure within Azure, citing others:
TL;DR: Azure has a nasty artificial limitation that results in being unable to use long-lived TCP connections that have >= 4 minutes of radio silence at any given point.
After some issues revolving around that had been popping up on the Confluent's issue trackers, @edenhill was able to add some mitigations to librdkafka just recently.
Apart from upgrading to the most recent version of librdkafka, I would like to emphasize that it is probably also crucial to apply the recommended configuration settings for Kafka on Azure as outlined within confluentinc/librdkafka#3109 (comment).
After applying these measures to our systems, things have improved tremendously.
With kind regards,
Andreas.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4