Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/fede1024/kafka-view below:

GitHub - fede1024/kafka-view: Kafka web UI

Kafka-view is an experimental web interface for Kafka written in Rust. Kafka-view creates and maintains a materialized view of the internal state of Kafka including cluster metadata, traffic metrics, group membership, consumer offsets etc. It uses the rdkafka Kafka client library for Rust, and rocket.

Click here for screenshots.

Kafka-view supports multiple clusters and implements a fast search functionality to quickly find a topic or consumer group by name or by regex, across all clusters.

• Available data:
  • Broker and topic metrics: byte rate and message rate for each broker and topic in every cluster.
  • Topic metadata: leader, replicas, ISR, topic health.
  • Group membership: show active consumer groups and members, easily find all the consumers for a given cluster or topic.
  • Consumer offsets: show the current consumer offsets, the high watermark and the difference between the two.
  • Consume topic content directly from the web UI.
• Search:
  • Omnisearch: search for broker, topics and consumers in a single query.
  • Search topics in all clusters by name or regex.
  • Search consumers in all clusters by name or regex.
  • Sort by any field (traffic, consumer lag, etc)

At the moment kafka-view is designed to be read-only. Functionality such as adding topics, changing consumer offsets etc. are not supported.

First, create a new configuration starting from the example configuration file. The new configuration should contain the list of clusters you want to monitor, and a special topic in one of the clusters that kafka-view will use for caching.

The caching topic should be configured to use compaction. Example setup:

# Create topic
kafka-topics.sh --zookeeper <zk> --create --topic <cache_topic_name> --partitions 3 --replication-factor 2
# Enable compaction
kafka-topics.sh --zookeeper <zk> --alter --topic <cache_topic_name> --config cleanup.policy=compact
# Compact every 10MB per partition
kafka-topics.sh --zookeeper <zk> --alter --topic <cache_topic_name> --config segment.bytes=10485760

To compile and run:

rustup override set $(cat rust-toolchain)
cargo run --release -- --conf config.yaml

To build Docker image and run(Assuming you have config.yaml in current working directory and set port to 8080 in it):

docker build -t kafka-view .
docker run --rm -p 8080:8080 -v `pwd`/config.yaml:/root/config.yaml kafka-view --conf config.yaml

Or you can use prebuilt image from Docker hub:

docker pull fede1024/kafka-view
docker run --rm -p 8080:8080 -v `pwd`/config.yaml:/root/config.yaml fede1024/kafka-view --conf config.yaml

Kafka exports metrics via JMX, which can be accessed via HTTP through jolokia. The suggested way to run jolokia on your server is using the JVM agent. Example:

KAFKA_OPTS="-javaagent:jolokia-jvm-1.3.7-agent.jar=port=8778" ./bin/kafka-server-start.sh config/server.properties

To verify that it's correctly running:

curl http://localhost:8778/jolokia/read/java.lang:type=Memory/HeapMemoryUsage/used

Once your cluster is running with Jolokia, just add the jolokia port to the kafka-view configuration and it will start reading metrics from the cluster.

• Metadata: cluster metadata is periodically polled using a background thread pool. Cluster metadata conatins: topic information (leader, replicas, ISR), broker information (broker id, hostname, etc), group membership (group state, members etc).
• Metrics: metrics such as byte rate and message rate per topic are polled in the background using a thread pool. Metrics are read using Jolokia, that mush be active on the Kafka brokers.
• Consumer offsets: Kafka-view consumes the __consumer_offsets topic and constantly receives the last offset commit for every consumer in every cluster.

Every data is internally stored using a set of in-memory data structures holding a normalized view of the last available value. When a web page is loaded, the normalized data is combined together to generate the required rapresentation of the data.

As a new update is received from the background polling threads or the __consumer_offsets topics, a new event is created. Each event will update the internal memory structures, and will also be periodically stored in a compacted topic in Kafka. Kafka compaction will guarantee that the last update for every key will be available on the topic.

When kafka-view restarts, the compacted topic is consumed and the internal memory structures are restored to the previous state. In future version this model will allow kafka-view to run in clustered mode, where multiple kafka-view instances will work together to poll data from Kafka and will share the information using the compacted topic.

Thanks to:

• messense

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4