Kafka Fail-Over Using Quarkus Reactive Messaging

Background Recently, I came across a scenario where, at first glance, we needed an active-active Kafka cluster with bi-directional replication using MirrorMaker2. However, while diving deeper into the nuances of the solution that this particular project demanded, a number of factors showed this wasn’t really the case. For example, a look at how the producers/consumers … Read more

Testing Schema Registry: Spring Boot and Kafka

Introduction Apache Kafka is the most widely used reliable and scalable eventing platform. It helps in providing the base for a fast, scalable, fault-tolerant, event-driven microservice architecture. Any type of data can be published into a Kafka topic and can be read from. Kafka does not provide an out-of-the-box schema validation system and hence is … Read more

apache beam – pyflink with kafka java.lang.RuntimeException: Failed to create stage bundle factory

・Python3.8 ・JDK 11 I’ve started learning pyflink and write a code instructed by official web which is https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/datastream/intro_to_datastream_api/ And here is my code from pyflink.common.serialization import JsonRowDeserializationSchema,JsonRowSerializationSchema from pyflink.common import WatermarkStrategy, Row from pyflink.common.serialization import Encoder from pyflink.common.typeinfo import Types from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream.connectors import FlinkKafkaConsumer,FlinkKafkaProducer def streaming(): env = StreamExecutionEnvironment.get_execution_environment() deserialization_schema =JsonRowDeserializationSchema.builder().type_info( … Read more

How To Deploy Apache Kafka With Kubernetes

Kafka is the de facto event store and distributed message broker solution for large microservice architecture systems. Kubernetes is the industry standard for orchestrating containerized services. For many organizations, deploying Kafka on Kubernetes is a low-effort approach that fits within their architecture strategy. In this post, we’ll look at the appeal of hosting Kafka on … Read more

java – Kafka Consumers under the same group are consuming the same partition after rebalancing

According to Kafka documentation: Kafka provides the guarantee that a topic-partition is assigned to only one consumer within a group. But I’m observing different behavior in my service. Here are some details: I’m using Kafka 2.8 and spring-kafka 2.2.13. Initially I had one Kafka topic topic.1 with 5 partitions in it, this topic was consumed … Read more