Building a Data Pipeline Using QuestDB and Confluent Kafka

A data pipeline, at its base, is a series of data processing measures that are used to automate the transport and transformation of data between systems or data stores. Data pipelines can be used for a wide range of use cases in a business, including aggregating data on customers for recommendation purposes or customer relationship … Read more

Kafka With Python: How To Get Your Projects Up and Running | by Evgenii Munin | Aug, 2022

Run streaming jobs with Kafka source: https://www.confluent.io/blog/author/martin-kleppmann/ In this article, we will discuss what Apache Kafka is and its use cases. We will also build a demo example of a Kafka Consumer using Python and Confluent Cloud. Apache Kafka is an open source streaming platform. Even though its code base was written in Java, some … Read more

Understanding Cursors in Apache Pulsar

In my previous blog that introduces Apache BookKeeper, it mentions that Apache Pulsar maintains a cursor ledger for each subscription in Apache BookKeeper. After a consumer has processed a message with an acknowledgment sent to the broker and the broker has received it, the broker updates the cursor ledger accordingly. In this blog, let’s take … Read more

Resilient Kafka Consumers With Reactor Kafka

We are a recipe for creating resilient Kafka consumers using Reactor Kafka. This approach is one that we’ve developed over time and incorporates the learnings from our experience with running Reactor Kafka – and all the challenges that come with that. The consumer described in this article provides at-least-once Delivery semantics using manual acknowledgments, which … Read more

Open API and Omnichannel with Apache Kafka in Healthcare

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, … Read more

Self hosted Kafka Connect Elasticsearch – SSL error

Note: My environment is dockerized I’m using Kafka connect with Elasticsearch sink plugin (extracted in the plugins folder) My elasticsearch cluster is secured with self signed SSL certificates. I have problems when configuring Kafka-connect to use my SSL secured elasticsearch cluster. Connect configuration: { “name”: “elasticsearch-sink”, “config”: { “connector.class”: “io.confluent.connect.elasticsearch.ElasticsearchSinkConnector”, “tasks.max”: “1”, “topics”: “test”, “key.ignore”: … Read more

Why Pulsar Beats Kafka for a Scalable, Distributed Data Architecture

The leading open-source event streaming platforms are Apache Kafka and Apache Pulsar. For enterprise architects and application developers, choosing the right event streaming approach is critical, as these technologies will help their apps scale up around data to support operations in production. Everyone wants results faster. We want applications that know what we want, even … Read more

Transmit Log Messages to a Kafka Topic: Log4j2

Context It could be a basic “Hello, World!” application or a complex banking solution like Stripe, but developing an application is a fascinating process. This process typically includes extensive testing and quality assurance to ensure that not only the requirements are met but also that the application is reliable enough for users to consume. While … Read more

Machine Learning and Data Science With Kafka in Healthcare

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, … Read more

Monitoring Kafka Topic Consumer Lag With AWS Lambda | by Naween Fonseka | Jul, 2022

Kafka consumer lag monitoring with Confluent Metric APIs Photo by Naween Fonseka on Unsplash Recently in my work, we experienced several service level issues that led to Kafka messages not being served and appropriately processed by the Kafka consumers. To identify such situations, we decided to come up with a mechanism where we can get … Read more