Design Pattern of Streaming Enrichment

Understanding the concept of streaming architecture Photo by Joao Branco on Unsplash Last time, we introduced streaming processing. In order to be able to handle batch and real-time data with a more pure infrastructure, we introduced Kafka and the streaming framework. In this article, we will introduce a common design pattern for streaming, enrichment, and … Read more

Process Streaming Data: Kappa, Lambda Architecture

Data is quickly becoming the new currency of the digital economy, but it is useless if it can’t be processed. The processing of data is essential for subsequent decision-making or executable actions either by the human brain or various devices/applications, etc. There are two primary ways of processing data: namely, batch processing and stream processing. … Read more

Integrate a Distributed DB With Event Streaming

This article is derived from the Distributed Data Systems Masterclass event that I had fun doing with Maheedhar Gunturu from AWS and Tim Spann from StreamNative. What’s Event Streaming? Events are facts or things that happen. In microservices architectures, we can imagine an event affecting a number of services and generating a series of interactions. … Read more

Bring Streaming to Apache Cassandra with Apache Pulsar

Twitch, YouTube, Instagram, Facebook — virtually every major brand nowadays uses live streaming to connect and engage their audience. For enterprises and developers building cloud-native applicationsthis growing trend creates a need for streaming technologies that can reliably handle the rush of massive amounts of data, while also being flexible and easy to manage for developers. … Read more

Spark streaming app resets kafka offset continuously

I have a spark streaming app that runs on spark cluster with 4 node. A few days ago the app keeps resetting Kafka offset and does not fetch Kafka data anymore while the AUTO OFFSET RESET is set, this is the log: 22/06/28 21:39:38 INFO AppInfoParser: Kafka version : 2.0.0 18|stream | 22/06/28 21:39:38 INFO … Read more

MySQL to DynamoDB: Build a Streaming Data Pipeline on AWS Using Kafka

This is the second part of the blog series which provides a step-by-step walkthrough of data pipelines with Kafka and Kafka Connect. I will be using AWS for demonstration purposes, but the concepts apply to any equivalent options (eg running these locally in Docker). This part will show Change Data Capture in action that lets … Read more

Streaming ETL with Apache Kafka in the Healthcare

IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, … Read more

Using Streaming, Pipelining, and Parallelization

Satya Nadella, the CEO of Microsoft, once said, “Every company is now a software company.” Take the online food ordering business; For example, digital ordering and delivery have grown 300% faster than dine-in traffic since 2014. During Covid, online ordering grew 3,868% between February and April in large suburbs in the United States. As a … Read more

Streaming Solution for Better Transparency

What do you do when you have million-dollar equipment in your manufacturing pipeline giving you sleepless nights? To mitigate risk, you might create a digital counterpart of your physical asset, popularly known as the Digital twin, and leverage augmented intelligence derived from data streams. IoT makes the solution affordab,le and big data enables analytics at … Read more

Most Used Streaming Protocols in 2022 With Technical Specifications

We will learn about the following streaming protocols in order: RTMP RTSP WebRTC HLS SRT CMAF RTMP (Real-Time Messaging Protocol) RTMP streaming protocol, a TCP-based technology, was developed by Macromedia for streaming audio, video, and data over the Internet between a Flash player and a server. Macromedia was purchased by its rival Adobe on December … Read more