International Journal on Science and Technology

E-ISSN: 2229-7677     Impact Factor: 9.88

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

Call for Paper Volume 17 Issue 1 January-March 2026 Submit your research before last 3 days of March to publish your research paper in the issue of January-March.

Event-Driven Microservices and Streaming Data Platforms: Applying Kafka + Confluent in Real-Time Microservice Mesh on AWS & Azure

Author(s) Girish Rameshbabu
Country United States
Abstract As enterprise ecosystems transition from monolithic architectures to distributed microservices, traditional synchronous communication patterns (REST/gRPC) increasingly lead to temporal coupling, operational fragility, and data silos. These challenges are exacerbated in multi-cloud strategies where data must reside and synchronize across Amazon Web Services (AWS) and Microsoft Azure. This paper proposes a solution through the implementation of a decentralized "Event Mesh" architecture using Apache Kafka and the Confluent Ecosystem as the streaming backbone.
We investigate the transition from request-response orchestration to asynchronous event choreography, addressing the "Dual-Write" problem via Change Data Capture (CDC) and ensuring data governance through the Confluent Schema Registry. A primary focus is placed on the technical mechanics of Confluent Cluster Linking to bridge the cloud divide without the overhead of traditional mirroring tools. Furthermore, we address critical implementation challenges, including Exactly-Once Semantics (EOS) for cross-cloud transactions, security via mTLS, and performance optimization across high-latency inter-cloud links. The resulting architecture provides a persistent, replayable, and scalable communication mesh that ensures real-time data consistency and fault tolerance across a heterogeneous cloud landscape.
Field Engineering
Published In Volume 17, Issue 1, January-March 2026
Published On 2026-01-09
DOI https://doi.org/10.71097/IJSAT.v17.i1.10298
Short DOI https://doi.org/hbphz5

Share this