design tables to consolidate the kafka messages
Posted by: Ankit Mani Tripathi
Date: August 02, 2018 12:51PM

I have below scenario:

2 DBs: DB1 and DB2 which will later replace DB1. Both needs to be in sync. DB1 is de-normalized and 1 table in DB1 might have multiple tables in DB2. When we are loading data from DB2 to DB1 then we need to consolidate the messages to populate a table in DB1 in one shot.

Need to design a configuration table having below information Target_System (DB1 but later can be DB2 as well) Target_Table Source_table (this will be 1-many with Target Table) Mandatory Source Columns/Optional Source columns

For the entries in this table the messages will be consolidated. To consolidate need to store the messages in landing DB and need to have an indicator set when all the mandatory columns have arrived, combine the message and pass to Target.

The messages will be in Kafka and Landing DB and configuration table needed in SQL server.


Options: ReplyQuote

Written By
design tables to consolidate the kafka messages
August 02, 2018 12:51PM

Sorry, only registered users may post in this forum.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.