{"id":162,"date":"2024-12-04T10:57:18","date_gmt":"2024-12-04T10:57:18","guid":{"rendered":"https:\/\/blogs.remiges.tech\/tech\/?p=162"},"modified":"2025-04-29T06:56:08","modified_gmt":"2025-04-29T06:56:08","slug":"debezium-eco-system-tutorial-part-2","status":"publish","type":"post","link":"https:\/\/blogs.remiges.tech\/tech\/debezium-eco-system-tutorial-part-2\/","title":{"rendered":"Debezium Eco-system Tutorial (Part 2)"},"content":{"rendered":"\n

Introduction<\/strong><\/p>\n\n\n\n

This is in continuation of the earlier part 1 blog. Part 2 of this blog has all the tutorial details on how to create a streaming CDC (Change Data Capture) data using Debezium and Apache Kafka.<\/p>\n\n\n\n

This blog describes the prerequisites, steps for installation and how to configure the components to implement a streaming CDC data from PostgreSQL RDBMS to Apache Kafka and then any consumer can utilize the CDC data.<\/p>\n\n\n\n

Past Context<\/strong><\/p>\n\n\n\n

Part 1 of the blog describes the eco system for the Debezium and advantages of using Debezium as a CDC software component along with the Apache Kafka.<\/p>\n\n\n\n

Prerequisites of Software Components<\/strong><\/p>\n\n\n\n

The following are the prerequisites for this blog, if you want to follow the steps for this developmental cluster. In this tutorial, I have used Windows Operating System, but you can easily follow the same steps for the MacOS and other linux OS variants.<\/p>\n\n\n\n

Following are the prerequisite components:<\/p>\n\n\n\n

    \n
  1. Download Debezium PostgreSQL connector – see the link at the end of this tutorial.<\/li>\n\n\n\n
  2. Install PostgreSQL RDBMS – if not already installed<\/li>\n\n\n\n
  3. Download the Apache Kafka from web site – Download Kafka<\/a><\/li>\n<\/ol>\n\n\n\n

    List of Commands are as follows:<\/strong><\/p>\n\n\n\n

    This creates the database inside the PostgreSQL RDBMS:CREATE database exampledatabase;<\/td><\/tr>
    This will change the current database to example database:\\c exampledatabase<\/td><\/tr>
    This shows all the JSON CDC payloads for the table account.
    bin\\windows\\kafka-console-consumer.bat –topic dbserver1.public.account –from-beginning –bootstrap-server localhost:9092<\/td><\/tr>
    This shows all the JSON CDC payloads for the table transactions.
    bin\\windows\\kafka-console-consumer.bat –topic dbserver1.public.transactions –from-beginning –bootstrap-server localhost:9092<\/td><\/tr>
    This shows all the list of topics inside the Kafka cluster.
    bin\\windows\\kafka-topics.bat –bootstrap-server localhost:9092 –list<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    Installation Steps<\/strong><\/p>\n\n\n\n

    The link to the Github README.md file, which has all the details.<\/p>\n\n\n\n

    README.md file location<\/a><\/p>\n\n\n\n

    Test the running System<\/strong><\/p>\n\n\n\n

    You can see the CRUD operations happening inside the RDBMS as the JSON payloads<\/p>\n\n\n\n

    in the Apache Kafka consumer window.<\/p>\n\n\n\n

    There are many Kafka consumers which can be used to sink the JSON payload data generated from the cluster. All CDC data of the RDBMS can be consumed by the Kafka consumers.<\/p>\n\n\n\n

    Sample JSON CDC Data Output  <\/strong><\/p>\n\n\n\n

     “payload”: {    
    “before”: null,    
    “after”: {      
    “id”: 1,     
      “full_name”: “John Doe”,    
      “date_of_birth”: 5644,    
      “address1”: “123 Main St, Apt 4B”,    
      “pin_code”: “10001”,    
      “created_on”: 1732796365350222,     
      “updated_on”: 1732796365350222   
     },
    }<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    Configuration Items<\/strong><\/p>\n\n\n\n

    The configuration items are available at the GitHub at this link:<\/p>\n\n\n\n

    Debezium Related all the config files<\/a><\/p>\n\n\n\n

    Conclusion<\/strong><\/p>\n\n\n\n

    The Debezium can stream the CDC data from the RDBMS and stream to the Kafka consumer. There can be multiple Kafka consumers which can consume the CDC data for ex, BI and Analytics tools, other event driven architecture microservices for reports and dashboards.<\/p>\n\n\n\n

    External References<\/strong><\/p>\n\n\n\n

      \n
    1. Debezium connector for PostgreSQL<\/a><\/li>\n\n\n\n
    2. Debezium Documentation<\/a><\/li>\n\n\n\n
    3. Debezium Part 1 Blog<\/a><\/li>\n\n\n\n
    4. Download Kafka<\/a><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

      Introduction This is in continuation of the earlier part 1 blog. Part 2 of this blog has all the tutorial details on how to create a streaming CDC (Change Data Capture) data using Debezium and Apache Kafka. This blog describes the prerequisites, steps for installation and how to configure the components to implement a streaming […]<\/p>\n","protected":false},"author":7,"featured_media":144,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-162","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/posts\/162","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/comments?post=162"}],"version-history":[{"count":2,"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/posts\/162\/revisions"}],"predecessor-version":[{"id":168,"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/posts\/162\/revisions\/168"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/media\/144"}],"wp:attachment":[{"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/media?parent=162"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/categories?post=162"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.remiges.tech\/tech\/wp-json\/wp\/v2\/tags?post=162"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}