CDC State Monitoring
State Monitoring
grun_p gpe "cat $(gadmin config get System.DataRoot)/gstore/0/part/cdc.yaml"
The cdc state file should only exist on GPE Replica 1 node of each partition:
-
GPE_1#1
-
GPE_2#1
------------------ Output example for cdc.yaml on 1x2 cluster ------------------- ### ---- (m1)_10.128.0.77 ---0-- tid: 3752092 safe_persistent_tid: 3752091 split_index: 0 index: 0 ### ---- (m2)_10.128.0.84 ---1-- cat: /home/tigergraph/tigergraph/data/gstore/0/part/cdc.yaml: No such file or directory
Field | Meaning |
---|---|
|
The tid is the tid of the delta batch (largest tid) that was most recently written to external kafka. For a transaction, it’s the tid of the last delta batch for the transaction (must be COMMIT), which can be used to represent the transaction id. |
|
The All delta batches with tid |
|
For non-transaction, it’s always 0. For transactions, the split_index is the index of the delta batch that was most recently written to external kafka among all delta batches for the same transaction. |
|
The index is an index of delta messages in a delta batch that was most recently written to external kafka. |
State of DIM Service
grun_p gpe "cat $(gadmin config get System.DataRoot)/gstore/0/part/deleted_idmap_state.yaml"
grun_p gpe "du -sh $(gadmin config get System.DataRoot)/gstore/0/part/deletedID_store"
The dim state file exists on all GPE nodes, unlike the cdc state file.
------------ Output example for deleted_idmap_state.yaml on 1x2 cluster ------------ ### ---- (m1)_10.128.0.77 ---0-- posted_delta_curr_tid: 3752092 posted_delta_next_split_index: 1 stored_deletedvid_curr_tid: 2275848 purged_deletedvid_curr_tid: 3752091 ### ---- (m2)_10.128.0.84 ---0-- posted_delta_curr_tid: 3752092 posted_delta_next_split_index: 1 stored_deletedvid_curr_tid: 2275848 purged_deletedvid_curr_tid: 3752091
The purging task runs every 30 minutes by default. |
Field | Meaning |
---|---|
|
When reading a delta message with |
|
The All |
|
When processing the special All Note: The |
|
Periodically, TigerGraph purges the entries in RocksDB based on the |
|
Purges Configure this with |
CDC Reset
Please run the following steps to reset cdc.
-
Stop related services
gadmin stop gpe -y
After a CDC reset, the previous data updates before
stop gpe
will be ignored by TigerGraph CDC. Only new data updates will be produced to external kafka. -
Clear state and data files for cdc
grun_p gpe "rm $(gadmin config get System.DataRoot)/files/cdc#*" grun_p gpe "rm $(gadmin config get System.DataRoot)/gstore/0/part/cdc.yaml" grun_p gpe "rm $(gadmin config get System.DataRoot)/gstore/0/part/deleted_idmap_state.yaml" grun_p gpe "rm -rf $(gadmin config get System.DataRoot)/gstore/0/part/deletedID_store"
-
Start services
gadmin start all