CDC Message Examples

Message Format Fields

mid

mid is a message identifier field composed of concatenated elements, as exemplified by 2|1701548849757|33|2|1. It is constructed by combining four or five elements, depending on its type, and each element is separated by |.

There are two types: Non-transaction and Transaction:

  • Non-transaction types have 4 elements: partitionID | timestamp | tid | index

  • Transaction types have 5 elements: partitionID | timestamp | tid | split_index | index

In the previous example, it signifies a Transaction with 5 elements: originating from partition 2|, committed at epoch time 1701548849757|, with transaction ID 33|, split_index of 2|, and lastly an index of 1|. The current message is the second message in the third batch of the transaction.

Table 1. This table breaks down the “mid” field elements:
“mid” Field Elements Explain

partitionID

Source GPE partition of the CDC message. It starts from 1.

timestamp

Same as the timestamp field in the CDC message.

tid

A CDC message consistently corresponds to a delta message, both of which are part of a "delta batch" in the delta Kafka queue. The "tid" denotes the position or index of the delta batch to which the original delta message is mapped. This index starts from 1, and a transaction encompasses multiple delta batches. For CDC messages within a transaction, the "tid" corresponds to the “tid” of the last message, specifically the COMMIT message.

split_index

This element is exclusive to transactions. In the context of a transaction, the split_index represents the position of the delta batch among all delta batches for the same transaction. It follows a 0-indexed format.

index

The index of the delta message in its delta batch. It’s 0-indexed.

Other Message Fields

Table 2. Other message fields are outlined in the table below:
Field Value Type: Description of values

type

String: “vertex”, ”vertex-type”, ”edge”

operator

String: ”insert”, ”insert-only”, ”delete”

timestamp

UINT64:

Non-transaction: start epoch time /

Transaction: epoch time of transaction commit message

graph [added in 4.1.1]

String: name of updated graph

typename

String: name of updated object type

content

See Content Field Value Types and "Possible Values" below

Content Field Value Types and "Possible Values"

  • Key is the affected attribute name.

  • Value is a JSON object with 2 fields:

    1. “op”: This defines how to apply the value to existing attribute value. It can be one of: "Overwrite", "Add", "Max", "Min", "And", "Or", "IgnoreIfExisted", "Minus", "ReplaceOrAdd".

    2. “value”: As in this example below:

      "content": {
    "actors": {
        "op": "Add",
        "value": [
            "i1",
            "m2"
        ]
    },
    ...
}

Message Examples

CDC Message with Type: Vertex

A CDC message for a vertex entity can describe an insertion, modification, or deletion.

Example: Update two attributes of the Company vertex comp1.
{
    "mid": "1|1701548849757|1|0",
    "operator": "insert",
    "timestamp": 1701548849757,
    "type": "vertex",
    "graph": "SocialGraph",
    "typename": "Company",
    "vid": 99999999,
    "uid": "comp1",
    "content": {
        "cid": {
            "op": "Overwrite",
            "value": 2345
        },
        "startDate": {
            "op": "Overwrite",
            "value": "2001-01-01 08:00:00"
        }
    }
}

Fields for CDC Message with Type: Vertex

Field Description

vid

Internal vertex id, eg: 1234567890.

uid

External vertex id, eg: "abc123".

This could be primary_id/primary key/composite keys. And it will not appear in the content.

Possible Operators for CDC Message with Type: Vertex

“operator” Note

insert

Insert a new vertex, or modify some attribute values

insert-only

Insert new vertex if not existing, otherwise ignore it.

For now, this is only triggered in one case: when inserting an edge, the target vertex will be implicitly inserted with this “insert-only” operator.

delete

Delete a vertex will implicitly delete all edges that use that vertex as source or target.

TigerGraph CDC service will NOT produce CDC messages for such “implicit edge deletion”.

CDC Message with Type: Vertex-Type

A CDC message for a vertex type corresponds to a change of all instances of the given type. This is not a schema change.

The only CDC message type supported currently is for deletion operations. Other changes to all instances of a vertex type are not yet supported by CDC.
Example: Delete all instances of the vertex type Person.
{
    "mid": "1|1701548854014|2|0",
    "operator": "delete",
    "timestamp": 1701548854014,
    "type": "vertex-type",
    "graph": "SocialGraph",
    "typename": "Person",
    "content": {}
}

Possible operators for CDC Message with type vertex-type

The "operator" can only be "deleted":

“operator” Note

delete

Delete will delete all vertices of a vertex type and will also implicitly delete all edges that use that vertex as source or target.

TigerGraph CDC service will NOT produce CDC messages for such “implicit edge deletion”.

CDC Message with Type: Edge

A CDC message for an edge entity can describe an insertion, modification, or deletion.

The following example also illustrates updates to list, map, and tuple attibute types.

  1. The CDC format for a map has changed in v4.1.1. The previous format was

{ "A":1, "B":2, "C":3, "D":5 }

The new format is

{ "keylist":["A","B","C","D"], "valuelist":[1,2,3,5] }

2. The CDC format for a tuple is deprecated and will change in v4.2.
The new format is expected to separate the field names and their values into two lists, similar to how maps are now structured as two lists.
Example: Insert an edge betwen Person person2 and Company comp2.
{
    "mid": "1|1701569966031|3|0",
    "operator": "insert",
    "timestamp": 1701569966031,
    "type": "edge",
    "graph": "SocialGraph",
    "typename": "Creates",
    "hasreverseedge": true,
    "discriminator": "c1:1",
    "from": {
        "type": "Person",
        "vid": 12345678,
        "uid": "person2"
    },
    "to": {
        "type": "Company",
        "vid": 87654321,
        "uid": "comp2"
    },
    "content": {
        "attr_list": {
            "op": "Add",
            "value": [
                2,
                3,
                1222819200
            ]
        },
        "attr_map": {
            "op": "Add",
            "value": {
                "keylist":["i", "f", "b"],
                "valuelist": [2, 3 ,122281920]
            }
        }
        "attr_tuple": {
            "op": "Add",
            "value": [
                "i: 2, u: 0, f: 3, d: 3, dt: 1222819200, b: 0, s: m2"
            ]
        }
    }
}
Example: Delete all edges from Person 'John', including multi-edges.
{
    "mid": "1|1701548849757|1|0",
    "operator": "delete-all",
    "timestamp": 1701548849757,
    "type": "edge",
    "graph": "SocialGraph",
    "typename": "Transfer",
    "hasreverseedge": true,
    "from": {
        "type": "Person",
        "vid": 999999999,
        "uid": "John"
    }
}

Fields for CDC Message with Type: Edge

Field Optional? Note

from

No

JSON object with 3 fields: type: vertex type name of source vertex vid: internal vertex id of source vertex uid: external vertex id of source vertex. This could be primary_id/primary key/composite keys. For composite keys, the key is separated by a ,.

to

No

Same as from field, but for target vertex.

hasreverseedge

Yes

Bool type. When the value is true, it means the updated edge type has also reverse edge types. If it’s false this field will be omitted.

discriminator

Yes

String type. Only exists if the updated edge is multi-edge. It’s the discriminator string separated by a :.

Possible Operators for CDC Message with Type: Edge

“operator” Note

insert

Insert a new edge, or modified attribute values.

If source vertex is missing when inserting edge, TigerGraph will implicitly insert source vertex with all default values on its attributes.

For now, TigerGraph CDC Service will NOT produce CDC messages for such “implicit source vertex insertion”.

delete

Delete an edge

delete-all

Delete all edges from the given source vertex. Currently, the use of delete-all is quite restricted. See note below.
Restrictions for delete-all

A delete-all message must satisfy all of the following constraints:

  • The edge type being deleted:

    • must have only one target vertex type

    • may not be a multi-edge type

    • may not have a reverse edge type

  • The CDC messsage may not specify edge type or the target vertex.

Extra CDC message for Edge Update

Edge updates may include additional information in specific scenarios.

In such cases, TigerGraph CDC will generate an additional CDC message for the corresponding "extra" edge.

Case Description

For directed edge without reverse edge type

For insertion or modification on directed edge type without a reverse edge type, TigerGraph CDC will generate an extra CDC message with a field "operator": "insert-only" for a target vertex, however, there is no CDC message for source vertex.

For undirected edge, and directed edge with reverse edge type

For insertion/modification/deletion on an undirected edge, or directed edge with a reverse edge type, TigerGraph will update 2 edges simultaneously: the “origin” edge and the “extra” edge with switched source and target vertex.