Set Up Cross-Region Replication

To set up cross-region replication (CRR), you set up Disaster Recovery (DR) clusters that sync with the primary cluster. Changes on the primary cluster are copied over to the DR cluster.

When necessary, you can fail over to a DR cluster, making it the new primary cluster.

This setup assumes that you are setting up a DR cluster for an existing primary cluster. If you are setting up both the primary cluster and DR cluster from scratch, you only need perform Step 3 after TigerGraph is installed on both clusters.

1. Prerequsites

  • The DR cluster and the primary cluster are running the same version of TigerGraph (preferably 3.9.3 or higher).

  • The DR cluster and the primary cluster have the same partitioning factor.

  • You have access to the TigerGraph Linux user account on the DR cluster.

    • The username and password of the TigerGraph database user creates on the DR cluster during installation matches one of the superusers on the primary cluster.

  • Initializing the DR cluster requires running a restore operation from a recent backup of the primary cluster; therefore, backup and restore must be configured on the DR cluster.

    • [3.9.1 and earlier] The backup must have the metadata backup that goes with the data backup.

  • If you choose to enable CRR and your DR cluster is in a different Virtual Private Cloud (VPC) than your primary cluster, make sure that TigerGraph is installed on your cluster with public IPs:

Make sure TigerGraph is not installed with a local loopback IP such as 127.0.0.1. You can verify if you are using loopback IP with gadmin config get System.HostList if this returns 127.0.0.1 then it means you have installed TigerGraph with loopback IP

2. Procedure

The following setup is needed in order to enable Cross Region Replication.

2.1. Backup primary data

Use GBAR to create a backup of the primary cluster. See Backup and Restore on how to create a backup.

If you are setting up both the primary cluster and the DR cluster from scratch, you can skip Steps 1, 2, and 4 and only perform Step 3.

2.2. Restore on the DR cluster

Copy the backup files from every node to every node on the new cluster. Restore the backup of the primary cluster on the DR cluster. See Backup and Restore on how to restore a backup.

2.3. Enable CRR on the DR cluster

Run the following commands on the DR cluster to enable CRR on the DR cluster.

# Enable Kafka Mirrormaker
$ gadmin config set System.CrossRegionReplication.Enabled true

# Kafka mirrormaker primary cluster's IPs, separator by ','
$ gadmin config set System.CrossRegionReplication.PrimaryKafkaIPs <PRIMARY_IP1,PRIMARY_IP2,PRIMARY_IP3>

# Kafka mirrormaker primary cluster's KafkaPort
$ gadmin config set System.CrossRegionReplication.PrimaryKafkaPort 30002

# The prefix of GPE/GUI/GSQL Kafka Topic, by default is empty.
$ gadmin config set System.CrossRegionReplication.TopicPrefix Primary

# Apply the config changes, init Kafka, and restart
$ gadmin config apply -y
$ gadmin init kafka -y
$ gadmin restart all -y

2.4. Force install queries on primary

Run the INSTALL QUERY -force ALL command on the primary cluster. After the command is finished, all other metadata operations on the primary cluster will start syncing to the DR cluster.

3. Restrictions on the DR cluster

After being set up, the DR cluster will be read-only and all data update operations will be blocked. This includes the following operations:

  • All metadata operations

    • Schema changes

    • User access management operations

    • Query creation, installation, and dropping

    • User-defined function operations

  • Data-loading operations

    • Loading job operations

    • RESTPP calls that modify graph data

  • Queries that modify the graph

4. Sync an outdated DR cluster

When the primary cluster executes an IMPORT, DROP ALL, or CLEAR GRAPH STORE GSQL command, or the gsql --reset bash command, the services on the DR cluster will stop syncing with the primary and become outdated.

To bring an outdated cluster back in sync, you need to generate a fresh backup of the primary cluster, and perform the setup steps detailed on this page again. However, you can skip Step 3: Enable CRR on the DR cluster, because CRR will have already been enabled.

5. Updating a CRR system

From time to time, you may want to update the TigerGraph software on a CRR system. To perform this correctly, follow this sequence of steps.

  1. Disable CRR on your DR cluster.

    $ gadmin config set System.CrossRegionReplication.Enabled false
$ gadmin config apply -y
$ gadmin restart all -y

  2. Upgrade both the primary cluster and DR cluster.

  3. Enable CRR on the DR cluster.