Cluster Expansion

As your workload changes, you can expand your cluster to improve its query performance, system availability, and fault tolerance. Expanding a cluster adds more nodes to the cluster. During an expansion, you can also change the replication factor of your cluster.

Before you begin

  • TigerGraph must already be installed on the new nodes as single-node instances in exactly the same version as the cluster.

  • No loading jobs, queries, or REST requests are running on the new node or the cluster.

  • Obtain a few key measures for the state of your data before the expansion, such as vertex counts/edge counts or certain query results. This will be useful in verifying data integrity after the expansion completes.

  • If the original cluster is a single node installation, make sure the IP used is not a local loopback address such as 127.0.0.1.

Procedure

Step 1: Identify new cluster replication and partition

Before running any commands to expand a cluster, make sure you have a clear idea of how the new cluster should be distributed. You should have the following information:

  • The new replication factor of the cluster

  • The new partitioning factor of the cluster

  • The IP addresses of the new nodes to be added to the cluster

Step 2: Expand the cluster

To expand the cluster, run the gadmin cluster expand command like below. If the expansion involves changing the replication factor, use the --ha option to indicate the new replication factor:

$ gadmin cluster expand node_ip_list [--ha <replication_factor>]

node_ip_list is the machine aliases of the nodes you are adding to the cluster mapped to their IP addresses with a colon(:), and separated by a comma. Below is an example:

$ gadmin cluster expand m3:10.128.0.81,m4:10.128.0.82 --ha 1

We suggest naming the new nodes following the convention of m<serial>, such as m1, m2, and m3 for a 3-node cluster. If you are adding a fourth node, then the fourth node would be named m4. If you decide to name them differently, make sure that all names are unique within the cluster.

Supply a staging location

Extra disk space is required during cluster expansion. If more space is not available on the same disk, you can supply a staging location on a different disk to hold temporary data:

$ gadmin cluster expand m3:192.168.1.3,m4:192.168.1.4 --stagingPath /tmp/

If you choose to supply a staging location, make sure that the TigerGraph Linux user has write permission to the path you provide. The overall amount of space required for expansion on each node is (1 + Math.ceil(oldPartition/newPartition) ) * dataRootSize. oldPartition and newPartition each stand for the partitioning factor of the cluster before and after expansion; dataRootSize stands for the size of the data root folder on the node.

For example, if you are expanding from a 6 node cluster with a replication factor of 2 and a partitioning factor of 3, to a 10-node cluster with a replication factor of 2 and a partitioning factor of 5, and the size of the data root folder on a node is 50GB. Then you would need more than (1 + Math.ceil(3/5)) * 50) = 100 GB of free space on the staging path.

Step 3: Verify success and delete temporary files

When the expansion completes, you should see a message confirming the completion of the cluster change. The message will also include the location of the temporary files created during the expansion.

Verify data integrity by comparing vertex/edge counts or query results. After confirming a successful expansion, delete the temporary files to free up disk space.