Admin Portal, gamin utility, GBAR backup and restore
Loading...
Loading...
Loading...
Loading...
Loading...
Managing TigerGraph Servers with gadmin
TigerGraph Graph Administrator (gadmin) is a tool for managing TigerGraph servers. It has a self-contained help function and a man page, whose output is shown below for reference. If you are unfamiliar with the TigerGraph servers, please see GET STARTED with TigerGraph.
To see a listing of all the options or commands available for gadmin, run any of the following commands:
After changing a configuration setting, it is generally necessary to run gadmin config apply
. Some commands invoke config apply automatically. If you are not certain, just run
gadmin config apply
Below is the man page for gadmin. Most of the commands are self-explanatory. Common examples are provided with each command.
NOTE: Some commands have changed in v3.0. In particular,
gadmin set <config | license>
has changed to
gadmin <config | license> set
Gadmin autocomplete is more of a feature than a command. It is an auto-complete feature that allows you to see all possible entries of a specific configuration. You can press tab when typing a command to either print out all possible entries, or auto-complete the entry you are currently typing.
The example below shows an example of the autocomplete for the command gadmin status
.
gadmin config
commands are used to manage the configuration for the TigerGraph system. To get a complete list of configuration parameters that are available, see Configuration Parameters.
gadmin config
has many sub-entries as well, they will be listed below.
Example: Change the retention size of the kafka queue to 10GB:
Show what configuration changes were made.
Discard the configuration changes without applying them.
Display all configuration entries.
Change a configuration entry.
Get the value of a specific configuration entry.
Configure entries for a specific service group. e.g. KAFKA, GPE, ZK
Initialize your configuration.
List all configurable entries or entry groups.
Options for configuring your license.
To generate a license seed, use the following command:
Depending on your host machine, you need to choose the appropriate host signature type. If you are generating the seed from a cloud instance, choose the corresponding cloud provider as your signature type.
If you are generating the seed from your own machine, choose either node-id
or hardware
.
The hardware
option tells gadmin
to collect information from your machine's hardware as the host signature to generate the license seed. A signature produced by using this parameter will not be altered by software changes on the machine, including OS reinstalls. This is the usual choice.
node-id
refers to the machine ID in the machine-id
file located at /etc/machine-id
and is a unique signature for the OS that identifies your machine. A reinstall of the OS may change the machine ID.
Example flow for applying a new license (which may be replacing an existing license key):
Once the license has been set and config has been applied, you can run gadmin license status
to view the details of your license, including the expiration date and time.
The gadmin log
command will reveal the location of all commonly checked log files for the TigerGraph system.
The gadmin restart
command is used to restart one, many, or all TigerGraph services. You will need to confirm the restarting of services by either entering y (yes) or n (no). To bypass this prompt, you can use the -y flag to force confirmation.
The gadmin start
command can be used to start one, many, or all services.
Check the status of TigerGraph component servers:
Use gadmin status
to report whether each of the main component servers is running (up) or stopped (off). The example below shows the normal status when the graph store is empty and a graph schema has not been defined:
You can also check the status of each instance using the verbose flag : gadmin status -v
or gadmin status --verbose
. This will show each machine's status. See example below
Here are the most common service and process status states you might see from running the gadmin status
command :
Online - The service is online and ready.
Warmup - The service is processing the graph information and will be online soon.
Stopping - The service has received a stop command and will be down soon.
Offline - The service is not available.
Down - The service has been stopped or crashed.
StatusUnknown - The valid status of the service is not tracked.
Init - Process is initializing and will be in the running state soon.
Running - The process is running and available.
Zombie - There is a leftover process from a previous instance.
Stopped - The process has been stopped or crashed.
StatusUnknown - The valid status of the process is not tracked.
The gadmin stop command can be used to stop one, many, or all TigerGraph services. You will need to confirm the restarting of services by either entering y (yes) or n (no). To bypass this prompt, you can use the -y flag to force confirmation.
TigerGraph offers two levels of memory thresholds using the following configuration settings:
SysAlertFreePct and SysMinFreePct
SysAlertFreePct setting indicates that the memory usage has crossed a threshold where the system will start throttling Queries to allow long-running queries to finish and release the memory.
SysMinFreePct setting indicates that the memory usage has crossed a critical threshold and the Queries will start aborting automatically to prevent GPE crash and system stability.
By default, SysMinFreePct is set at 10%, at which point Queries will be aborted.​
Example:
SysAlertFreePct=30 means when the system memory consumption is over 70% of the memory, the system will enter an alert state, and Graph updates will start to slow down.
SysMinFreePct=20 means 20% of the memory is required to be free. When memory consumption enters a critical state (over 80% memory consumption) queries will be aborted. automatically.
Follow the steps documented in this support article to update the Nginx configurations of your TigerGraph instance.
GSQL restricts where a query can produce output to files through a file output policy. The policy consists of a whitelist and a blacklist.
GSQL queries must only output to the directories and their descendants or the files indicated by paths in the whitelist.
GSQL queries cannot output to the directories and their descendants or the files indicated by paths in the blacklist. The blacklist takes precedence over the whitelist.
By default, the file output policy allows outputs to all files.
GSQL.FileOutputPolicy
The file output policy is implemented through the system configuration parameterGSQL.FileOutputPolicy
, which is a JSON array of strings that represents a list of paths. If there is an exclamation mark (!
) preceding a path, the path is on the blacklist. If there is no exclamation mark preceding a path, the path is on the whitelist.
For example, if the value for GSQL.FileOutputPolicy
is ["/home/tigergraph", "!/home/tigergraph/documents", "!/home/tigergraph/desktop"]
, then below are the paths on the white list and on the black list:
Whitelist: /home/tigergraph
and all its descendants
Blacklist: /home/tigergraph/documents, /home/tigergraph/desktop
and all their descendants.
Since the blacklist takes precedence, GSQL will allow queries to write to all files and directories under /home/tigergraph
except the documents
and destktop
folders.
To edit the file policy, ensure that you are logged in as the TigerGraph Linux user, and run the following command:
In the prompt, enter the new value for the parameter:
Apply the new configurations and restart GSQL
After implementing the file output policy, queries that write to paths that are not on the whitelist are forbidden:
If a FILE
object is defined with an empty string, it is regarded as a null file. The file output policy will not block the definition of the FILE
object, but writing to a null file would cause a runtime error.
Additionally, queries that write to paths on the whitelist, but also on the blacklist are also forbidden:
This page documents a list of advanced Linux commands that simplify platform operations that are performed often during debugging, especially on high availability (HA) clusters. Only the TigerGraph platform owner - the Linux user created during installation has access to the commands on this page.
Users are advised to use these commands only at the guidance and recommendation of TigerGraph support.
This command allows you to connect to another node in your cluster via SSH.
With huge data volumes, data loading can be time-consuming. If you find yourself often loading huge volumes of data into an empty graph, and your data volume is so large that your loading jobs are taking hours to complete, you might consider using offline loading to speed up data loading.
In order to use offline loading, all the filename variables in the loading job must take an initial path value. After creating the loading job and ensuring that all the data files are referenced correctly in the loading job, use the options -g
and -j
to specify the graph and loading job to run. During offline loading, your database is focused on loading data and will not be able to handle requests and queries.
Offline loading deletes all existing graph data before it starts. Back up your data before using offline loading.
-g <graph_name>
: Name of the graph whose loading job to run
-j <loading_job_name>
: Name of the loading job to run
The following command runs the loading job load_ldbc_snb
on the graph ldbc_snb
:
You can also provide the graph name and the loading job name with a config file written in Bash:
Once you have the config file, you can run gautoloading.sh
with the config file instead of the -g
and -j
options:
This command allows you to copy files from the current node to target folders on multiple nodes at the same time. The file or directory on the current node specified by the source path will be copied into the target folder on every node. If the target folder does not exist at the path given, the target folder will be created automatically. You can also specify multiple source files or directories, in which case, the source paths need to be absolute paths, put in quotes, and separated by space.
You can specify the nodes where you want the copy operation to occur in the following ways:
gscp all <source_path> <target_dir>
will execute the command on all nodes
gscp <component_name> <source_path> <target_dir>
will execute the command on nodes where the component you specified is running
gscp <node_list> <source_path> <target_dir>
will execute the command on the nodes you specify in the node list
This command downloads a file or directory from every specified node to the target directory on the current node.
This command allows you to run commands on a specified list of nodes in your cluster one by one, and the output from every node will be visible to the terminal. grun
will wait for the command to finish running on one node before executing the command on the next node.
You can specify which nodes to run commands on in the following ways:
grun all '<command>'
will execute the command on all nodes
grun <component_name> '<command>'
will execute the command on nodes where the component you specified is running
grun <node_list> '<command>'
will execute the commands on the nodes you specify in the node list
This command allows you to run commands on a specified list of nodes in your cluster in parallel, and the output will be visible to the terminal where the grun_p
command was run. You can specify the nodes to run commands on in the following ways:
grun_p all '<command>'
will execute the command on all nodes
grun_p <component_name> '<command>'
will execute the command on nodes where the component you specified is running
grun_p <node_list> '<command>'
will execute the commands on the nodes you specify in the node list. The list of nodes should be separated by a comma, e.g.: m1,m2
This command returns the private IP address of your current node.
This command returns your current node number as well as all servers that are running on the current node.
In this example, m1
is the current node number, and ADMIN#1
, admin#1
etc. are all servers that are running on m1
.
The gssh
command, when used without arguments, outputs information about server deployments in your cluster. The output contains the names and IP addresses of every node. For each node, the output shows the full list of servers that are running on the node. For each server, the output shows the full list of the nodes that the server is running on.
This command returns the size of your data, the number of existing vertices and edges, as well as the number of deleted and skipped vertices on every node in your cluster. If you are running TigerGraph on a single node, it will return the same information that one node.
Export/Import is a complement to Backup/Restore, not a substitute.
The GSQL EXPORT
and IMPORT
commands perform a logical backup and restore. A database export contains the database's data, and optionally some types of metadata, which can be subsequently imported in order to recreate the same database, in the original or in a different TigerGraph platform instance.
To import an exported database, ensure that the export files are from a database that was running the exact same version of TigerGraph as the database that you are importing into.
Known Issues (Updated Feb 16th):
User-defined loading jobs containingDELETE
statements are not exported correctly.
If a graph contains vertex or edge types with a composite key, the graph data is exported in a nonstandard format that cannot be reimported.
The EXPORT GRAPH ALL
command reads the data and metadata for all graphs in the TigerGraph system and writes the information to a zip file in the designated folder. If no options are specified, then a full backup is performed, including schema, data, template information, and user profiles.
EXPORT_GRAPH
The export directory should be empty before running the command because all contents are zipped and compressed.
The EXPORT GRAPH
command exports all graphs in the database.
The export contains four categories of files:
Data files in CSV format, one file for each type of vertex and each type of edge.
GSQL DDL command files created by the export command. The import command uses these files to recreate the graph schema(s) and reload the data.
Copies of the database's queries, loading jobs, and user-defined functions.
GSQL command files used to recreate the users and their privileges.
The following files are created in the specified directory when exporting and are then zipped into a single file called ExportedGraphs.zip.
If the file is password-protected, it can only be unzipped using GSQL IMPORT. The security feature prevents users from directly unzipping it.
A DBImportExport_<graphName>.gsql for each graph called <graphName> in a multigraph database that contains a series of GSQL DDL statements that do the following:
Create the exported graph, along with its local vertex, edge, and tuple types,
Create the loading jobs from the exported graphs
Create data source file objects
Create queries
A graph_<graphName>/ folder for each graph in a multigraph database containing data for local vertex/edge types in <graphName>. For each vertex or edge type called <type>, there is one of the following two data files:
vertex_<type>.csv
edge_<type>.csv
global.gsql - DDL job to create all global vertex and edge types, and data sources.
tuple.gsql - DDL job to create all User Defined Tuples.
Exported data and jobs used to restore the data:
GlobalTypes/ - folder containing data for global vertex/edge types
vertex_name.csv
edge_name.csv
run_loading_jobs.gsql - DDL created by the export command which will be used during import:
Temporary global schema change job to add user-defined indexes. This schema job is dropped after it is has run.
Loading jobs to load data for global and local vertex/edges.
Database's saved queries, loading jobs, and schema change jobs
SchemaChangeJob/ - folder containing DDL for schema change jobs. See section "Schema Change Jobs" for more information
Global_Schema_Change_Jobs.gsql contains all global schema change jobs
graphName_Schema_Change_Jobs.gsql contains schema change jobs for each graph "graphName"
Tokenbank.cpp - copy of <tigergraph.root.dir>/app/<VERSION_NUM>/dev/gdk/gsql/src/TokenBank/TokenBank.cpp
ExprFunctions.hpp - copy of <tigergraph.root.dir>/app/<VERSION_NUM>dev/gdk/gsql/src/QueryUdf/ExprFunctions.hpp
ExprUtil.hpp - copy of <tigergraph.root.dir>/app/<VERSION_NUM>/dev/gdk/gsql/src/QueryUdf/ExprUtil.hpp
Users:
users.gsql - DDL to create all exported users and import Secrets and Tokens, and grant permissions.
If not enough disk space is available for the data to be exported, the system returns an error message indicating not all data has been exported. Some data may have already been written to disk. If an insufficient disk error occurs, the files will not be zipped, due to the possibility of corrupted data which would then corrupt the zip file. The user should clear enough disk space, including deleting the partially exported data, before reattempting the export.
It is possible for all the files to be written to disk and then to run out of disk space during the zip operation. If that is the case, the system will report this error. The unzipped files will be present in the specified export directory.
If the timeout limit is reached during export, the system returns an error message indicating not all data has been exported. Some data may have already been written to disk. If a timeout error occurs, the files will not be zipped. The user should delete the export files, increase the timeout limit and then rerun the export.
The timeout limit is controlled by the session parameter export_timeout. The default timeout is ~138 hours. To change the timeout limit, use the command:
The IMPORT GRAPH ALL
command unzips the .zip
file ExportedGraph.zip
located in the designated folder, unzips it, and then runs the GSQL command files within to import the graph.
WRITE_SCHEMA
, WRITE_QUERY
, WRITE_LOADINGJOB
, EXECUTE_LOADINGJOB
, DROP ALL
, WRITE_USERS
IMPORT GRAPH
looks for specific filenames. If either the zip file or any of its contents are renamed by the user, IMPORT GRAPH may fail.
IMPORT GRAPH
erases the current database (equivalent to running DROP ALL). The current version does not support incremental or supplemental changes to an existing database (except for the --keep-users option)
There are two sets of loading jobs:
Those that were in the catalog of the database which was exported. These are embedded in the file DBImportExport_graphName.gsql
Those that are created by EXPORT GRAPH and are used to assist with the import process. These are embedded in the file run_loading_jobs,gsql.
The catalog loading jobs are not needed to restore the data. They are included for archival purposes.
Some special rules apply to importing loading jobs. Some catalog loading jobs will not be imported.
If a catalog loading job contains DEFINE FILENAME F = "/path/to/file/"
, the path will be removed and the imported loading job will only contain DEFINE FILENAME F
.
This is to allow a loading job to still be imported even though the file may no longer exist or the path may be different due to moving to another TigerGraph instance.
If a specific file path is used directly in the LOAD statement, and the file cannot be found, the loading job cannot be created and will be skipped.
For example, LOAD "/path/to/file" to vertex v1
cannot be created if /path/to/file
does not exist.
Any file path using $sys.data_root
will be skipped.
This is because the value of $sys.data_root
is not retained from export. During import, $sys.data_root
is set to the root folder of the import location.
There are two sets of schema change jobs:
Those that were in the catalog of the database which was exported. These are stored in the folder /SchemaChangeJobs.
Those that were created by EXPORT GRAPH and are used to assist with the import process. These are in the run_loading_jobs.gsql command file. The jobs are dropped after the import command is finished with them.
The database's schema change jobs are not executed during the import process. This is because if a schema change job had been run before the export, then the exported schema already reflects the result of the schema change job. The directory /SchemaChangeJobs contains these files:
Global_Schema_Change_Jobs.gsql contains all global schema change jobs
<graphName>_Schema_Change_Jobs.gsql contains schema change jobs for each graph <graphName>.
In the current version, importing and exporting clusters is not fully automated. The database can be exported and imported by following some additional steps.
Rather than creating a single export zip file, export will create a file for each machine. Before exporting, specific folders must be created on each server using the following commands:
Then run the export command on one server. The EXPORT command does not bundle all the files to one server, and it does not compress each server's files to one zip. Some files, including the data files, will be exported to each server, to the folders created above. Some files will be only on the local server where EXPORT GRAPH was run.
You may only import to a cluster that has the same number and configuration of servers as the data from which the export originated. Transfer the files from one export server to a corresponding import server. That is, copy the files from
export_server_n:/path/to/export_directory
to
import_server_n:/path/to/import/directory
2. Manually modify the loading jobs
On the main server, edit the run_loading_jobs.gsql files as follows.
Find the line(s) of the form:
LOAD "sys.data_root/.../<vertex_or_edge_type>.csv"
Close to it should be similar line that is commented out which have the "all:" data source directive:
#LOAD "all:sys.data_root/.../<vertex_or_edge_type>.csv"
See the example below:
Comment out the LOAD line and uncomment the LOAD all: line. Be sure that you do this for all data source files.
3. Run the IMPORT GRAPH command from the main server (e.g., the one that corresponds to the server where EXPORT GRAPH was run).
GBAR - Graph Backup and Restore
Graph Backup And Restore (GBAR), is an integrated tool for backing up and restoring the data and data dictionary (schema, loading jobs, and queries) of a TigerGraph instance or cluster.
The backup feature packs TigerGraph data and configuration information into a directory on the local disk or a remote AWS S3 bucket. Multiple backup files can be archived. Later, you can use the restore feature to roll back the system to any backup point. This tool can also be integrated easily with Linux cron to perform periodic backup jobs.
The current version of GBAR is intended for restoring the same machine that was backed up. For help with cloning a database (i.e., backing up machine A and restoring the database to machine B), please contact support@tigergraph.com.
The -y
option forces GBAR to skip interactive prompt questions by selecting the default answer. There is currently one interactive question:
At the start of restore, GBAR will always ask if it is okay to stop and reset the TigerGraph services: (y/N)? The default answer is yes.
Before using the backup or the restore feature, GBAR must be configured.
Run gadmin config entry system.backup
. At each prompt, enter the appropriate values for each config parameter.
After entering the configuration values, run the following command to apply the new configurations
Note:
You can specify the number of parallel processes for backup and restore.
You must provide username and password using GSQL_USERNAME and GSQL_PASSWORD environment variables.
To perform a backup, run the following command as the TigerGraph Linux user:
Depending on your configuration settings, your backup archive will be output to your local backup path and/or your AWS S3 bucket. If you are running a cluster, there will be a backup archive on every node in the same path.
A backup archive is stored as several files in a folder, rather than as a single file. The backup tag acts like a filename prefix for the archive filename. The full name of the backup archive will be <backup_tag>-<timestamp>
, which is a subfolder of the backup repository.
If System.Backup.Local.Enable
is set to true
, the folder is a local folder on every node in a cluster, to avoid massive data moving across nodes in a cluster.
If System.Backup.S3.Enable
is set to true
, every node will upload data located on the node to the s3 repository. Therefore, every node in a cluster needs access to Amazon S3.
GBAR Backup performs a live backup, meaning that normal operations may continue while the backup is in progress. When GBAR backup starts, GBAR will check if there are running loading jobs. If there are, it will pause loading for 1 minute to generate a snapshot and then continue the backup process. You can specify the loading pausing interval by the environment variable PAUSE_LOADING
.
GBAR then sends a request to the admin server, which then requests the GPE and GSE to create snapshots of their data. Per the request, the GPE and GSE store their data under GBAR’s own working directory. GBAR also directly contacts the Dictionary and obtains a dump of its system configuration information. In addition, GBAR gathers the TigerGraph system version and customized information including user-defined functions, token functions, schema layouts and user-uploaded icons. Then, GBAR compresses each of these data and configuration information files in tgz format and stores them in the <backup_tag>-<timestamp>
subfolder on each node. As the last step, GBAR copies that file to local storage or AWS S3, according to the Config settings, and removes all temporary files generated during backup.
The current version of GBAR Backup takes snapshots quickly to make it very likely that all the components (GPE, GSE, and Dictionary) are in a consistent state, but it does not fully guarantee consistency.
Backup does not save input message queues for REST++ or Kafka.
This command lists all generated backup files in the storage place configured by the user. For each file, it shows the file’s full tag, its size in human-readable format, and its creation time.
Before restoring a backup, you should ensure that the backup you are restoring from is in the same exact version as your current version of TigerGraph.
To restore a backup, run the following command:
If GBAR can verify that the backup archive exists and that the backup's system version is compatible with the current system version, GBAR will shut down the TigerGraph servers temporarily as it restores the backup. After completing the restore, GBAR will restart the TigerGraph servers. If you are running a cluster, and you have copied the backup files to each individual node in the cluster, running gbar restore
on any node will restore the entire cluster.
Restore is an offline operation, requiring the data services to be temporarily shut down. The user must specify the full archive name ( <backup_tag>-<timestamp>
) to be restored. When GBAR restore begins, it first searches for a backup archive exactly matching the archive name supplied in the command line. Then it decompresses the backup files to a working directory. Next, GBAR will compare the TigerGraph system version in the backup archive with the current system's version, to make sure that the backup archive is compatible with that current system. It will then shut down the TigerGraph servers (GSE, RESTPP, etc.) temporarily. Then, GBAR makes a copy of the current graph data, as a precaution. Next, GBAR copies the backup graph data into the GPE and GSE and notifies the Dictionary to load the configuration data. Also, GBAR will notify the GST to load backup user data and copy the backup user-defined token/functions to the right location. When these actions are all done, GBAR will restart the TigerGraph servers.
Note: GBAR restore does not estimate the uncompressed data size and check whether there is sufficient disk space.
The primary purpose of GBAR is to save snapshots of the data configuration of a TigerGraph system, so that in the future the same system can be rolled back (restored) to one of the saved states. A key assumption is that Backup and Restore are performed on the same machine, and that the file structure of the TigerGraph software has not changed.
Restore needs enough free space to accommodate both the old gstore and the gstore to be restored.
To remove a backup, run the gbar remove
command:
The command removes a backup from the backup storage path. To retrieve the tag of a backup, you can use the gbar list
command.
Run gbar cleanup
to delete the temporary files created during backup or restore operations:
The following example describes a real example, to show the actual commands, the expected output, and the amount of time and disk space used, for a given set of graph data. For this example, an Amazon EC2 instance was used, with the following specifications:
Single instance with 32 vCPU + 244GB memory + 2TB HDD.
Naturally, backup and restore time will vary depending on the hardware used.
To run a daily backup, we tell GBAR to backup with the tag name daily.
The total backup process took about 31 minutes, and the generated archive is about 49 GB. Dumping the GPE + GSE data to disk took 12 minutes. Compressing the files took another 20 minutes.
To restore from a backup archive, a full archive name needs to be provided, such as daily-20180607232159. By default, restore will ask the user to approve to continue. If you want to pre-approve these actions, use the "-y" option. GBAR will make the default choice for you.
For our test, GBAR restore took about 23 minutes. Most of the time (20 minutes) was spent decompressing the backup archive.
Note that after the restore is done, GBAR informs you were the pre-restore graph data (gstore) has been saved. After you have verified that the restore was successful, you may want to delete the old gstore files to free up disk space.
Parameter
Description
directory_name
The path of the directory to output export files to. Must be an absolute path.
Option
Description
-S
or --SCHEMA
Only export graph schema.
-T
or --TEMPLATE
Only export graph schema, queries, loading jobs and UDFs.
-D
or --DATA
Must be used with either -S
or -T
. Export data in CSV in addition to graph schema or template.
-U
or --Users
Must be used with either -S
or -T
. Export users, role assginments, secrets, and tokens.
-P
or --Password
Encrypt the exported file with a password. Users will be prompted to enter a password when using this option.
Parameter
Description
filename
The path to the zip file produced by the EXPORT GRAPH ALL
command. Must be an absolute path.
Option
Description
-P
or --PASSWORD
Decrypt with password. You will be prompted to enter a password when using this option.
-KU
or --KEEP-USERS
Keep the current users for during the import operation. New users from the imported graph will still be added. If there are users from the same name, GSQL will report an error when importing the users, and the current users will be kept.
GStore size
Backup file size
Backup time
Restore time
219GB
49GB
31 mins
23 mins