Workload Management

This page explains the different methods available to manage system workloads in TigerGraph.

In distributed or replicated clusters, users have multiple processing units at their disposal. Depending on their needs, they can configure query routing schemes such as round-robin or CPU load-aware routing. For replicated clusters, queries can also be directed to specific replicas.

Workload Queues

Workload Queues help manage queries by routing them to predefined queues with specific configurations. Each queue can have properties like the maximum number of concurrent queries and delay queue size, preventing system overload. Queries can be assigned to queues based on user roles and permissions.

In 3.10, workload queues were introduced. Multiple independent queues can be configured, allowing application designers to decide how specific operations are handled.

Workload queues manage the following types of operations:

  • Running an installed query.

  • Interpreting ad-hoc queries.

  • Executing heavy built-in queries, commonly used for "Explore Graph" in GraphStudio.

Configure Workload Queues

Workload queues help control how queries are executed within the system by defining limits such as the maximum number of concurrent queries and the maximum size of the delay queue. Administrators can configure workload queues by creating, modifying, or deleting them as needed.

Each queue has the following properties:

Field

Description

Data type

description

The description of the queue.

STRING (< 256 characters)

isDefault (optional)

The flag to indicate if the queue is the default queue. Must be set to true for exactly one queue.

BOOL

maxConcurrentQueries

The maximum number of concurrent queries allowed in the queue.

UINT in the range of (0, 131072)

maxDelayQueueSize

The maximum number of queries that can be queued in the delay queue.

UINT in the range of [0, 131072)

maxConcurrentQueries and maxDelayQueueSize

maxConcurrentQueries and maxDelayQueueSize are enforced on per machine level. More specifically, it puts a limit on how many requests ONE GPE process can handle. For example, in a TigerGraph cluster with 4 nodes (there will be 4 GPE processes), the total number of qureies allowed for a WorkloadQueue is 4*maxConcurrentQueries. Similarly, the total number of queries can be put into the corresponding delay queue is 4*maxDelayQueueSize. TigerGraph internally would try to evenly distribute queries evenly among the nodes, hence, the WorkloadQueue from each GPE would be filled at a similar pace.

Before making changes, it’s a good practice to retrieve the current configuration using the GET Workload Queue command. After reviewing, you can update or apply new configurations using the PUT Workload Queue command. These commands are available both as GSQL commands and as REST endpoints.

GSQL Commands

  • Retrieve Current Configuration:

GET WORKLOAD QUEUE

  • Update Configuration:

PUT WORKLOAD QUEUE FROM "/path/to/queue.json"

REST Endpoints

  • Retrieve Current Configuration:

GET /gsqlserver/gsql/workload-manager/configs

  • Configure Workload Queues

POST /gsqlserver/gsql/workload-manager/configs

Examples

  1. Retrieve Current Configuration:

curl -X GET -u tigergraph:tigergraph \
  <hostname>:<nginx-port>/gsqlserver/gsql/workload-manager/configs

  1. Modify Workload Queue Configuration:

curl -X POST -u tigergraph:tigergraph \
<hostname>:<nginx-port>/gsqlserver/gsql/workload-manager/configs \
-d '{"isEnabled":true,"queues":{"OLTP":{"description":"OLTP queries","isDefault":false,"maxConcurrentQueries":100,"maxDelayQueueSize":200}}}'

Response Status Codes

For GET Workload Queue

Status Code

Description

200

The queue configurations were successfully retrieved.

403

The user does not have the READ_WORKLOAD_QUEUE privilege.

500

Server error occurred while processing the request.

For PUT Workload Queue

Status Code

Description

200

The queue configurations were successfully uploaded.

400

The payload is ill-formed.

403

The user does not have the WRITE_WORKLOAD_QUEUE privilege.

409

Conflict in the configuration, such as multiple queues set as default.

500

Server error occurred while processing the request.

maxConcurrentQueries and maxDelayQueueSize Each workload queue has two critical properties to manage system resources:

  1. maxConcurrentQueries: Defines the maximum number of queries that can run simultaneously in a queue. This helps prevent resource contention and ensures that the system does not become overloaded.

  2. maxDelayQueueSize: Specifies the maximum number of queries that can wait in a queue when the current workload exceeds the concurrent query limit. If this limit is reached, new queries are rejected until the queue has capacity.

Both properties are enforced at the per-machine level in a cluster. Setting the values too high may degrade performance. It’s recommended to keep maxConcurrentQueries below three times the number of physical CPU cores in a machine.

Any configuration changes require a GPE restart to take effect.

List Workload Queues

Displays a list of all workload queues available to the current user, including their configurations and permissions.

GSQL Command

LIST WORKLOAD QUEUE

REST Endpoints

GET /restpp/workload-manager/queue

Example Request

curl -X GET -u tigergraph:tigergraph \
  <hostname>:<nginx-port>/restpp/workload-manager/queue

Example Response The response will include the information available to the general users.

[
  {
    "id": "AdHoc",
    "description": "Ad-hoc queries",
    "isDefault": true
  },
  {
    "id": "OLTP",
    "description": "OLTP queries"
  }
]

Response Status Codes

Status Code Description

200

The queue info has been retrieved successfully.

403

The user doesn’t have the privilege READ_DATA.

Grant/Revoke Workload Queue Access

You can grant or revoke workload queues to a user based on its user name, groups, and/or roles.

GSQL Command

# GRANT
GRANT WORKLOAD QUEUE <queue_name> TO USER <user1>, <user2>

# REVOKE
REVOKE WORKLOAD QUEUE <queue_name> FROM USER <user1>, <user2>

REST Endpoint

POST /gsqlserver/gsql/workload-manager/permission

Request Body The request body expects a JSON object with the following schema:

{
  "OLTP": {
    "granted": {
      "USER": []
      "GROUP": ["*"]
      "ROLE": ["r1", "r2"]
    }
  }
}

The request body must have the following fields at the top level:

Field

Description

Data type

action

GRANT or REVOKE (case insensitive)

STRING

queue

The ID of the queue to be granted or revoked.

STRING

user (optional)

The list of the user names to be granted/revoked.

STRING or STRING[]

group (optional)

The list of the group names to be granted/revoked.

STRING or STRING[]

role (optional)

The list of the role names to be granted/revoked.

STRING or STRING[]

  • You can use the wildcard " * " to grant/revoke the queue to all users, groups, or roles.

  • Note that " * " must be the only entry in the list when available.

Example Request

Grant the queue OLTP to the user u1 and u2:

curl -X GET -u tigergraph:tigergraph \
  <hostname>:<nginx-port>/gsqlserver/gsql/workload-manager/permission \
  -d '{"action": "grant", "queue": "OLTP", "user": ["u1", "u2"]}'

Revoke the queue scheduled_jobs from all users and the role r1:

curl -X GET -u tigergraph:tigergraph \
  <hostname>:<nginx-port>/gsqlserver/gsql/workload-manager/permission \
  -d '{"action": "REVOKE" "queue": "scheduled_jobs", "user": "*", role": ["r1"]}'

Response Status Codes

Status Code

Description

200

The queue has been granted/revoked successfully.

400

The payload is ill-formed so none of the given entities could be granted/revoked.

403

The user doesn’t have the privilege WRITE_WORKLOAD_QUEUE

Unlike REST API, the GSQL commands don’t allow you to specify USER, GROUP, and ROLE in a command. You must use separate commands for each entity type.

Show Workload Queue Permissions

The SHOW WORKLOAD QUEUE command lists detailed information about workload queues, including their permissions, descriptions, and limits. It is primarily used to inspect queue settings and permissions.

GSQL Command To show the permission info of all queues:

SHOW WORKLOAD QUEUE

To show the permission info of a specific queue, for example OLTP:

SHOW WORKLOAD QUEUE OLTP

REST Endpoint

GET /gsqlserver/gsql/workload-manager/permission

Example Request To retrieve the permission info of the queue OLTP:

curl -X GET -u tigergraph:tigergraph \
  localhost:14240/gsql/v1/workload-manager/permission?id=OLTP

Example Response

The response will be the combination of configs and permission, e.g.

{
  "OLTP": {
    "description": "OLTP queries",
    "isDefault": false,
    "maxConcurrentQueries": 100,
    "maxDelayQueueSize": 200,
    "granted": {
      "USER": [],
      "GROUP": ["*"],
      "ROLE": ["r1", "r2"]
    }
  }
}

Response Status Codes

Status Code Description

200

The queue info has been retrieved successfully.

403

The user doesn’t have the privilege READ_WORKLOAD_QUEUE.

  • Use SHOW WORKLOAD QUEUE to inspect queue configurations and access permissions.

  • This command focuses on visibility of queue settings, unlike GET WORKLOAD QUEUE, which exports configurations.

Check Queue Status

You can use the following API to check the status of the workload queues for monitoring purposes.

POST /restpp/workload-manager/queuestatus

Return the status of the given workload queue on each GPE instance.

Request Body

Field

Description

Data type

queuelist (optional)

The list of the ID of the WorkloadQueue. If not specified, all queues will be shown.

STRING[]

mode (optional)

stats or verbose (case-sensitive). If not specified, stats will be used.

STRING

For mode field, if stats is specified, response only gives the numbers of queries waiting and running. If verbose is specified, the response will include the the request Ids of the queries that are waiting and running.

If Request Body is not provided, response is generated as if both fields are using the default values.

Example Request

curl -X POST -u tigergraph:tigergraph \
  <hostname>:<nginx-port>/restpp/workload-manager/queuestatus \
   -d '{"queuelist": ["AdHoc"], "mode": "verbose"}'

Example Response

{
  "version": {
    "edition": "enterprise",
    "api": "v2",
    "schema": 0
  },
  "error": false,
  "message": "Completes",
  "WorkloadQueueStatusByInstances": [
    {
      "version": {
        "edition": "enterprise",
        "api": "v2",
        "schema": 0
      },
      "error": false,
      "message": "",
      "results": {
        "GPE_2_1": [
          {
            "WorkloadQueueName": "AdHoc",
            "maxConcurrentQueries": 1,
            "maxDelayQueueSize": 2,
            "runningQueries": [
              "196702.RESTPP_1_1.1707799387957.N"
            ],
            "delayQueries": [
              "65630.RESTPP_1_1.1707799387958.N"
            ]
          }
        ]
      }
    },
    {
      "version": {
        "edition": "enterprise",
        "api": "v2",
        "schema": 0
      },
      "error": false,
      "message": "",
      "results": {
        "GPE_1_1": [
          {
            "WorkloadQueueName": "AdHoc",
            "maxConcurrentQueries": 1,
            "maxDelayQueueSize": 2,
            "runningQueries": [
              "94.RESTPP_1_1.1707799387957.N"
            ],
            "delayQueries": [
              "131167.RESTPP_1_1.1707799387959.N"
            ]
          }
        ]
      }
    }
  ],
  "code": "REST-0000"
}

Use Cases

Suppose we have configured the following workload queues that are the output of the SHOW WORKLOAD QUEUE command:

{
  "OLTP": {
    "description": "OLTP queries",
    "isDefault": true,
    "maxConcurrentQueries": 100,
    "maxDelayQueueSize": 100,
    "granted": {
      "USER": [],
      "GROUP": ["g1", "g2"],
      "ROLE": []
    }
  },
  "scheduled_jobs": {
    "description": "Scheduled jobs",
    "maxConcurrentQueries": 5,
    "maxDelayQueueSize": 0,
    "granted": {
      "USER": ["u1"],
      "GROUP": [],
      "ROLE": ["r1"]
    }
  },
  "AdHoc": {
    "description": "Ad-hoc queries",
    "isDefault": false,
    "maxConcurrentQueries": 10,
    "maxDelayQueueSize": 10,
    "granted": {
      "USER": [],
      "GROUP": ["g3"],
      "ROLE": ["r2"]
    }
  }
}

Running a Query

When running a query, you can specify the workload queue to run the query on. If the queue is not specified, the query will be routed to the default queue. To specify the queue in the GSQL shell, you can use the -queue option, e.g.

RUN QUERY -queue AdHoc q1()

or you can use the HTTP header Workload-Queue:

curl -X POST -u tigergraph:tigergraph \
  -H "Workload-Queue: AdHoc" \
  <hostname>:14240/restpp/query/ldbc_snb/q1"

If the given queue is not granted to the current user, the query will be rejected with the error code REST-14000 and return HTTP 422 Unprocessable Entity.

For example, if the user tigergraph who does not belong to the group g3 or holds the role r2 tries to run a query on the queue AdHoc, the query will be rejected.

If the queue is full of capacity, the query will be rejected.

Other Query Workload Management Methods

Limit number of concurrent heavy queries

This configuration is deprecated as of TG 3.10.0 and will be removed in a future release. This is ignored once the workload queue feature is enabled.

TigerGraph has a few built-in queries that are memory-intensive, here referred to as "heavy". These queries tend to be invoked by applications such as GraphStudio. You can set a limit of how many of these heavy queries are allowed to run concurrently by configuring the parameter RESTPP.WorkLoadManager.MaxHeavyBuiltinQueries with the gadmin config command.

For example, to set the maximum number of heavy built-in queries to 10, run the following command:

$ gadmin config set RESTPP.WorkLoadManager.MaxHeavyBuiltinQueries 10

You must restart the RESTPP service for the change to take effect.

Limit number of concurrent queries

This configuration is deprecated as of TG 3.10.0 and will be removed in a future release. This is ignored once the workload queue feature is enabled.

You can use the RESTPP.WorkLoadManager.MaxConcurrentQueries parameter to set a limit of how many queries are allowed to be running concurrently. The count of these queries does not include the built-in heavy queries.

For example, to specify that there can only be 50 concurrent queries at a time, excluding the heavy built-in queries, change the value of the configuration parameter to 50 with the gadmin config command:

$ gadmin config set RESTPP.WorkLoadManager.MaxConcurrentQueries 50

If the maximum number of concurrent queries is reached, newly submitted queries are placed in a delay queue, and begin to run as the currently running queries finish. If the queue is at capacity, newly submitted queries are rejected. and you need wait until there is capacity to run the query again. You can adjust the size of the queue with the configuration parameter RESTPP.WorkLoadManager.MaxDelayQueueSize.

For example, to specify that a maximum 20 queries may remain in the queue, run the following command:

$ gadmin config set RESTPP.WorkLoadManager.MaxDelayQueueSize 20

You must restart the RESTPP service for the change to take effect.

Specify replica to run query on

On a distributed cluster, you can specify on which replica you want a query to be run through the Run Query REST endpoint.

For example, to run the query on the primary cluster, use the GSQL-REPLICA header when running a query and set its value to 1:

Specify that the query run on the primary cluster
curl -X POST -H "GSQL-REPLICA: 1" -d '{"p":{"id":"Tom","type":"person"}}'
"http://localhost:14240/restpp/query/social/hello"

Query Routing Schemes

In a distributed or replicated cluster, REST++ automatically routes queries to different GPEs, in order to spread the workload.

If GSQL-REPLICA header is used when invoking a query, this header overrides the routing scheme for that query.

Round Robin routing

The default query routing scheme is round-robin. The first query is managed by GPE 0, the next query by GPE 1, and so on. After the last GPE, the cycle returns to GPE 0.

Version 3.9.3 adds a system configuration parameter RESTPP.CPULoadAware.Mode to enable system administrators to select other query routing schemes:

  • Mode = 0 (default): Round-Robin

  • Mode = 1: CPU Load Aware

CPU Load Aware Query Routing

When this query routing mode is selected, REST++ tries to direct incoming queries to the GPEs that are currently less busy.

Specifically, the system periodically polls CPU usage data to find a GPE whose CPU usage percentage is below RESTPP.QueryRouting.TargetSelectionCPUThreshold (default 50).

If no GPE satisfies the CPU threshold condition, REST++ falls back to the default behavior (round-robin selection).

Example: Change CPU Load Threshold and Enable CPU Load Aware routing
$ gadmin config entry RESTPP.QueryRouting.TargetSelectionCPUThreshold 40
$ gadmin config entry RESTPP.QueryRouting.Mode 1