Map Data To Graph

Map data to graph

After you have created a graph schema, the next major step is to map your data to the schema. Click "Map Data To Graph" on the left side menu bar. The working panel is split into a left panel and a right panel. Before any data is mapped, the left panel will display only the graph schema.

map data to graph overview

The main steps are

  1. Select a data source.

  2. Add data file(s)

  3. Map data file(s) to vertex/edge types

  4. Map data file columns to vertex/edge fields

  5. Publish data mapping

Select a data source type

GraphStudio supports loading data from local files on the TigerGraph server, from files stored on Amazon S3, or from files stored on Google Cloud Storage (GCS).

Click the Add Data File button add data source btn in the toolbar to add data files.

choose a data source type
  • If you select Local File, no more configuration is needed. Skip the sections for external sources and go to Map Data To Graph.

  • For AWS S3, skip to the section Create S3 data source.

  • For Google Cloud Storage, skip to the section Create a GCS data source.

Add data files

A data file is a file containing structured data to be loaded into the graph, creating vertex and/or edge instances. After specifying your data source type, the next step is to specify your data files.

This section contains subsections for each data source.

Add local data file

After clicking Local File, you will be prompted to choose a local file to upload through the GUI.

choose a local file

Click the Upload File button upload file btn and choose the data files you want to upload to the server data folder, then click ADD.

upload files

If all the files are uploaded successfully, the progress window will close automatically, otherwise the window will stay open to notify users of any errors.

Once the files are uploaded to the server, they will appear in the "Files on server" list on the left side of the Add Data Files window.

Data Files must be .csv files to allow data mapping in GraphStudio (.json file upload is allowed in TigerGraph V3.0.5+ through GraphStudio)

The Add Data File box will only upload files which end in ".csv". If you manually place files in the <TigerGraph_root_dir>/data/gui/loading_data folder, please don’t put any files into subfolders because they will be ignored.

In TigerGraph V3.0.5, you can upload files which end in ".json". This allows users to upload ".json" files to the filesystems and use a remote GSQL client, or GSQL web shell in TigerGraph Cloud to map data to the graph using GSQL commands. Currently, data mapping for ".json" files through GraphStudio UI is not supported.

Configure the File Parser

In this step, you tell GraphStudio how to parse your data file. If your data file is in tabular format, the parser will split each line into a series of tokens.

Choose a file from the file list to show a preview of the parsed data:

Screen Shot 2019 05 16 at 12.03.19 PM

If the parsing is not correct, choose a different option for the file format, delimiter, or end of line character.

The enclosing character is used to mark the boundaries of a token, overriding the delimiter character. For example, if your delimiter is a comma, but you have commas in some strings, then you can define single or double quotes as the enclosing character to mark the endpoints of your string tokens.

It is not necessary for every token to have enclosing characters; the parser will use enclosing characters when it encounters them.

GraphStudio allows users to edit the header line of the parsing result to make the data mapping more intuitive. This doesn’t affect the data loading because the header line will be ignored.

edit header

Once you are satisfied with the file parsing configuration, click the ADD button to add the data file into the left working panel.

If a file is no longer needed, you can remove it from the server by clicking the delete button delete forever to the left of each file.

After removing a file from the server, you also need to manually remove data mapping using that file. Otherwise, a "file not on server" error will be triggered when loading data.

Create S3 data source

TigerGraph creates a connection to S3 in order to access the file system in an S3 bucket. Only one file may be selected at a time.

After you click the S3 data source icon, you will be prompted to first connect an S3 data source.

choose file from s3 bucket

Click the Add new data source button upload file btn to bring up the new S3 data source window. Give a name to the data source and provide the access key id and secret access key to connect to S3.

new s3 data source

Click the ADD button and the data source will appear in the Data Source list.

Clicking the data source in the list expands a file tree listing all buckets and folders accessible by the credentials provided.

Choose the file you want to add and change the parsing options if necessary. The first ten lines of the selected file appear as a preview.

Screen Shot 2019 05 16 at 12.56.32 PM

Data files, after decompression, must be in either csv or parquet format.

TigerGraph supports loading from archived and compressed S3 files directly. Currently supported file extensions include zip, tar.gz, tgz and tar. GraphStudio detects the file extension and automatically chooses the corresponding file format. If the file is encoded with one of these formats but has a non-standard file extension, you can manually specify the file format.

Confirm that the data has parsed correctly (see Configure the File Parser) and click ADD to add the data source to your project, where it will appear in the working panel.

Create a GCS data source

After clicking Local File when selecting a data source type, you will be prompted to enter a custom name for your GCS data source.

add data source from gcs

Underneath the name line, upload your GCS account key file. Google provides a guide on generating and downloading key files at this link: Getting a service account key.

After you enter your key, enter the gsutil URI for your data file in your Google Cloud Storage bucket.

gcs data source

In addition to single files, TigerGraph also supports loading an entire folder by entering the gsutil URI for that folder.

All data files in this folder must share the same data schema. The folder preview, like the file preview, is limited to the first ten lines of uploaded data. If a folder contains more than one file and the first file has more than ten lines, only the first ten lines of the first file will appear in the preview.

TigerGraph supports loading from archived and compressed GCS files directly. Currently supported file extensions include zip, tar.gz, tgz and tar. GraphStudio detects the file extension and automatically chooses the corresponding file format. If the file is encoded with one of these formats but has a non-standard file extension, you can manually specify the file format.

Confirm that the data has parsed correctly (see Configure the File Parser) and click ADD to add the data source to your project, where it will appear in the working panel.

Map data files to vertex type or edge type

In this step, you link (map) a data file to a target vertex type or edge type. The mapping can be many-to-many, which means one data file can map to multiple vertex and/or edge types, and multiple data files can map to the same vertex or edge type. Click the map data file to vertex or edge button map file to ve to enter map data file to vertex or edge mode.

Then, click the data file icon. A hint will appear over the icon:

Screen Shot 2019 05 16 at 1.05.30 PM

Next, click the target vertex type circle or edge type link. A dashed link will appear between the data file and the target vertex or edge type:

Screen Shot 2019 05 16 at 2.20.53 PM

A red hint will appear if the target type has not yet received a mapping for its primary id(s).

Map data columns to vertex or edge attributes

In this step, you link particular columns of a data file to particular ids or attributes of a vertex type or edge type.

First, choose one data mapping from one data file to one vertex or edge type (represented as a dashed green link on the left working panel).

When selected, the dashed line becomes orange (active), and the right working panel will show two tables with the data file and target vertex or edge fields.

1

To map a column in the data file to a vertex or edge field, first select the row representing the data column in the left table, then click the target field in the right table.

A green arrow appears to show the mapping.

3

Repeat as needed to create all the mappings for this table-to-vertex/edge pair. Since many-to-one mapping is allowed, it is not necessary for one table to provide a mapping for every field in the target vertex/edge.

Using a Token Function

GraphStudio gives you access to both a set of built-in functions and user-defined token functions to preprocess data file tokens before loading them in to the graph. For example, you can concatenate two columns in the data file and load them as an attribute. This section describes how to use these token functions.

Click the add token function button add token function btn to open the Add Token Function window. Select a token function from the drop-down list under Function name, then click the ADD button.

For some functions, you may also specify the number of input parameters. (Most token functions have a fixed number of input parameters; gsql_concat can accept any positive number of inputs).

add token function window

GraphStudio currently does not support creating new user-defined functions. If a user-defined function has been added via the GSQL interface, it will be listed here. To use a user-defined token function, you must manually specify the number of input parameters. The C++ code is shown in the Description section for your reference:

user defined token functions

A token function table will be added to the attribute mapping panel.

Token functions act as intermediate steps in the mapping. Create mappings from the data file table to the token function table, and then from the token function table to the vertex/attribute table.

4

Auto Mapping

If the data file columns and the vertex/edge attributes have very similar names (only capitalization and hyphen differences), you can click the auto mapping button auto mapping btn. All similar columns will be mapped automatically.

Map a constant value to an attribute or token function input

Sometimes, a user may need to load a constant value to an id or attribute. Here we show how to do this in GraphStudio.

Loading a constant to an attribute

In the right working panel, double-click on the target id or attribute in the left column of the right table. In the example below, the attribute "label" has been double-clicked:

6

This will cause the Load Constant window to pop up. Type in the constant value, and click the Add button to apply the mapping.

load constant panel

After adding the constant value, the attribute’s label will change to id/attribute = "(your valid input value)" .

7

To modify or remove a constant mapping, double-click the id/attribute again. In the Load Constant window, enter the new value, or erase the value if you want to remove the mapping. Click the Add button to apply.

Use a constant input for a token function

First add the token function. Then double-click on the target input (in the left column of the token function table). In the example below, "Input 0" has been double-clicked.

double click token function input

This will cause the Load Constant window to pop up. Type in the constant value and click the Add button to apply the mapping. After adding the constant value, the input’s label will change to Input = "(your input value)" .

token function with constant input

The constant value can be modified or removed by double-clicking the label and editing the value in the Load Constant window.

Add data filter

You can add a data filter to a data mapping so that only data records which meet conditions that you specify will be loaded into the graph. This is equivalent to the WHERE clause in a GSQL load statement.

You can add one data filter for each data mapping from a data file to a vertex type or edge type, and the data filter only applies to that one mapping. Consider this data mapping:

8

By default, there is no data filter. Click the Data Filter button filter (2) (1) (2)to start creating a data filter. The Add Data Filter window will appear. The window contains three parts:

  1. The top section shows one row of sample data from your file, as a handy reference to the file’s contents.

  2. The middle sections shows what the data filter looks like when it is converted a to GSQL WHERE clause. For more details, see the WHERE Clause section in the GSQL Language Reference Part 1 - Defining Graphs and Loading Data

  3. The bottom section is where you define your data filter. The data filter will be converted to a GSQL WHERE clause and shown in real time.

add data filter window

A data filter condition is a Boolean expression, which can be a nested set of conditions. TigerGraph data loader evaluates the condition for each line in your input file. If the condition evaluates to be true, then the line of data is loaded.

First, click the Build Data Filter chooser (with default value "None"). A menu will appear, with many Boolean expression templates. Choose one of the options. If you plan to build a nested condition, start with your top level. The first several options are for comparison expressions:

data filter choose expression

After this are several more options, using operators such as AND, OR, NOT, IN, BETWEEN…​AND, IS NUMERIC, and IS EMPTY.

data filter choose operator

Note that each of these expressions calls for 1, 2, 3, or a list of operands, and the operands themselves can be expressions. When you select an expression, additional choosers will appear below for you to specify the operand expressions. The operand choices are context-sensitive, but typically they include

  • a Data Column from the input file

  • A constant value

  • If the operator is AND, OR, or NOT, then the operand can be another condition. Thus is how conditions can be nested.

Suppose you are loading friendship edges where the input data fields are (person1, person2, friendship_start_date). You want to load only the records where person1 is Tom and the friendship began on or before 2017-06-10. The data filter looks like the following:

complete data filter

After adding the data filter, the right working panel will look like this:

9

Hovering the mouse over the data filter indicator image will make the data filter condition appear. If you want to modify the data filter, click the Data Filter button image or double-click the data filter indicator. The Add Data Filter panel will appear.

To remove a data filter, select "None" at the top level dropdown of the Build Data Filter section and then click ADD. The data filter will be deleted.

image

Advanced Features

More advanced data mapping features are grouped in the dropdown list in the three-dot menu image.

Map data to a map type attribute

Click image in the dropdown list, then choose key type and value type. The types must match the key type and value type of the attribute you are mapping towards.

image

A Map widget will be added to the attribute mapping panel.

image

Create the mapping from the data columns to the Map widget, and from the Map widget to the attribute.

image

Map data to a UDT type attribute

Choose a UDT name from the dropdown list. The name must match the UDT type of the attribute you are mapping towards.

image

A UDT widget will be added to the attribute mapping panel.

image

Create the mapping from the data columns to the UDT widget, and from the UDT widget to the attribute.

image

Map data to a map type attribute with UDT value type

If you want to map data to an attribute of map type with UDT value type, you have to combine a Map widget with a UDT widget.

Choose UDT as the value type and then choose the UDT name when adding the Map widget.

image

Create data mapping between data columns, the UDT widget, the Map widget, and the attribute.

image

Delete options

In the Map Data To Graph page, you can delete anything that you added. Choose what you want to delete, then click the delete button image . Press the "Shift" key to select multiple icons you want to delete. Note that you cannot delete vertex or edge types in this page.

Delete data files

Select the data file icon(s), then click the delete button.

image

Delete data file to vertex or edge mapping

Select the dashed green link(s) between data file and mapped vertex/edge type, then click the delete button.

image

Delete data column to vertex or edge attribute mapping

Select the green arrow(s) between data file table and vertex/edge attributes table, then click the delete button.

image

Delete token functions

Select the token function table(s), then click the delete button.

image

Undo and redo

You can undo or redo changes by clicking the Back or Forward buttons, respectively: image . The whole history since the time you entered the Map Data To Graph page is recorded.

Publish data mapping

Once you are satisfied with the data loading procedure, click the publish schema button image to publish the data loading procedure to the TigerGraph system. It takes about 2 to 3 seconds for publishing each data file mapping.

Expand panels

The following three buttons allow you to expand the left or right working panel: image .

By default, the two panels have equal widths. Click the left button to expand the left working panel, or click the right button to expand the right working panel.