Self-Managed TigerGraph CoPilot

TigerGraph CoPilot is an open-source project on GitHub [https://github.com/tigergraph/CoPilot] which can be deployed to your own infrastructure.

If you don’t need to extend the source code of CoPilot, the quickest way is to deploy its Docker images with the Docker Compose file in the repo.

You will need the following prerequisites.

Prerequisites

In order to utilize TigerGraph CoPilot, you will need:

  • To install:

    • Docker

    • TigerGraph DB 3.9+. (For 3.x, you will need to install a few user defined functions (UDFs). Please see Step 5 below for details.)

      For installing Docker and a TigerGraph DB, see our documentation on Get Started with Docker.

      For a graph schema and data refer to our Example Graphs and see our documentation on Defining a Schema and Loading Data in the GSQL 101 tutorial.

  • An LLM provider and its API key

    An LLM provider refers to a company or organization that offers Large Language Models (LLMs) as a service. The API key verifies the identity of the requester, ensuring that the request is coming from a registered and authorized user or application.

    Currently, TigerGraph CoPilot Supports:

As an example of how to obtain your API key from providers such as OpenAI, please refer to https://platform.openai.com/api-keys.

1) Setup

First, users need to get the TigerGraph CoPilot images and prep the environment.

You can either:

The Docker Compose file contains all dependencies for CoPilot including a Milvus database. If you do not need a particular service, you make edit the Compose file to remove it or set its scale to 0 when running the Compose file (details later). Moreover, CoPilot comes with a Swagger API documentation page when it is deployed. If you wish to disable it, you can set the PRODUCTION environment variable to true for the CoPilot service in the Compose file.

2) Set up Configuration Files

Next, in the same directory as the Docker Compose file is in, create and fill in the following configuration files.

  • configs/db_config.json

  • configs/llm_config.json

  • configs/milvus_config.json

  • configs/chat_config.json

TigerGraph Database Configuration

Copy the below into configs/db_config.json and edit the hostname and getToken fields to match your database’s configuration.

Set the timeout, memory threshold, and thread limit parameters to control how much of the database’s resources are consumed when answering a question.

  • If you use the Docker Compose file as is, you don’t need to change anything. “ecc” and “chat_history_api” are the addresses of internal components of CoPilot.

  • If token authentication is enabled in TigerGraph, set getToken to true.

 
{
    "hostname": "http://tigergraph",
    "getToken": true,
    "default_timeout": 300,
    "default_mem_threshold": 5000,
    "default_thread_limit": 8,
    "ecc": "http://eventual-consistency-service:8001",
    "chat_history_api": "http://chat-history:8002"
}

LLM Provider Configuration

Next, in the configs/llm_config.json file, paste in your LLM provider’s JSON config template from below, and fill out the appropriate fields.


  • OpenAI

  • GCP

  • Azure

  • AWS Bedrock

  • Ollama

  • Hugging Face

  • Groq

In addition to the OPENAI_API_KEY, the llm_model and model_name, can be edited to match your specific configuration details.

{
    "model_name": "GPT-4",
    "embedding_service": {
        "embedding_model_service": "openai",
        "authentication_configuration": {
            "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"
        }
    },
    "completion_service": {
        "llm_service": "openai",
        "llm_model": "gpt-4-0613",
        "authentication_configuration": {
            "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"
        },
    "model_kwargs": {
        "temperature": 0
    },
    "prompt_path": "./app/prompts/openai_gpt4/"
    }
}

  1. Follow the GCP authentication information found here and create a Service Account with VertexAI credentials.

  2. Then, add the following to the docker run command:

    -v $(pwd)/configs/SERVICE_ACCOUNT_CREDS.json:/SERVICE_ACCOUNT_CREDS.json -e GOOGLE_APPLICATION_CREDENTIALS=/SERVICE_ACCOUNT_CREDS.json

  3. Finally, your JSON config should as below:

    {
    "model_name": "GCP-text-bison",
    "embedding_service": {
        "embedding_model_service": "vertexai",
        "authentication_configuration": {}
    },
    "completion_service": {
        "llm_service": "vertexai",
        "llm_model": "text-bison",
        "model_kwargs": {
            "temperature": 0
        },
    "prompt_path": "./app/prompts/gcp_vertexai_palm/"
    }
}

In addition to the AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, and azure_deployment, the llm_model and model_name can be edited to match your specific configuration details.

{
    "model_name": "GPT35Turbo",
    "embedding_service": {
        "embedding_model_service": "azure",
        "azure_deployment":"YOUR_EMBEDDING_DEPLOYMENT_HERE",
        "authentication_configuration": {
            "OPENAI_API_TYPE": "azure",
            "OPENAI_API_VERSION": "2022-12-01",
            "AZURE_OPENAI_ENDPOINT": "YOUR_AZURE_ENDPOINT_HERE",
            "AZURE_OPENAI_API_KEY": "YOUR_AZURE_API_KEY_HERE"
        }
    },
    "completion_service": {
        "llm_service": "azure",
        "azure_deployment": "YOUR_COMPLETION_DEPLOYMENT_HERE",
        "openai_api_version": "2023-07-01-preview",
        "llm_model": "gpt-35-turbo-instruct",
        "authentication_configuration": {
            "OPENAI_API_TYPE": "azure",
            "AZURE_OPENAI_ENDPOINT": "YOUR_AZURE_ENDPOINT_HERE",
            "AZURE_OPENAI_API_KEY": "YOUR_AZURE_API_KEY_HERE"
        },
        "model_kwargs": {
            "temperature": 0
        },
        "prompt_path": "./app/prompts/azure_open_ai_gpt35_turbo_instruct/"
    }
}

Specify, your configuration details in the sample file below:

{
    "model_name": "Claude-3-haiku",
    "embedding_service": {
        "embedding_model_service": "bedrock",
        "embedding_model":"amazon.titan-embed-text-v1",
        "authentication_configuration": {
            "AWS_ACCESS_KEY_ID": "ACCESS_KEY",
            "AWS_SECRET_ACCESS_KEY": "SECRET"
        }
    },
    "completion_service": {
        "llm_service": "bedrock",
        "llm_model": "anthropic.claude-3-haiku-20240307-v1:0",
        "authentication_configuration": {
            "AWS_ACCESS_KEY_ID": "ACCESS_KEY",
            "AWS_SECRET_ACCESS_KEY": "SECRET"
        },
        "model_kwargs": {
            "temperature": 0,
        },
        "prompt_path": "./app/prompts/aws_bedrock_claude3haiku/"
    }
}

Example configuration for a model from Ollama is shown below. Please specify your configuration details:

{
    "model_name": "GPT-4",
    "embedding_service": {
        "embedding_model_service": "openai",
        "authentication_configuration": {
            "OPENAI_API_KEY": ""
        }
    },
    "completion_service": {
        "llm_service": "ollama",
        "llm_model": "calebfahlgren/natural-functions",
        "model_kwargs": {
            "temperature": 0.0000001
        },
        "prompt_path": "./app/prompts/openai_gpt4/"
    }
}

Example configuration for a model on Hugging Face with a dedicated endpoint is shown below. Please specify your configuration details:

{
    "model_name": "llama3-8b",
    "embedding_service": {
        "embedding_model_service": "openai",
        "authentication_configuration": {
            "OPENAI_API_KEY": ""
        }
    },
    "completion_service": {
        "llm_service": "huggingface",
        "llm_model": "hermes-2-pro-llama-3-8b-lpt",
        "endpoint_url": "https:endpoints.huggingface.cloud",
        "authentication_configuration": {
            "HUGGINGFACEHUB_API_TOKEN": ""
        },
        "model_kwargs": {
            "temperature": 0.1
        },
        "prompt_path": "./app/prompts/openai_gpt4/"
    }
}

Example configuration for a model on Hugging Face with a serverless endpoint is shown below. Please specify your configuration details:

{
    "model_name": "Llama3-70b",
    "embedding_service": {
        "embedding_model_service": "openai",
        "authentication_configuration": {
            "OPENAI_API_KEY": ""
        }
    },
    "completion_service": {
        "llm_service": "huggingface",
        "llm_model": "meta-llama/Meta-Llama-3-70B-Instruct",
        "authentication_configuration": {
            "HUGGINGFACEHUB_API_TOKEN": ""
        },
        "model_kwargs": {
            "temperature": 0.1
        },
        "prompt_path": "./app/prompts/llama_70b/"
    }
}

Example configuration for a model from Groq is shown below. Please specify your configuration details:

{
    "model_name": "mixtral-8x7b-32768",
    "embedding_service": {
        "embedding_model_service": "openai",
        "authentication_configuration": {
            "OPENAI_API_KEY": ""
        }
    },
    "completion_service": {
        "llm_service": "groq",
        "llm_model": "mixtral-8x7b-32768",
        "authentication_configuration": {
            "GROQ_API_KEY": ""
        },
        "model_kwargs": {
            "temperature": 0.1
        },
        "prompt_path": "./app/prompts/openai_gpt4/"
    }
}

Milvus Configuration

Milvus is CoPilot’s vector database. Copy the below into configs/milvus_config.json and edit the host and port fields to match your Milvus configuration. If running the Docker Compose file as is, you don’t need to change anything. The host address in the template is used by the Docker Compose file.

  • username and password can also be configured below if required by your Milvus setup.

  • enabled should always be "true" for now as Milvus is the only embedding store supported.

{
    "host": "milvus-standalone",
    "port": 19541,
    "username": "",
    "password": "",
    "enabled": "true",
    "sync_interval_seconds": 60
}

Chat History Configuration

Copy the below code into configs/chat_config.json`. You shouldn’t need to change anything unless you change the port of the chat history service in the Docker Compose file.

{
    "apiPort":"8002",
    "dbPath": "chats.db",
    "dbLogPath": "db.log",
    "logPath": "requestLogs.jsonl",
    ​​"conversationAccessRoles": ["superuser", "globaldesigner"]
}

3) (Optional) Logging

Users can also configure logging in TigerGraph CoPilot service.

Create log configuration file

Copy the below into configs/log_config.json and edit the appropriate values to suit your needs.

{
    "log_file_path": "logs",
    "log_max_size": 10485760,
    "log_backup_count": 10
}

The log is rotated and the rotation is based on the size and backups. These configurations are applied in the LogWriter to the standard Python logging package.

Operational and audit logs are recorded.

Outputs include:

  • log.ERROR

  • log.INFO

  • and log.AUDIT-COPILOT

Configure Logging Level in Dockerfile

To configure the logging level of the service, edit the Docker Compose file.

By default, the logging level is set to "INFO".
ENV LOGLEVEL="INFO"

This line can be changed to support different logging levels.

Table 1. The levels are described below:
Level Description

CRITICAL

A serious error.

ERROR

Failing to perform functions.

WARNING

Indication of unexpected problems, e.g. failure to map a user’s question to the graph schema.

INFO

Confirming that the service is performing as expected.

DEBUG

Detailed information, e.g. the functions retrieved during the GenerateFunction step, etc.

DEBUG_PII

Finer-grained information that could potentially include PII, such as a user’s question, the complete function call (with parameters), and the LLM’s natural language response.

NOTSET

All messages are processed.

4) Start CoPilot

Now, simply run docker compose up -d and wait for all the services to start. If you don’t want to use the included Milvus DB, you can set its scale to 0 to not start it: docker compose up -d --scale milvus-standalone=0 --scale etcd=0 --scale minio=0.

5) Install UDFs

This step is not needed for TigerGraph databases version 4.x. For TigerGraph 3.x, we need to install a few user defined functions (UDFs) for CoPilot to work.

  1. On the machine that hosts the TigerGraph database, switch to the user of TigerGraph: sudo su - tigergraph. If TigerGraph is running on a cluster, you can do this on any one of the machines.

  2. Download the two files tg_ExprFunctions.hpp and tg_ExprUtil.hpp.

    If you already have custom UDFs, copy them to those two files. Installing the UDF files will overwrite any UDFs already installed in the database.

  3. In a terminal, run the following command to enable UDF installation:

    gadmin config set GSQL.UDF.EnablePutTgExpr true
gadmin config set GSQL.UDF.Policy.Enable false
gadmin config apply
gadmin restart GSQL

  4. Enter a GSQL shell, and run the following command to install the UDF files.

    PUT tg_ExprFunctions FROM "./tg_ExprFunctions.hpp"
PUT tg_ExprUtil FROM "./tg_ExprUtil.hpp"

  5. Quit the GSQL shell, and run the following command in the terminal to disable UDF installation for security purpose.

gadmin config set GSQL.UDF.EnablePutTgExpr false
gadmin config set GSQL.UDF.Policy.Enable true
gadmin config apply
gadmin restart GSQL

Next Steps

Once, that is running now you can move on to the five ways to Use TigerGraph Co-Pilot.

Return to TigerGraph CoPilot for a different topic.