Self-Managed TigerGraph CoPilot
TigerGraph CoPilot is an open-source project on GitHub [https://github.com/tigergraph/CoPilot] which can be deployed to your own infrastructure.
If you don’t need to extend the source code of CoPilot, the quickest way is to deploy its Docker images with the Docker Compose file in the repo. |
You will need the following prerequisites.
Prerequisites
In order to utilize TigerGraph CoPilot, you will need:
-
To install:
-
Docker
-
TigerGraph DB 3.9+. (For 3.x, you will need to install a few user defined functions (UDFs). Please see Step 5 below for details.)
For installing Docker and a TigerGraph DB, see our documentation on Get Started with Docker.
For a graph schema and data refer to our Example Graphs and see our documentation on Defining a Schema and Loading Data in the GSQL 101 tutorial.
-
-
An LLM provider and its API key
An LLM provider refers to a company or organization that offers Large Language Models (LLMs) as a service. The API key verifies the identity of the requester, ensuring that the request is coming from a registered and authorized user or application.
Currently, TigerGraph CoPilot Supports:
-
Groq
Only one provider is required.
As an example of how to obtain your API key from providers such as OpenAI, please refer to https://platform.openai.com/api-keys.
1) Setup
First, users need to get the TigerGraph CoPilot images and prep the environment.
-
Download the file docker-compose.yml directly.
-
Or, clone the repo
git clone https://github.com/tigergraph/CoPilot
.
The Docker Compose file contains all dependencies for CoPilot including a Milvus database. If you do not need a particular service, you make edit the Compose file to remove it or set its scale to 0 when running the Compose file (details later). Moreover, CoPilot comes with a Swagger API documentation page when it is deployed.
If you wish to disable it, you can set the PRODUCTION
environment variable to true for the CoPilot service in the Compose file.
2) Set up Configuration Files
Next, in the same directory as the Docker Compose file is in, create and fill in the following configuration files.
-
configs/db_config.json
-
configs/llm_config.json
-
configs/milvus_config.json
-
configs/chat_config.json
TigerGraph Database Configuration
Copy the below into configs/db_config.json
and edit the hostname
and getToken
fields to match your database’s configuration.
Set the timeout
, memory threshold
, and thread limit
parameters to control how much of the database’s resources are consumed when answering a question.
|
{
"hostname": "http://tigergraph",
"getToken": true,
"default_timeout": 300,
"default_mem_threshold": 5000,
"default_thread_limit": 8,
"ecc": "http://eventual-consistency-service:8001",
"chat_history_api": "http://chat-history:8002"
}
LLM Provider Configuration
Next, in the configs/llm_config.json
file, paste in your LLM provider’s JSON config template from below, and fill out the appropriate fields.
In addition to the OPENAI_API_KEY
, the llm_model
and model_name
, can be edited to match your specific configuration details.
{
"model_name": "GPT-4",
"embedding_service": {
"embedding_model_service": "openai",
"authentication_configuration": {
"OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"
}
},
"completion_service": {
"llm_service": "openai",
"llm_model": "gpt-4-0613",
"authentication_configuration": {
"OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"
},
"model_kwargs": {
"temperature": 0
},
"prompt_path": "./app/prompts/openai_gpt4/"
}
}
-
Follow the GCP authentication information found here and create a
Service Account
withVertexAI
credentials. -
Then, add the following to the
docker run
command:-v $(pwd)/configs/SERVICE_ACCOUNT_CREDS.json:/SERVICE_ACCOUNT_CREDS.json -e GOOGLE_APPLICATION_CREDENTIALS=/SERVICE_ACCOUNT_CREDS.json
-
Finally, your JSON config should as below:
{ "model_name": "GCP-text-bison", "embedding_service": { "embedding_model_service": "vertexai", "authentication_configuration": {} }, "completion_service": { "llm_service": "vertexai", "llm_model": "text-bison", "model_kwargs": { "temperature": 0 }, "prompt_path": "./app/prompts/gcp_vertexai_palm/" } }
In addition to the AZURE_OPENAI_ENDPOINT
, AZURE_OPENAI_API_KEY
, and azure_deployment
, the llm_model
and model_name
can be edited to match your specific configuration details.
{
"model_name": "GPT35Turbo",
"embedding_service": {
"embedding_model_service": "azure",
"azure_deployment":"YOUR_EMBEDDING_DEPLOYMENT_HERE",
"authentication_configuration": {
"OPENAI_API_TYPE": "azure",
"OPENAI_API_VERSION": "2022-12-01",
"AZURE_OPENAI_ENDPOINT": "YOUR_AZURE_ENDPOINT_HERE",
"AZURE_OPENAI_API_KEY": "YOUR_AZURE_API_KEY_HERE"
}
},
"completion_service": {
"llm_service": "azure",
"azure_deployment": "YOUR_COMPLETION_DEPLOYMENT_HERE",
"openai_api_version": "2023-07-01-preview",
"llm_model": "gpt-35-turbo-instruct",
"authentication_configuration": {
"OPENAI_API_TYPE": "azure",
"AZURE_OPENAI_ENDPOINT": "YOUR_AZURE_ENDPOINT_HERE",
"AZURE_OPENAI_API_KEY": "YOUR_AZURE_API_KEY_HERE"
},
"model_kwargs": {
"temperature": 0
},
"prompt_path": "./app/prompts/azure_open_ai_gpt35_turbo_instruct/"
}
}
Specify, your configuration details in the sample file below:
{
"model_name": "Claude-3-haiku",
"embedding_service": {
"embedding_model_service": "bedrock",
"embedding_model":"amazon.titan-embed-text-v1",
"authentication_configuration": {
"AWS_ACCESS_KEY_ID": "ACCESS_KEY",
"AWS_SECRET_ACCESS_KEY": "SECRET"
}
},
"completion_service": {
"llm_service": "bedrock",
"llm_model": "anthropic.claude-3-haiku-20240307-v1:0",
"authentication_configuration": {
"AWS_ACCESS_KEY_ID": "ACCESS_KEY",
"AWS_SECRET_ACCESS_KEY": "SECRET"
},
"model_kwargs": {
"temperature": 0,
},
"prompt_path": "./app/prompts/aws_bedrock_claude3haiku/"
}
}
Example configuration for a model from Ollama is shown below. Please specify your configuration details:
{
"model_name": "GPT-4",
"embedding_service": {
"embedding_model_service": "openai",
"authentication_configuration": {
"OPENAI_API_KEY": ""
}
},
"completion_service": {
"llm_service": "ollama",
"llm_model": "calebfahlgren/natural-functions",
"model_kwargs": {
"temperature": 0.0000001
},
"prompt_path": "./app/prompts/openai_gpt4/"
}
}
Example configuration for a model on Hugging Face with a dedicated endpoint is shown below. Please specify your configuration details:
{
"model_name": "llama3-8b",
"embedding_service": {
"embedding_model_service": "openai",
"authentication_configuration": {
"OPENAI_API_KEY": ""
}
},
"completion_service": {
"llm_service": "huggingface",
"llm_model": "hermes-2-pro-llama-3-8b-lpt",
"endpoint_url": "https:endpoints.huggingface.cloud",
"authentication_configuration": {
"HUGGINGFACEHUB_API_TOKEN": ""
},
"model_kwargs": {
"temperature": 0.1
},
"prompt_path": "./app/prompts/openai_gpt4/"
}
}
Example configuration for a model on Hugging Face with a serverless endpoint is shown below. Please specify your configuration details:
{
"model_name": "Llama3-70b",
"embedding_service": {
"embedding_model_service": "openai",
"authentication_configuration": {
"OPENAI_API_KEY": ""
}
},
"completion_service": {
"llm_service": "huggingface",
"llm_model": "meta-llama/Meta-Llama-3-70B-Instruct",
"authentication_configuration": {
"HUGGINGFACEHUB_API_TOKEN": ""
},
"model_kwargs": {
"temperature": 0.1
},
"prompt_path": "./app/prompts/llama_70b/"
}
}
Example configuration for a model from Groq is shown below. Please specify your configuration details:
{
"model_name": "mixtral-8x7b-32768",
"embedding_service": {
"embedding_model_service": "openai",
"authentication_configuration": {
"OPENAI_API_KEY": ""
}
},
"completion_service": {
"llm_service": "groq",
"llm_model": "mixtral-8x7b-32768",
"authentication_configuration": {
"GROQ_API_KEY": ""
},
"model_kwargs": {
"temperature": 0.1
},
"prompt_path": "./app/prompts/openai_gpt4/"
}
}
Milvus Configuration
Milvus is CoPilot’s vector database. Copy the below into configs/milvus_config.json
and edit the host
and port
fields to match your Milvus configuration. If running the Docker Compose file as is, you don’t need to change anything. The host
address in the template is used by the Docker Compose file.
-
username
andpassword
can also be configured below if required by your Milvus setup. -
enabled
should always be"true"
for now as Milvus is the only embedding store supported.
{
"host": "milvus-standalone",
"port": 19541,
"username": "",
"password": "",
"enabled": "true",
"sync_interval_seconds": 60
}
Chat History Configuration
Copy the below code into configs/chat_config.json`
. You shouldn’t need to change anything unless you change the port of the chat history service in the Docker Compose file.
{
"apiPort":"8002",
"dbPath": "chats.db",
"dbLogPath": "db.log",
"logPath": "requestLogs.jsonl",
"conversationAccessRoles": ["superuser", "globaldesigner"]
}
3) (Optional) Logging
Users can also configure logging in TigerGraph CoPilot service.
Create log configuration file
Copy the below into configs/log_config.json
and edit the appropriate values to suit your needs.
{
"log_file_path": "logs",
"log_max_size": 10485760,
"log_backup_count": 10
}
The log is rotated and the rotation is based on the size and backups.
These configurations are applied in the LogWriter
to the standard Python logging package.
Operational and audit logs are recorded.
-
log.ERROR
-
log.INFO
-
and
log.AUDIT-COPILOT
Configure Logging Level in Dockerfile
To configure the logging level of the service, edit the Docker Compose file.
ENV LOGLEVEL="INFO"
This line can be changed to support different logging levels.
Level | Description |
---|---|
|
A serious error. |
|
Failing to perform functions. |
|
Indication of unexpected problems, e.g. failure to map a user’s question to the graph schema. |
|
Confirming that the service is performing as expected. |
|
Detailed information, e.g. the functions retrieved during the |
|
Finer-grained information that could potentially include |
NOTSET |
All messages are processed. |
4) Start CoPilot
Now, simply run docker compose up -d
and wait for all the services to start. If you don’t want to use the included Milvus DB, you can set its scale to 0 to not start it: docker compose up -d --scale milvus-standalone=0 --scale etcd=0 --scale minio=0
.
5) Install UDFs
This step is not needed for TigerGraph databases version 4.x. For TigerGraph 3.x, we need to install a few user defined functions (UDFs) for CoPilot to work.
-
On the machine that hosts the TigerGraph database, switch to the user of TigerGraph:
sudo su - tigergraph
. If TigerGraph is running on a cluster, you can do this on any one of the machines. -
Download the two files tg_ExprFunctions.hpp and tg_ExprUtil.hpp.
If you already have custom UDFs, copy them to those two files. Installing the UDF files will overwrite any UDFs already installed in the database.
-
In a terminal, run the following command to enable UDF installation:
gadmin config set GSQL.UDF.EnablePutTgExpr true gadmin config set GSQL.UDF.Policy.Enable false gadmin config apply gadmin restart GSQL
-
Enter a GSQL shell, and run the following command to install the UDF files.
PUT tg_ExprFunctions FROM "./tg_ExprFunctions.hpp" PUT tg_ExprUtil FROM "./tg_ExprUtil.hpp"
-
Quit the GSQL shell, and run the following command in the terminal to disable UDF installation for security purpose.
gadmin config set GSQL.UDF.EnablePutTgExpr false gadmin config set GSQL.UDF.Policy.Enable true gadmin config apply gadmin restart GSQL
Next Steps
Once, that is running now you can move on to the five ways to Use TigerGraph Co-Pilot.
Return to TigerGraph CoPilot for a different topic.