Set up Log Viewing with Elasticsearch, Kibana and Filebeat
The TigerGraph system produces extensive and detailed logs about each of its components. Starting with TigerGraph 3.2, TigerGraph provides a gadmin
utility that allows users to easily view log files through an Elasticsearch, Kibana, and Filebeat setup. This page offers a step-by-step guide to set up log viewing for all components in a TigerGraph cluster with Elastic search, Kibana, and Filebeat.
Before you begin
Install Elasticsearch on a machine that is running TigerGraph.
If you have a TigerGraph cluster, you only need to install Elasticsearch on one node.
Install Kibana on the same machine where you installed Elasticsearch.
If you have a TigerGraph cluster, you need to install Filebeat on all nodes in the cluster.
Procedure
1. Configure Elasticsearch for remote access
The default Elasticsearch settings only allow the Elasticsearch service to be accessed from the same machine it starts from. In order to allow Elasticsearch to receive log files from other servers in the cluster, we have to make the following edits to the file at /etc/elasticsearch/elasticsearch.yml
After editing the configurations, restart the Elasticsearch service.
Elasticsearch is a memory-intensive service. For more information on memory management for Elasticsearch, see Managing and Troubleshooting Elasticsearch Memory.
2. Configure Kibana with Elasticsearch and enable remote access
You need to make the following changes to the file at /etc/kibana/kibana.yml:
To allow remote access, change the value of
server.host
to the IP address or DNS name of the Kibana server. Since the Kibana server is on the same machine as Elasticsearch, this value should be the same private IP that you specified as Elasticsearch'snetwork.host
.Additionally, you need to provide the address of the Elasticsearch server in the
elasticsearch.hosts
setting. By default, Elasticsearch is on port 9200, so the value for this setting should be["server_private_ip:9200"]
After editing the configurations, restart the Kibana service.
3. Configure Filebeat
Finally, we need to configure Filebeat to have each component on each node send its logs to the Elasticsearch server. To do so, run the following gadmin command:
The command outputs a Filebeat configuration file filebeat.yml
. The following options are available:
Option | Description |
| Required. The list of IP addresses of the nodes whose logs you want to send to the Elasticsearch server. Example:
|
| Optional. If this flag is provided, Filebeat will harvest all log files including the oldest. If not included, Filebeat will only harvest the logs since the most recent time each service started. |
| Optional. The path to output the configuration file. By default, the command outputs the configuration file |
| Optional. The services you want Filebeat to monitor. By default, all services are included. Example: --service= |
After generating the filebeat.yml
file, copy it to the directory /etc/filebeat
on every node, and restart the Filebeat service on each node.
After the service restarts, you should be able to view the logs through Kibana's user interface in your browser at server_ip:5601
.
Last updated