Our blog post GraphDB Cluster Deployment Strategies explained how to deploy a high-availability GraphDB cluster. This can be done both on-premises and in the cloud. Ontotext collaborates with major cloud providers, Azure and AWS, to offer the GraphDB high-availability cluster either as a managed service in Ontotext’s account or in the customer’s account.
This post focuses on deploying a GraphDB high-availability cluster in your own Azure account, using Terraform scripts provided by Ontotext.
The scripts
The Terraform scripts are located in the terraform-azure-graphdb repository of the Ontotext-AD GitHub site. The scripts deploy the cluster using pre-built VM images with the latest GraphDB version. Building the images with Packer is part of the GraphDB release process. The implementation can also be found in the packer-azure-graphdb repository.
The Terraform scripts bring up all the required Azure components and run additional bash scripts that set up the GraphDB cluster, security, backups, and related infrastructure. The default cluster size is three nodes, and this can be configured based on your license. The architecture is described in more detail in How to deploy GraphDB in Azure in GraphDB’s documentation.
The Terraform script also sets up important features that are critical for a live production system. Those include:
- Backup — the data is crucial for any enterprise and a regular backup is a must-have.
- Monitoring and alerting — having a centralized monitoring and alerting system lets you track the cluster’s health.
- SSL Termination — decrypting encrypted data using the Azure Application Gateway.
Deploying a 3-node cluster
You will need the following prerequisites before proceeding with the deployment:
- An Azure account with sufficient permissions to deploy the different resources
- Knowledge of Terraform (Although not strictly necessary, this will help when troubleshooting potential problems)
- Azure CLI installed and configured
- GraphDB license (the 3-node cluster requires a GraphDB Enterprise Edition License, which can be a trial license)
The first step to set up a GraphDB cluster on Azure is to clone or download the repository with the Terraform scripts or use the repository as a sub-module from the Terraform registry. Make sure that you clone the latest release. Before executing the scripts it’s good to review the README.md file, which gives some additional information about how to deploy and configure the cluster.
Below is a .tfvars file that contains example values for some of the variables that need to be passed to the Terraform to deploy the cluster correctly. In order to use it create a file called terraform.tfvars to store these values;
# This is the resource name prefix for all the resources
# that will be created.
resource_name_prefix = "gdb-cluster"
# The regions where we will deploy.
location = "East US"
# Custom tags in case you need them. We recommend using those
tags = {
Environment: "integration"
CostCenter: "organisation"
}
# This variable enables locking of all resources that support it.
# For test purposes we set it to false.
lock_resources = false
# The path to the GraphDB license.
# This should be a file that you have received from Ontotext.
graphdb_license_path = "/Users/user/licenses/graphdb.license"
# We recommend setting up an admin password here, you can always change it later
graphdb_password = "some-password"
# The version of GraphDB. If not set it will always deploy
# the pre-built VM with the latest version.
graphdb_version = "10.7.0"
# The type of the machines that will be used for the deployment. Think of your requirements and chose carefully
instance_type = "Standard_E2as_v5"
# This is the SSH key that will be used when we need to ssh
# into the machines from the cluster. Typically,this is the
# public key of your machine
ssh_key = "path-to-ssh-key"
management_cidr_blocks = ["0.0.0.0/0"]
# Path to certificate for TLS.
tls_certificate_path = "/Users/user/.terraform/certificate.p12"
# We recommend deploying and using Bastion to allow
# secure ssh connection.
deploy_bastion = true
# For the sake of a test deployment we turn off the monitoring.
# By default this will also deploy the monitoring and alerting.
deploy_monitoring = false
# Enables soft delete for anything stored in keyvault, for a test
# deployment can be set to false.
key_vault_enable_purge_protection = falseOnce we have the file with all the input parameters, we are ready to initialize it with the following command, which will look for the terraform.tfvars file in the directory where you execute this command:
Next, deploy with:
The deployment process takes around 15 minutes. Sometimes it can take more, depending on Azure. After this, you should see console output like the following:
The public_address variable is the IP address of your GraphDB instance. Keep in mind that when Terraform has completed and you have the address of your new GraphDB instance, it will take some time until the cluster is fully set up because the scripts need to set up the cluster, backups, security, and more. The Workbench will not open until all the scripts are done. You can follow this progress in the cloud-init log file.
Some organizations will not want a public GraphDB address, so the Terraform script also supports a private link.
Operating GraphDB
The GraphDB nodes are deployed in Azure Virtual Machines (VM), which are a part of an Azure Scale Set. To improve security, public SSH access to these VMs is blocked. If you need to SSH into any of them, you should use the method recommended by Microsoft: Bastion. Below is a sample Azure CLI invocation that uses Bastion to SSH into a VM using its resource ID.
% az network bastion ssh --name "bas-ivo-gdb" --resource-group "rg-gdb-cluster" --target-resource-id "resource-id-from-azure-console" --auth-type "ssh-key" --username "graphdb" --ssh-key "./azure" Once you are logged in, it might be useful to know what is where. Each VM contains a GraphDB instance and an external GraphDB cluster proxy where both are started as a system.d service. The folders where their files are stored are:
- /var/opt/graphdb/node — for the GraphDB instance
- /var/opt/graphdb/cluster-proxy — for the cluster proxy
Those might be needed if you need to change configurations or to examine logs when you don’t have the monitoring enabled.
The backups are another important part of the deployment. They are created and uploaded in Azure Blob storage daily at midnight, although you can configure this and many more options via Terraform or change them manually. The duration of the backup process will depend on your data size and machines. The following are some variables that you can use to configure the backups. For more options, please refer to the variables.tf file in the terraform-azure-graphdb github repository.
- You can use the backup_schedule to change the cron expression that defines when the backup is executed.
- storage_blobs_max_days_since_creation — defines how many days to store the backups, with the default being 31.
Monitoring
The monitoring module is disabled in the .tfvars example file but if you enable it, you can find all the metrics and alerts in the interactive Azure Application Insights tool. The setup and configuration have been integrated into the VM image and Terraform scripts.
After a successful deployment in Azure via the VM https://github.com/Ontotext-AD/packer-azure-graphdb/blob/main/files/graphdb.envimage and scripts you will notice the following resources:
- Log Analytics workspace: a unique environment for logging data from Azure Monitor and other Azure services such as Application Insights
- Application Insights: a feature of Azure Monitor that offers near-real-time metrics and alerts as well as dashboards and availability tests
- Two Action groups:
- <resource_group>-notifications: used for notifications (push|e-mail|sms) to users/roles.
- Application Insights Smart Detection: automatically created for the smart alerts that Azure offers as part of Application Insights. It will notify all users with the roles of Monitoring Contributor and Monitoring Reader.
- Additional alerts: We’ve set up several alerts based on ingested logs, signals, and availability tests:
- Low memory alert: will send a notification if the used memory exceeds the configured percentage
- Availability alert: will send a notification if the availability drops beneath 100
- Low disk space alert: iwill send a notification if the “low disk space” message is detected in the ingested logs
- Replication in progress alert: will send a notification if a request for snapshot replication is detected in the ingested logs
Application Insights also includes predefined workbooks that cover detailed analytics for Availability, Failures, Performance, and more, which can be further modified and saved.
What’s next?
This is a rough guideline on how to deploy a GraphDB high-availability cluster. More technical details can be found in the GraphDB documentation and you can see all parameters in the GitHub repository. Also, we are active on GitHub and are interested in suggestions for improvements or new features.
If you don’t want to bother with deploying GraphDB yourself, contact Ontotext’s sales team and ask about the SaaS version. You can also buy it directly from the Azure Marketplace.
Stay tuned for the next post where we will cover the same process for AWS.
Maximize the power of your enterprise data with GraphDB on Azure
5
Share this article
