Configuration Files Details

Overview

The ˜/qarbine.service/config folder contains several configuration files. Many of these use JSON to describe the settings. The JSON file has several sections which are described below.

The “main'' files are only used on the main node whereas the “compute” support files are used on all Qarbine compute nodes. Multiple Qarbine compute nodes may be wanted when distributing load and locating a node closer to the data it is meant to process. You may also configure multiple endpoints on the same host by using different port numbers.

The usage of the various config folder files are described below.

File	Resides	Comment
cfg.shared.json	Main	Share configuration settings.
env.common.txt	All	General environment variables for any compute node.
env.main.txt	Main	Environment variable only for the main node.
env.supportMainHost.txt	Main	Environment variables for support endpoint running on the main compute node.
jwtPrivate.key	Main	The key used to encrypt JWT tokens.
jwtPublic.key	All	The key used to decrypt JWT tokens by any compute node.
mainLicense.txt	Main	The license file.
qarbine.crt	Main	SSL private certificate. The internal name must match the host. It is used by the Apache web server and the Qarbine endpoint.
qarbine.key	All	SSL public certificate.
service.main.json	Main	The main compute node configuration.

Production HTTPS Certificates

The certificates used by production hosts should be signed by a valid certificate authority. This avoids web browser issues. There are several references and locations for these files.

• There are certificate references in /etc/sysconfig/qarbineCustomVariables. The Apache account must have permissions to the referenced files,

export CERT_FILE = /var/www/qarbine/qarbine.crt
export CERT_KEY_FILE = /var/www/qarbine/qarbine.key

• There are certificate references in ~/qarbine.service/config/env.common.txt. The general purpose account must have access to the referenced files. Typically these files are copies of the ones used by Apache, but with other account permissions. They typically reside in the ~/qarbine.service/config folder.

SSL_CERTIFICATE = qarbine.crt
SSL_PRIVATE_KEY = qarbine.key

Node Name

Within a service.NODE_NAME.json file the first area is generally.

"comment" : "The presentation name of the node",
"name" : "BigIronBox",

The name is the user-friendly text shown in various list dialogs. For example, to choose which Qarbine compute node you would like a request to be carried out. It is independent of the DNS name.

Network Location

The next section of the JSON file relates to network location information in the service.*.json file.

"comment" : "A node is either 'inside' the SAME network as the main node, 'outside', or 'isMain'",
"relativeToMain" : "isMain",

Only the service.main.json file has a relativeToMain value of ‘isMain’. Secondary nodes may reside ‘inside’ the same network as the main node or ‘outside’ it. Use ‘inside’ if the compute node is visible within the main’s network. Use ‘outside’ if the public or some other routing is to be used.

Plugins

The next section lists the plugins to run on this node. All nodes must have the monitoring functions installed.

"./plugin/monitoringFunctions.js",

The main endpoint always has:

"./plugin/accessFunctions.js",
"./plugin/catalogFunctions.js",
"./plugin/helpFunctions.js",
"./plugin/userProfileFunctions.js"
"./plugin/systemAdministrationFunctions.js",
"./plugin/queryFunctions.js",

The plugins on the other endpoints may vary based on your requirements. The order they are listed does not matter.

"comment" : "These are the functions to load for this endpoint.",
"comment" : "To not load either set the first character be # or remove the line.",
"plugins" :[
    "./plugin/monitoringFunctions.js",
    "# Only on main",
    "./plugin/accessFunctions.js",
    "./plugin/catalogFunctions.js",
    "./plugin/helpFunctions.js",
    "./plugin/userProfileFunctions.js",
    "./plugin/systemAdministrationFunctions.js",
    "# At least on main",
    "./plugin/queryFunctions.js",
    "# Optionally on any",
    "./plugin/proxyFunctions.js",
    "./plugin/taskRequestFunctions.js"
 ],

Data Drivers

The next section lists the supporting data drivers to enable on this node. Order does not matter. All endpoints always have:

"./driver/mongoDriver.js",

A sample set of drivers is shown below.

"comment" : "These are the drivers to load to support the functions of this endpoint",
"comment" : "To not load either set the first character be # or remove the line.",
"drivers" : [
    "# At least on main",
    "./driver/mongoDriver.js",
    "# Optionally on any",
    "./driver/atlasApiDriver.js",
    "./driver/dynamoDbDriver.js",
    "./driver/fileDriver.js"
  ]

Physical Location

The next section of the JSON file relates to the physical location information in the service.NODE_NAME.json file. The location information includes geographic longitude and latitude. It is automatically determined when running in a hypercloud service provider such as Azure or AWS .

"comment" : "Set to be empty if the compute node is running in AWS, Azure, or GCP",
"location" : {
"vendor" : "BigRed",
"region" : "DC-West coast",
"latitude": 47.6062095,
"longitude": -122.3320708
},

The service.*.json files for the same host will have the same location information.

Determining Non-Hypercloud Location Information

Depending on the Qarbine edition, release and deployment topology, the location information can be used to prompt for where to run tasks. It is also used by several report templates and Qarbine operational status pages. One template uses this information to display a map of the compute nodes for example. You can manually add it to the configuration file. The location section of the configuration file includes “vendor”, region, longitude, and latitude fields. A sample entry is shown below.

"location" : {
"vendor": "In-house",
"region": "NY Metro",
"longitude": -74.0059728,
"latitude": 40.7127753
},

Also verify the display name in the configuration file. This is used to prefix the compute node’s URL address in various pages.

"name": "Eastern US"

Online Geo Coordinate Resource

There are several online resources to help determine geographic coordinates. Here are steps describing using one of them.

Navigate to

https://www.whatsmygps.com/

Click

  

Enter an address into the entry field.

  

Click

  

Copy the coordinates from the output.

  

With an Existing Qarbine Installation

When deploying secondary compute nodes the geographic information can be looked up using a utility in the System Administration Tool. In the System Administration window activate the Utilities tab.

  

Choose “Lookup location” from the drop down.

  

Enter the address in the dialog.

  

Click OK.
Sample output is shown below.

  

Appendix

Primary Node’s Configuration

Environment Variables File

The ./config/env.main.txt file is used by the primary service endpoint.

RELATIVE_TO_MAIN = isMain
# Node environment variables
JWT_SECRET=aaaa.bbbb.cccc.dddd.eeee.ffff.gg
JWT_DURATION=12h
# For post log on redirection.
REDIRECT_BASE = https://%MAIN_COMPUTE_HOST%/app
MAIN_DB_PORT = 37000
INTERNAL_OPTIONS=
COMPUTE_CONNECTION=mongodb://q_compute:zPassword@%MAIN_COMPUTE_HOST%:%MAIN_DB_PORT%
COMPUTE_OPTIONS=
CONFIG_CONNECTION=mongodb://q_config:zPassword@%MAIN_COMPUTE_HOST%:%MAIN_DB_PORT%
CONFIG_OPTIONS=
CATALOG_CONNECTION=mongodb://q_catalog:zPassword@%MAIN_COMPUTE_HOST%:%MAIN_DB_PORT%
CATALOG_OPTIONS=
PWD_SECRET = aaaa.bbbb.cccc.dddd.eeee.ffff.gg

JWT_SECRET	This is a 32 character long string.
COMPUTE_CONNECTION	At startup, the given Qarbine database account name will have its password updated to the specified value. Note changing this value with other compute nodes running will invalidate their database connection. Restart the compute nodes from the Qarbine Administration tool.
CONFIG_CONNECTION	As above.
CATALOG_CONNECTION	As above.
PWD_SECRET	This is a 32 character long string.

Secondary Compute Node Configurations

Overview

A compute node may be configured to have one or more endpoints. This can be beneficial when balancing different loads. You may also configure Qarbine Data Services to point to different endpoints on the same host perhaps based on the data being accessed.

Environment Variables File

The ˜/qarbine.service/config/env.common.txt is shared across all Qarbine endpoint processes.

MAIN_COMPUTE_HOST= myhost.ddns.net
MAIN_COMPUTE_URL = %PROTOCOL%://%MAIN_COMPUTE_HOST%
SSL_PRIVATE_KEY=qarbine.key
SSL_CERTIFICATE=qarbine.crt
# Uses alias 'QarbineFirstNoninternalIpAddress' which is determined at startup
PRIVATE_BASE_URL = https://QarbineFirstNoninternalIpAddress
# To fetch headless images
BASE_FILE_URL = https://%MAIN_COMPUTE_HOST%/app
HTTP_ALLOW_ORIGIN = *
MAIN_SERVICE_PORT = 4000
MAIN_SERVICE_URL = %MAIN_COMPUTE_URL%:%MAIN_SERVICE_PORT%/dispatch
PRIVATE_BASE_URL= %PROTOCOL%://QarbineFirstNoninternalIpAddress
PRIVATE_MAIN_SERVICE_URL=%PRIVATE_BASE_URL%:%MAIN_SERVICE_PORT%/dispatch
API_KEY=fcbf5b3e844fc…
BOOTSTRAP_URL = %MAIN_SERVICE_URL%?action=getSharedConfiguration&apiKey=%API_KEY%

Environment Variable	Description
MAIN_COMPUTE_HOST	The DNS name or IP of the main Qarbine server.
SSL_PRIVATE_KEY	The SSL private certificate for this node. The default name is qarbine.key.
SSL_CERTIFICATE	The SSL private certificate for this node. It should not be self-signed as many web browsers reject these unless their configuration is changed. The default name is qarbine.crt.
THIS_COMPUTE_HOST	The public IP that web browsers may use to access this compute node.
MAIN_SERVICE_PORT	The port number of the main Qarbine server’s access service.
COMPUTE_SERVICE_X_PORT	Since there may be multiple compute services running on a compute node, the env.common.txt file is shared across them. It is a convenient spot to define most values.

Compute Node JSON File

Edit config/service.NODE_NAME.json. There is one of these files per endpoint.

"network" :
{
"relativeToMain" : "outside",
"publicHost" : "myOtherQarbineHost.acme.com",
"port" : "%COMPUTE_SERVICE_1_PORT%",
"protocol" : "https"
},
"comment" : "This is the name of this configuration entry",
"name": "Analytics4",

Setting	Description
relativeToMain	The relativeToMain value may be either “outside” or “inside”. Use the latter when this compute node is within the same VPC as the main Qarbine server. This information is used for heartbeats from the main server.
publicHost	Specify either the DNS name or the public IP address.
name	Provide a name to easily distinguish this compute node from others in variable listings. This name appears in the Administrator tool and in the compute node prompts.