Configuration Files Details
Overview
The ˜/qarbine.service/config folder contains several configuration files. Many of these use JSON to describe the settings. The JSON file has several sections which are described below.
The “main'' files are only used on the main node whereas the “compute” support files are used on all Qarbine compute nodes. Multiple Qarbine compute nodes may be wanted when distributing load and locating a node closer to the data it is meant to process. You may also configure multiple endpoints on the same host by using different port numbers.
The usage of the various config folder files are described below.
File | Resides | Comment |
---|---|---|
cfg.shared.json | Main | Share configuration settings. |
env.common.txt | All | General environment variables for any compute node. |
env.main.txt | Main | Environment variable only for the main node. |
env.supportMainHost.txt | Main | Environment variables for support endpoint running on the main compute node. |
jwtPrivate.key | Main | The key used to encrypt JWT tokens. |
jwtPublic.key | All | The key used to decrypt JWT tokens by any compute node. |
mainLicense.txt | Main | The license file. |
qarbine.crt | Main | SSL private certificate. The internal name must match the host. It is used by the Apache web server and the Qarbine endpoint. |
qarbine.key | All | SSL public certificate. |
service.main.json | Main | The main compute node configuration. |
Production HTTPS Certificates
The certificates used by production hosts should be signed by a valid certificate authority. This avoids web browser issues. There are several references and locations for these files.
- There are certificate references in /etc/sysconfig/qarbineCustomVariables. The Apache account must have permissions to the referenced files,
export CERT_FILE = /var/www/qarbine/qarbine.crt
export CERT_KEY_FILE = /var/www/qarbine/qarbine.key
- There are certificate references in ~/qarbine.service/config/env.common.txt. The general purpose account must have access to the referenced files. Typically these files are copies of the ones used by Apache, but with other account permissions. They typically reside in the ~/qarbine.service/config folder.
SSL_CERTIFICATE = qarbine.crt
SSL_PRIVATE_KEY = qarbine.key
Node Name
Within a service.NODE_NAME.json file the first area is generally.
"comment" : "The presentation name of the node",
"name" : "BigIronBox",
The name is the user-friendly text shown in various list dialogs. For example, to choose which Qarbine compute node you would like a request to be carried out. It is independent of the DNS name.
Network Location
The next section of the JSON file relates to network location information in the service.*.json file.
"comment" : "A node is either 'inside' the SAME network as the main node, 'outside', or 'isMain'",
"relativeToMain" : "isMain",
Only the service.main.json file has a relativeToMain value of ‘isMain’. Secondary nodes may reside ‘inside’ the same network as the main node or ‘outside’ it. Use ‘inside’ if the compute node is visible within the main’s network. Use ‘outside’ if the public or some other routing is to be used.
Plugins
The next section lists the plugins to run on this node. All nodes must have the monitoring functions installed.
"./plugin/monitoringFunctions.js",
The main endpoint always has:
"./plugin/accessFunctions.js",
"./plugin/catalogFunctions.js",
"./plugin/helpFunctions.js",
"./plugin/userProfileFunctions.js"
"./plugin/systemAdministrationFunctions.js",
"./plugin/queryFunctions.js",
The plugins on the other endpoints may vary based on your requirements. The order they are listed does not matter.
"comment" : "These are the functions to load for this endpoint.",
"comment" : "To not load either set the first character be # or remove the line.",
"plugins" :[
"./plugin/monitoringFunctions.js",
"# Only on main",
"./plugin/accessFunctions.js",
"./plugin/catalogFunctions.js",
"./plugin/helpFunctions.js",
"./plugin/userProfileFunctions.js",
"./plugin/systemAdministrationFunctions.js",
"# At least on main",
"./plugin/queryFunctions.js",
"# Optionally on any",
"./plugin/proxyFunctions.js",
"./plugin/taskRequestFunctions.js"
],
Data Drivers
The next section lists the supporting data drivers to enable on this node. Order does not matter. All endpoints always have:
"./driver/mongoDriver.js",
A sample set of drivers is shown below.
"comment" : "These are the drivers to load to support the functions of this endpoint",
"comment" : "To not load either set the first character be # or remove the line.",
"drivers" : [
"# At least on main",
"./driver/mongoDriver.js",
"# Optionally on any",
"./driver/atlasApiDriver.js",
"./driver/dynamoDbDriver.js",
"./driver/fileDriver.js"
]
Physical Location
The next section of the JSON file relates to the physical location information in the service.NODE_NAME.json file. The location information includes geographic longitude and latitude. It is automatically determined when running in a hypercloud service provider such as Azure or AWS .
"comment" : "Set to be empty if the compute node is running in AWS, Azure, or GCP",
"location" : {
"vendor" : "BigRed",
"region" : "DC-West coast",
"latitude": 47.6062095,
"longitude": -122.3320708
},
The service.*.json files for the same host will have the same location information.
Determining Non-Hypercloud Location Information
Depending on the Qarbine edition, release and deployment topology, the location information can be used to prompt for where to run tasks. It is also used by several report templates and Qarbine operational status pages. One template uses this information to display a map of the compute nodes for example. You can manually add it to the configuration file. The location section of the configuration file includes “vendor”, region, longitude, and latitude fields. A sample entry is shown below.
"location" : {
"vendor": "In-house",
"region": "NY Metro",
"longitude": -74.0059728,
"latitude": 40.7127753
},
Also verify the display name in the configuration file. This is used to prefix the compute node’s URL address in various pages.
"name": "Eastern US"
Online Geo Coordinate Resource
There are several online resources to help determine geographic coordinates. Here are steps describing using one of them.
Navigate to
https://www.whatsmygps.com/
Click
Enter an address into the entry field.
Click
Copy the coordinates from the output.
With an Existing Qarbine Installation
When deploying secondary compute nodes the geographic information can be looked up using a utility in the System Administration Tool. In the System Administration window activate the Utilities tab.
Choose “Lookup location” from the drop down.
Enter the address in the dialog.
Click OK.
Sample output is shown below.
Appendix
Primary Node’s Configuration
Environment Variables File
The ./config/env.main.txt file is used by the primary service endpoint.
RELATIVE_TO_MAIN = isMain
# Node environment variables
JWT_SECRET=aaaa.bbbb.cccc.dddd.eeee.ffff.gg
JWT_DURATION=12h
# For post log on redirection.
REDIRECT_BASE = https://%MAIN_COMPUTE_HOST%/app
MAIN_DB_PORT = 37000
INTERNAL_OPTIONS=
COMPUTE_CONNECTION=mongodb://q_compute:zPassword@%MAIN_COMPUTE_HOST%:%MAIN_DB_PORT%
COMPUTE_OPTIONS=
CONFIG_CONNECTION=mongodb://q_config:zPassword@%MAIN_COMPUTE_HOST%:%MAIN_DB_PORT%
CONFIG_OPTIONS=
CATALOG_CONNECTION=mongodb://q_catalog:zPassword@%MAIN_COMPUTE_HOST%:%MAIN_DB_PORT%
CATALOG_OPTIONS=
PWD_SECRET = aaaa.bbbb.cccc.dddd.eeee.ffff.gg
JWT_SECRET | This is a 32 character long string. |
---|---|
COMPUTE_CONNECTION | At startup, the given Qarbine database account name will have its password updated to the specified value. Note changing this value with other compute nodes running will invalidate their database connection. Restart the compute nodes from the Qarbine Administration tool. |
CONFIG_CONNECTION | As above. |
CATALOG_CONNECTION | As above. |
PWD_SECRET | This is a 32 character long string. |
Secondary Compute Node Configurations
Overview
A compute node may be configured to have one or more endpoints. This can be beneficial when balancing different loads. You may also configure Qarbine Data Services to point to different endpoints on the same host perhaps based on the data being accessed.
Environment Variables File
The ˜/qarbine.service/config/env.common.txt is shared across all Qarbine endpoint processes.
MAIN_COMPUTE_HOST= myhost.ddns.net
MAIN_COMPUTE_URL = %PROTOCOL%://%MAIN_COMPUTE_HOST%
SSL_PRIVATE_KEY=qarbine.key
SSL_CERTIFICATE=qarbine.crt
# Uses alias 'QarbineFirstNoninternalIpAddress' which is determined at startup
PRIVATE_BASE_URL = https://QarbineFirstNoninternalIpAddress
# To fetch headless images
BASE_FILE_URL = https://%MAIN_COMPUTE_HOST%/app
HTTP_ALLOW_ORIGIN = *
MAIN_SERVICE_PORT = 4000
MAIN_SERVICE_URL = %MAIN_COMPUTE_URL%:%MAIN_SERVICE_PORT%/dispatch
PRIVATE_BASE_URL= %PROTOCOL%://QarbineFirstNoninternalIpAddress
PRIVATE_MAIN_SERVICE_URL=%PRIVATE_BASE_URL%:%MAIN_SERVICE_PORT%/dispatch
API_KEY=fcbf5b3e844fc…
BOOTSTRAP_URL = %MAIN_SERVICE_URL%?action=getSharedConfiguration&apiKey=%API_KEY%
Environment Variable | Description |
---|---|
MAIN_COMPUTE_HOST | The DNS name or IP of the main Qarbine server. |
SSL_PRIVATE_KEY | The SSL private certificate for this node. The default name is qarbine.key. |
SSL_CERTIFICATE | The SSL private certificate for this node. It should not be self-signed as many web browsers reject these unless their configuration is changed. The default name is qarbine.crt. |
THIS_COMPUTE_HOST | The public IP that web browsers may use to access this compute node. |
MAIN_SERVICE_PORT | The port number of the main Qarbine server’s access service. |
COMPUTE_SERVICE_X_PORT | Since there may be multiple compute services running on a compute node, the env.common.txt file is shared across them. It is a convenient spot to define most values. |
Compute Node JSON File
Edit config/service.NODE_NAME.json. There is one of these files per endpoint.
"network" :
{
"relativeToMain" : "outside",
"publicHost" : "myOtherQarbineHost.acme.com",
"port" : "%COMPUTE_SERVICE_1_PORT%",
"protocol" : "https"
},
"comment" : "This is the name of this configuration entry",
"name": "Analytics4",
Setting | Description |
---|---|
relativeToMain | The relativeToMain value may be either “outside” or “inside”. Use the latter when this compute node is within the same VPC as the main Qarbine server. This information is used for heartbeats from the main server. |
publicHost | Specify either the DNS name or the public IP address. |
name | Provide a name to easily distinguish this compute node from others in variable listings. This name appears in the Administrator tool and in the compute node prompts. |