8. All command line utilities

The following is an overview of each of the ClusterVisor command line utilities outlining their purpose and usage (listed in alphabetical order).

cv-actutil-import

This is used to take the act_nodes.conf file from actutils2 and migrate those nodes into ClusterVisor. To use the tool, simply point to the configuration file and optionally use the --convert-eth flag to convert any ethX interfaces to be called enX instead.

cv-authsync

This is used to synchronize native users if ClusterVisor's LDAP server is not being used to manage users. For setup and additional information on how to use the utility, see section Synchronizing native users of this guide.

cv-auth-watchd

This is the daemon used with cv-authsync to automate running the utility whenever changes are made to the native users on the source node (which is explained further in section Synchronizing native users of this guide). The following flags also be can be used to adjust the behavior of the daemon:

• --debug - This can add additional debugging information.
• --authsync - This takes a path as an argument and is used to point to the cv-authsync executable if it exists in an unconventional location.
• --delay - This takes a number of seconds as an argument and can be used to have the daemon wait longer in between calling cv-authsync. This is used for when the watched files are changed rapidly and will throttle how often cv-authsync is called after each of those changes.

Do note, this command will usually not be called directly and one should instead just use the systemd cv-auth-watchd daemon.

cv-clientd

This is ClusterVisor's client daemon which sends and receives data from the ClusterVisor server daemon. By default, this will operate over the TCP port 13202, but it can be changed using cv-conf under config.global.cv-clientd.listen_port. The only configuration option the daemon provides is from the --no-reconfigure flag which will prevent it from re-configuring the node immediately after the daemon is up and running (which can be useful when troubleshooting ClusterVisor).

Do note, this command will usually not be called directly and one should instead just use the systemd cv-clientd daemon.

cv-cloner

Due to the complexity of ClusterVisor's cloning utility, it has its own section dedicated to explaining it and all of its sub-commands / flags. For more information, see the section Using cloner of this guide.

cv-cockpit-helper

The web interface for ClusterVisor is built atop Red Hat's Cockpit project and uses the cv-cockpit-helper utility to help facilitate interactions between the ClusterVisor server and Cockpit. Due to the nature of the utility, it will only pass back all data in JSON. Everything this utility does can be accomplished more easily by the other ClusterVisor command line utilities, so this utility should not be used directly.

cv-commit

This is used to commit saved changes in ClusterVisor's configuration so that the nodes can be reconfigured against the changes. The following controls are available through the flags:

• --show - Will display the queued changes that have not yet been committed
• --commit - Triggers the queued changes to be committed
• --status - Will display the status of any changes that failed to be applied to the nodes
• --clear - Will clear the failed changes from the queue

cv-conf

Any changes made to ClusterVisor's configurations can be made using cv-conf. The details of how to use the bulk of the utility can be found in the section Configuring nodes of this guide under the "Using the command line" sub-sections. The only feature not covered in those sections is the --ignore-lastupdate flag, which can be used to force the utility to save the changes made (even if someone else may have saved their own changes in the meantime). For this reason, that flag should be used only if absolutely necessary and otherwise avoided.

cv-console

Sometimes an SSH connection to a node is not enough and it would be help to actually see what would actually be displayed on a monitor for a node. The cv-console utility provides this feature by using the serial console provided through IPMI and allows for both observing and interacting with the node remotely (e.g. changing boot options in GRUB). While this command requires no arguments outside of the node selector(s) the other flags available are the following:

• --timeout - Determines how many seconds the utility will wait before giving up on making a connection to a node
• --escape-char - Which escape character will close the console session
• --serial - Instead of copying the files in parallel (the default), it will do one at a time for each node

cv-cp

The cv-cp utility can be used to copy anything stored on the current node to other nodes. The other nodes can be selected using the node selector flags (explained further in the "Using the command line" sub-section of the Accessing ClusterVisor section of this guide); and the source of the location and the desired destination location on the other nodes are specified, respectively, using the positional arguments of the utility. The other controls for the utility can accessed through the following flags:

• --timeout - Determines how many seconds the utility will wait before giving up on making a connection to a node
• --recursive - Used when copying directories, will recursively copy everything contained within the directory
• --preserve - Will apply both the ownership and permissions of the source file/directory to the copy
• --verbose - Displays all of the files/directories being copied to the nodes
• --serial - Instead of copying the files in parallel (the default), it will do one at a time for each node

cv-db-image

This provides a means to import to / export from the database. Regardless of whether importing or exporting, one of the requiring flags is needed to specify the connection info:

• --database-uri - Explicitly provides the connection URI of the MongoDB database
• --config-file - Points to the cv-serverd.conf file to extract the MongoDB connection information and database used
• --auto-detect - Will automatically detect the ClusterVisor database installed

Unless if the non-default database is being used or the config file has been moved from its default location, the --auto-detect flag is the correct choice. The --import-from and --export flags can then be used to specify the path to the file to import from or desired file name and path to export the database as a file to, respectively. Lastly, if importing over an existing database the --overwrite flag needs to be used to signal that the database should be overwritten.

cv-dumpcfg

The cv-conf utility has --dump to dump the entire contents of all configurations, cv-db-image can export the database as an image that can be used to import from at a later point, and cv-dumpcfg can be used to dump specific portions of the configurations in different formats. The scope of how much the utility will dump is determined by using one of the following:

• --collection - Will dump an entire collection
• --section - Will dump a specific section from a collection
• --all - Will dump all configurations

The --format flag is used to specify what format to dump the configurations as (YAML, TOML, JSON, or Apache config), and --type is used to determine if the config (the raw contents from the database) or expanded (the more verbose, post-processed form) type should be used. Using the expanded type can be particularly helpful for debugging the value of each node after all of the values are expanded and variables are replaced.

cv-exec

This command can be remotely executed across the nodes in the cluster using cv-exec. The positional arguments are used to specify the command and its arguments to run on the nodes, although quotes should be wrapped around the command if Unix pipes should be included with the command and not process the response from the cv-exec command (e.g. cv-exec -g nodes dmesg | grep error is not the same as cv-exec -g nodes "dmesg | grep error"). The following flags can be used to further control the cv-exec command:

• --format - Changes the format for how the results from the command is displayed
  • inline - Displays the node name and the output on the same line
  • header - Places the node name and the output on a different line
  • consolidate - Presents nodes with the same output as a single consolidated name to remove duplicate entries
  • realtime - Displays the output from nodes as they complete rather than all at once
  • json - Will present the response back as JSON so it can be parsed by another program
• --timeout - Determines how many seconds the utility should wait before giving up on making a connection to a node
• --serial - Instead of executing the commands in parallel (the default), it will do one at a time for each node
• --serialdelay - How many seconds to wait between running the command on each node in serial mode

cv-external-ipmi-statsd

By default, the cv-clientd daemon will collect the IPMI statistics from the node it is running on and dispatch it to the node running the ClusterVisor server. However, some other devices (e.g. node chassis) may also have IPMI data and to aggregate that data the cv-external-ipmi-statsd utility needs to be running on a node (preferrably, but not necessarily, the same one running cv-serverd). It will gather all of the known IPMI connection details for all of the non-node devices in the cluster and periodically collect and dispatch those statistics to the ClusterVisor server. The only flag it has is the --daemon that will run the process in background as a daemon.

cv-identify

To help make it easier to locate a single node from its peers, almost all chassis / sleds come with an ID LED on both the front and back of the node. This LED that can be enabled / disabled by either physically pressing the LED or issuing an IPMI command. The cv-identify utility allows for one or many nodes to have their ID LED enabled / disabled by using either the "on" or "off" actions, respectively. There is, unfortunately, no easy way to query the current status of the ID LED, but issuing the "off" action on all of the nodes using the --all node selector flag will disable any lingering ID LEDs.

cv-installer

The cv-installer is used to install and setup ClusterVisor. If launched without any arguments, it will enter an interactive prompt to guide the user through the install/setup options. However, to automatically select all of the default options in the installer the --auto-install flag can be passed to automatically fill out those questions. The following flags can also be used to control the installation process:

• --config-file - Used to optionally point to the cv-serverd.conf file to automatically obtain the MongoDB connection info being used for the install / setup (if left blank it will be asked in a prompt later or automatically populated if using --auto-install)
• --database-uri - Can optionally specify the MongoDB connection URI desired for ClusterVisor's database install target (if left blank it will be asked in a prompt later or automatically populated if using --auto-install)
• --skip - Will skip over sections of the installer (particularly if running the installer a second time or if only part of the installation is needed); the sections being munge, 389ds, influx, and mongo

The sections that the installer can install and setup, along with what is being done, are outlined below:

• munge - Because munge is a dependency of ClusterVisor it is already installed; so the installer will only setup munge by creating the munge key, setting the correct permissions to the key, and starting the munge daemon
• 389ds - This will install the 389 directory server for ClusterVisor, along with setting up the ClusterVisor domain controller and the directory manager user
• influx - InfluxDB is needed for storing the stats data from the nodes; so this section will add the InfluxDB repository, install the InfluxDB package, and setup the user credentials and security settings for the Influx database
• mongo - MongoDB is needed as the primary database for ClusterVisor; so this section will add the MongoDB repository, install the MongoDB package, and setup the user credentials and security settings for the Mongo database

cv-monitoring

The monitored stats from the nodes in the cluster can be viewed via the cv-monitoring utility. By default, they will be displayed with the monitored stat as the header and the value of that stat for each node listed below the stat. This can be inverted to see each node as the header and its respective monitored stat values listed beneath it by using the flag --format=bynode.

cv-netboot

While the netboot field of a node can be changed manually using cv-conf, it can also be quickly observed and changed using the cv-netboot utility. The current netboot value of the node(s) can be queried using the --get flag while it can be set to a new value using the --set value, whose argument will be used to replace the existing netboot value of the node(s).

cv-nodenames

This simple utility is used to display all of the node names within the given node selection. For individual node selections this holds little value, but for selections using groups or chassis this can be helpful to see exactly what nodes are included in the selection. By default, each node name will be listed on its own line, but the behavior changed using the following flags:

• --delimiter - Specify what will separate each node name entry
• --format - This can be used to get the output in JSON (by default it is in plain-text)

cv-power

The power for the node(s) can be remotely managed using the cv-power utility. Internally, this uses IPMI so it will work on the node as long IPMI is setup on the nodes and power is connected to the nodes. The following power operations can be performed by the utility:

• status - Query the current power status of the node(s)
• on-hours - How many hours the node(s) have been powered on
• restart-cause - The cause of the last restart performed by the node(s)
• on - Attempts to remotely power on the node(s)
• soft-off - Attempts to softly power off the node(s), is effectively the equivalent of pressing the power button once on a node
• off - Attempts to remotely power off the node(s), is effectively the equivalent of pressing and holding the power button on a node (only use if the node is locked without any other way to power off)
• reset - Attempts to remotely restart the node(s)
• cycle - Attempts to remotely power cycle the node(s)

The behavior of the utility can be adjusted using the following flags:

• --debug - Will display the debug output from the utility
• --timeout - How many seconds the utility will wait for a response from the node before giving up (default is 5 seconds, but any duration between 2-60 seconds can be used)
• --format - Can change the format the output is presented in; "inline" to put the node name and output on the same line, "consolidate" to merge node names with identical outputs into a single entry, and "json" to present the output in JSON
• --format - Changes the format for how the results from the command is displayed
  • inline - Displays the node name and the output on the same line
  • consolidate - Presents nodes with the same output as a single consolidated name to remove duplicate entries
  • json - Will present the response back as JSON so it can be parsed by another program

cv-python

This is just the Python binary from the virtual environment ClusterVisor uses to load some of its dependencies. Unless debugging ClusterVisor, there is no reason to use this.

cv-reconfigure

The nodes can be manually reconfigured using the current configurations in ClusterVisor using the cv-reconfigure utility. The only flag that changes the behavior of the utility is the --yes flag, which will automatically agree to any prompts from the utility rather than needing to manually fill them out.

cv-sel

The IPMI interface of a node will over time accumulate hardware events that have occurred on the node that can later be use for troubleshooting / diagnosing. IPMI calls this list of events the "System Event Log" (a.k.a. SEL) and they can be easily viewed from one or many nodes using the cv-sel utility. Running the utility only specifying the desired nodes to pull the entries from will list all of the events logged on each node, but this can be refined down by using the following flags:

• --level - By default all event levels are shown but can be filtered to display only "critical", "nominal", or "warning" level events
• --search - Will filter out all but entries that match the provided regular expression
• --head - Will display the number of events specified starting from the top
• --tail - Will display the number of events specified starting from the bottom

The other flags available are:

• --timeout - How many seconds the utility will wait for a response from the node before giving up (default is 5 seconds, but any duration between 2-60 seconds can be used)
• --format - Changes the format for how the results from the command is displayed
  • inline - Displays the node name and the output on the same line
  • header - Places the node name and the output on a different line
  • json - Will present the response back as JSON so it can be parsed by another program
• --clear - The SEL can only hold a limited number of lines and can be manually cleared if it is filled using this flag

cv-sensor

The IPMI interface of a node is always collecting data from its various sensors (e.g. temperatures and fan speeds) and the current values of those sensors can be queried using the cv-sensor utility for one or many nodes. Running the utility only specifying the desired nodes to pull the sensor data from will list all of the sensors on each node, but this can be refined down by using the following flags:

• --type - Will filter out all but sensors whose type matches the provided regular expression(e.g. temperatures and fans)
• --level - By default all sensor levels are shown but can be filtered to display only "critical", "nominal", or "warning" level sensors
• --name - Will filter out all but sensors whose name matches the provided regular expression
• --value-gt - Will filter out all but sensors whose value is greater than the provided value
• --value-lt - Will filter out all but sensors whose value is less than the provided value

All other flags available are:

• --timeout - How many seconds the utility will wait for a response from the node before giving up (default is 5 seconds, but any duration between 2-60 seconds can be used)
• --format - This can be used to get the output in JSON (by default it is in plain-text)
• --thresholds - Will also display the threshold for each sensor before they are in a critical state
• --value-mode - Can be used to only show the values rather than highlighting them

cv-serveradm

Changes can be made to the internal settings of the ClusterVisor server daemon, cv-serverd, using the cv-serveradm utility. This tool is primarily used for debugging problems with cv-serverd without needing to restart it, which can be done using the following flags:

• --dump-cache - Can present the plugin data on the server for either the cache of fields that will trigger plugins or the cache of init instructions that gets sent to each node on startup (or both if "all" is passed)
• --flush-cache - Will instruct the server to flush out the cache of either the trigger or init cache and will then re-generate it (do note, this task will take a few seconds to complete on the server end and will be CPU intensive during that time)
• --field-triggers - Will present the fields that trigger plugins from a given collection
• --plugin-cache - Will present the plugin instructions cached for a given node
• --debug-level - Can query or change the debug level (by default it should be 0, anything above 0 will enable verbose logging of exceptions)
• --mem-dump - Can query, enable, or disable the memory dump mode of the server's logging (by default it should be disabled)
• --function-timer - Can query, enable, or disable the function timer mode of the server's logging (by default it should be disabled)

This tool can also be used to manually send HTTP GET or POST requests to the server using the following flags:

• --get-api - Will return the response from the server given the URL for an HTTP GET route from the server's API
• --post-api - Will return the response from the server given the URL for an HTTP POST route from the server's API

When using the API flags, the following flags can optionally be used:

• --output - Determines whether the response from the server will be returned as raw JSON, prettified JSON, or YAML (will use prettified JSON by default)
• --server - Manually specify what server address to connect to
• --port - Manually specify what server port to connect to
• --post-input - Point to a file with the HTTP POST data to send with the request
• --parallel - How many parallel requests should be made to the API
• --hide-time - Hides the timing output of the threads from the response

cv-serverd

This is the ClusterVisor server daemon, which will communicate with the other components that make up ClusterVisor (e.g. the client daemon, web interface, etc). By default, this will operate over the TCP port 13201, but can be changed in the configuration file under /etc/clustervisor/cv-serverd.conf by altering the value of port in the file. The daemon can provide further logging information by using the --debug flag to raise the debug level of the daemon (by default this is 0, but can be set from 1-4).

Do note, this command will usually not be called directly, and should instead just use the systemd cv-serverd daemon.

cv-stats

The stats from the nodes can be viewed using the cv-stats utility. It will display all of the stats collected by ClusterVisor for all of the selected nodes. To change the behavior of the utility, use the following flags:

• --regex - Use a regular expression to filter the stats that are presented
• --format - Can change the format the results are presented in to be consolidate (where nodes with the same stat value will be consolidated into a single entry) or JSON

cv-useradm

The users managed by ClusterVisor can be handled using the cv-useradm utility. The usage of this utility is explained in the "Using the command line" sub-sections of the Managing users section of this guide.