3. Remote management
The previous section mentions the tabs for monitoring the node. There are other tabs that provide controls for remotely managing the selected node along with any other node managed by ClusterVisor in the cluster. Those tabs are the following:
- Execute command - Provides a way to quickly run a command on itself and/or multiple other nodes and get back the result of the command. The nodes can be selected individually by name, as well as in bulk using the nodes' group name. The results will all be listed with the most recently executed command at the top and with all the previous outputs following beneath it in descending order.
.png?inst-v=e35f0730-77e9-4077-8e2d-09844b127151)
- Boot/Power control - Similar to Execute command, nodes can be selected individually by name and/or in bulk using group names. However, instead of executing commands on the nodes, this interface provides the ability to set a node's network boot image and/or control the power of the node through SSH or IPMI as well as view if the node is accessible through SSH or IPMI.
These controls can also be accessed from the Execute command page and Boot / Power control page in ClusterVisor.
For Boot / Power control, the distinction between using SSH or IPMI is that SSH requires the node to be online and can only power off or restart the node, while IPMI can power on, power off, and restart the node regardless of if the node is powered off or powered on. However, also keep in mind that the power controls through SSH are the safe means of powering down since it is being done through the OS, while the "Power Off" option from IPMI is a "hard" shutdown that is equivalent to pulling the power from the node (although this may be needed if the node becomes completely unresponsive). However, the "Soft Off" IPMI option from IPMI will deliver the equivalent of once pressing the power off button on the node to instruct it to begin shutting down, which would be useful to safely power off the node where it is not locked but also cannot be reached via SSH.
Do note that the IPMI power controls may take a few seconds to show up, since the test for IPMI connectivity is not immediate. However, the controls have not shown up after a minute has passed, this likely means that IPMI is either not setup on the node or is no longer working properly. If that is the case, see the Troubleshooting section below for how to handle that issue.
For changing the node's network boot image, select the desired image under the node's Netboot column and it will be used the next time the node is rebooted. To view or change the available network boot images navigate to the Netboot tab under the Configuration page (for more on this see section 4. Configuring nodes of this guide).
There are two other places in ClusterVisor that can be used to remotely manage the cluster, the Log viewer page and the Terminal page. The former can be used to view various log files of nodes, while the latter provides a terminal window to run commands on the node in real-time (this being different from Execute command because the commands can be run interactively and use tools like vim to edit files). Both pages provide a drop-down menu to select the desired node to interact with. As well, the Log viewer page contains the log files for ClusterVisor, so it can be used for debugging if something is not working properly.
Using the command line
The command line utilities responsible for executing commands and power control are cv-exec and cv-power.
The cv-exec command will execute any shell command provided to it onto the selected nodes and will display their outputs with each node name in brackets with the output of the command listed below. If any commands ran into an error, beneath all of the outputs will be a separate section with the node name and errors listed together.
The cv-power command can access the current power status of the selected nodes, how long the nodes have been online, why the nodes were last restarted, and controls for remotely powering on/off the nodes. This is done through IPMI, so the same concerns to keep in mind for hard powering off the nodes still apply, but the "soft-off" option explained above is still available.
The cv-netboot command can view or change the network boot image of the selected nodes.
Since the cv-exec internally communicates to the nodes through SSH, it would also be the utility used to perform the same SSH power control actions as the web interface. This would be accomplished by passing the nodes either the "shutdown", "poweroff", or "reboot" command.
Lastly, the log files for ClusterVisor by default are stored under /var/log/clustervisor. Each component is listed as a separate ".log" file and can be viewed like any other log file from the command line using utilities like cat, tail, less, and so on.