Checking and Clearing InfiniBand Errors

An easy way to check for errors on your entire cluster IB network is to run the command ‘ibcheckerrors.’ If you need to install it: dnf install infiniband-diags-compat Also, infiniband-diags has the command 'ibqueryerrors' which replaces the depreciated ‘ibcheckerrors’ command.

This will print any errors that can range from a port being down (even just unplugged temporarily) to transmission errors. After troubleshooting any errors you find, you can clear out the error counters with the command ‘ibclearerrors’ (Depreciated see Red Hat : Please use ibqueryerrors -k instead of ibclearerrors command. Please use ibqueryerrors -K instead of ibclearcounters command.).
(Note: most IB errors can be resolved by reseating the cables on both ends for any ports that showed errors.)
 
The output of ‘ibcheckerrors’ can be confusing when you’re trying to determine which physical ports the errors are happening on. A good way to see which ‘lid’ an error happens on is by using the tool ‘ibnetdiscover’ to print out your entire, active IB network layout.