An easy way to check for errors on your entire cluster IB network is to run the command ‘ibcheckerrors.’

This will print any errors that can range from a port being down (even just unplugged temporarily) to transmission errors. After troubleshooting any errors you find, you can clear out the error counters with the command ‘ibclearerrors’.
(Note: most IB errors can be resolved by reseating the cables on both ends for any ports that showed errors.)
The output of ‘ibcheckerrors’ can be confusing when you’re trying to determine which physical ports the errors are happening on. A good way to see which ‘lid’ an error happens on is by using the tool ‘ibnetdiscover’ to print out your entire, active IB network layout.