When working with clustered nodes you will often want to run the same command on all the nodes in the system.  This may be to copy a file, start or stop a daemon or any type of operation.  On Linux there are many distributed shells that are designed to do just this.  But often I find it easier just run the list of nodes in to a for loop to do much the same thing.

Prerequisite check – the below tutorial assumes that you have password free root access via ssh keys to the clustered nodes.

To do this I first make a file called "nodes.list", this is a text file with the hostnames of all the nodes in the cluster.  I always find that it is best to put one hostname per line.  So the file may look like this. 

node001.mydomain

node002.mydomain

node003.mydomain

node004.mydomain

Store the file somewhere safe such as ~/cluster/nodes.list.  Next create a bash variable that contains the list.  I normally do this by adding the following lines in to ~/.bashrc

NODES="`cat ~/cluster/nodes.list`"

export NODES

After adding the variable to ~/.bashrc you can source it to import the changes to the file

# source ~/.bashrc

Check that this has worked by printing the variable.# echo $NODES

 If you printed the list of nodes then you are ready to continue.

With the variable wet you can start to issue commands to all the nodes.  For example, to print the time and date on all nodes you can run the following command.

 # for i in $NODES ; do ssh root@$i date ; done

You can also run multiple commands and use pipes.  For example, say you wanted to grep /var/log/messages for a pattern and display the last 5 results on each node.  You could run this command.

 # for i in $NODES ; do ssh root@$i "(grep kernel /var/log/messages | tail –n 5)" ; done

It is also nice to print the hostname of each node before the output for that node.  A simple way of printing the hostname of a system is running the hostname command with no arguments.  But be warned, if you are not careful it is easy to accidently rename all the nodes in the cluster, which is not advised.  That is why I prefer to print the $HOSTNAME variable.  Just insert a backslash in front of the dollar sign so the variable isn't substituted by the local host.  You can also add a trailing blank line so the output is nicely formatted.

 # for i in $NODES ; do ssh root@$i "(echo "$HOSTNAME" ; grep kernel /var/log/messages | tail –n 5 ; echo)" ; done

Please leave a comment if you found this article of use.