How I built a gpfs cluster

Scenario: I will build the initial cluster using 2 Shared SAN disks connected to two lpars of a
P550 Server and then I will add another node from a p550 Lpar with no SAN disk and will create
few sets of GPFS filesystem using internal disks. Later I will add a new disk to the 1st GPFS
filesystem. If time permits I will add at least one more node to the cluster. So my final cluster
will have one primary configuration server node one secondary configuration server node.

You will notice that sometimes I ran commands from different server. It really doesn't matter
for GPFS as you can run any command from any node. I had several ssh session open so whatever
window I found open I used that window.

Step 1. I will install GPFS software and latest fix packs to the 1st node which will be primary
configuration server too. I copied all the GPFS software and downloaded latest available fix
packs to /utility/SOFT/gpfs directory. Here is the list of all the fileset.

Now I will run

Result below

Now I will do the same in 2nd node now.

Here is the list of all disks in p550a_lp1 ( 1st node). I will use the highlighted disks
for now. These disks are shared disks in both p550b_lp01 and p550b_lp02 node.

1st node

And in 2nd node.

Now I can match the serial no. of each disk.

In 2nd node

Now I know that hdisk4 in p550b_lp01 and hdisk6 in p550b_lp02 is same disk and I will note
it down. Similarly I will check the 2nd San disk too.

In 2nd node

Now we see that hdisk5 on p550b_lp01 is hdisk7 on p550b_lp02. I will check their size as
I will use one of them as tie breaker disk. I will check them from any of the two nodes.

The disk which is hdisk6 in p550b_lp02 and hdisk4 in p550b_lp01 is 1GB and I will use as
tie-breaker disk.

I have collected enough information to create the initial configuration. In next step
what I need to do is ssh password less root login from each node to all nodes. I will
not show those steps here as every System Administrator knows how to do it.

So after enabling password less root login from every node to every node I will add t
he /usr/lpp/mmfs/bin entry to PATH variable in /etc/environment file of each node so
that I don't have to type the full path to run GPFS commands every time I log in.
PATH looks like this.

Now I will run mmcrcluster command. It can be done directly from command line as I did or it
can be done creating a Node file. This can be ran from either of these nodes.

Now I will run few commands to check the configuration and status using mmlscluster
and mmlsconfig, mmgetstate see output below.

Now I will add the NSD disks to the Cluster. First I will add the tie breaker disk.
I will create a file called nsd_tb1 in /root directory and it will look like below

Now I will create the 1st NSD disk

Now I will check it with mmlsnsd

Let's see how that file nsd_tb1 looks now. mmcrnsd changed this file as below.

Next I will define this disk as tie breaker disk. I will try to run mmchconfig
and add the disk and see what happens.

It seems like that I forgot to shutdown the cluster before adding tiebreaker disk. 
I will issue mmshutdown command now.

Now I will re run the previous command

I will check the mmlsconfig and see if anything changed.

Looks like I can see the tie-breaker disk now.  Now I will bring up the cluster back

Now I will run mmlsnsd –X command to verify the NSD disks from both servers and I will check
whether they are showing correctly or not. From p550b_lp01 server. Seems configured correctly

From P550B_lp02 server. Exactly same output as above. So far good.

Now I will create a NSD and GPFS file system and mount it. I will create another file
nsd_db in /root directory and it will look like below

Now I will run mmcrnsd command as below

Now mmlsnsd and mmlsnsd –X from both nodes to see the device files are showing correctly or not.

From the above output it seems that dbdisk is correctly configured on the 200GB shared SAN
disk as I noted earlier hdisk5 and hdisk7 in p550b_lp01 & p550B_lp02 server.

Next step I will create the filesystem and them mount it. From the same node I created
the NSD disk I will create the filesystem so that I can use the same file.

From another node I will run mmlsfs command to check the filesystem status before I mount it.

Also mmlsconfig showing a filesystem below.

Now I will mount the filesystem. From any node I will issue mmmount all -a

Now I will check from p550b_lp02 first.

Now from p550b_lp01

So I could successfully build my first GPFS cluster. But it was very simple setup as
the SAN disks were shared device and both p550b_lp01 and p550b_lp02 could see the disks.

Now I will add another node and this node has no san disk and will use network to use GPFS.

I will install GPFS software and latest fix pack in the 3rd node. It is called p550a_lp01
(previous nodes were p550b_lp01 & p550b_lp02). After installing GPFS software I will
enable password less root ssh entry to and from all the existing nodes.

Once the above tasks are done I will add this node to the cluster. I will use all the
default values for this node. From any of the existing node I will issue the following
command.

Now I will verify whether this node has been added to the cluster or not. From the
new node I will issue mmlscluster and see the result.

From the above output I can see the 3rd node added and I ran this command from that node so
looks like I successfully added the 3rd node. Before I can mount the /oradb filesystem I
have to check the device files.

Now I will run mmstartup –a from this server and then check df output

Looks like the 3rd node is also working. Now I will run a small test. I will create one
file from each of the nodes and will try to read them from another node or will list them.

Now from any of the nodes I will run ls command ( ll is alias for ls –l)

And the contents of the files.

Next I will create a GPFS filesystem using 3 internal disks of all three nodes.
I will create the description file as below

Now I will create the NSD device

Results below

And the device files

Now creating filesystem on these three NSD Disks

Mounting  it now

Now df –g output of all nodes.  First p550b_lp01

Now on p550b_lp02

And now on p550a_lp01

Next step I will add another SAN disk to p550b_lp01 & p550B_lp02 and add it to
existing /oradb filesystem. See below.

After adding the new 200GB Lun on p550b_lp01

And on p550b_lp02

Now I will add the above disk as NSD device. But before doing that I need to collect
some information from existing disk of the filesystem.

Now using the description file as below I will create a new NSD device

Creating the NSD device

Verifying the NSD device from a different node

Size of oradb filesystem now

Now adding the new NSD device to oradb filesystem

Now the df –g output of /oradb filesystem

Now mmlsdisk output

Finally verifying

Now I want to add one more node to my cluster. After installing GPFS software & latest
fix packs, enabled password less root ssh and edited /etc/environment file to add
/usr/lpp/mmfs/bin in PATH statement.

From any of the existing node I will run

From newly added node I will run

Above output is looking good now.  One more command from another node

Now I need to enable sharing existing GPFS filesystems.

First time I ran to start GPFS in mmaa_lp01 2nd time to verify that it really started.
Now after few minutes I will run df –g to check whether GPFS filesystems are mounted or not.

Looks ok. Now I will add one internal disk of mmaa_lp01 to siddata filesystem. I will add hdisk21 which is
146GB SCSI disk. So I will edit the description file as below and run the mmcrnsd command.

And

Now I will verify the new NSD from another node.

df –g output of /siddata

Now I will add the new NSD device to siddata (few lines deleted from screenshot)

Now df –g output is showing extra space I added

mmlsdisk showing the new disk siddata4

mmdf output here

Everything looks as expected.  Here concludes my test.