Tight-LAM-Integration-Notes

From GridWiki
Revision as of 19:25, 9 August 2012 by LindaLopez (talk | contribs) (minor updates)
Jump to: navigation, search

Background

Author --Chris Dagdigian 17:34, 7 June 2006 (EDT)

This page documents the experience of following the process outlined here:

"Loose and tight integration of the LAM/MPI library into SGE" ... with the goal of trying to achieve tight LAM-MPI integration on a 12-node Linux cluster running SGE-6.0u8

People completly new to "Loose" vs "Tight" Parallel Environment integration may want to read this article: "Parallel Environments (PEs) - Loose vs. Tight Integration"


Get, patch, build and install the LAM-MPI software

The wget utility is used to download the LAM-MPI soure code:

# wget http://www.lam-mpi.org/download/files/lam-7.1.2.tar.gz

Uncompress and open the tar archive:

# zcat lam-7.1.2.tar.gz | tar xvf -

Discover that the LAM-MPI 7.1.2 source code has been updated so that it does not need the patch that Reuti recommends for avoiding a race condition within hboot. The existing HOWTO recommends manually patching the hboot.c code to add the "setsid()" line but this has already been done within the lam-7.1.2 distribution.

The code in lam-7.1.2/tools/hboot/hboot.c now looks like this with no changes necessary:

                else if (pid == 0) {            /* child */
                        /* Put this setsid() here mainly for SGE --
                           their tight integration with LAM/MPI does
                           something like this:

                           lamboot -> qrsh (to a remote node) -> hboot
                           -> qrsh -> lamd

                           Without having a setsid() here, there is a
                           race condition between when hboot quits and
                           SGE thinks the job is over (and therefore
                           starts killing things) and when the second
                           qrsh is able to establish itself and/or the
                           lamd and tell SGE that the job is, in fact,
                           *not* over.  So putting a setsid() here in
                           the child, then the hboot child (and
                           therefore the vulnerable period of the 2nd
                           qrsh) escape being killed by SGE while
                           still making progress on the overall
                           lamboot.
                        */
                        setsid();

This is excellent news and means that we can built LAM-MPI from source with no patching or file editing required for tight Grid Engine integration.

Decide how you want to build LAM-MPI. In my case there are a few things about my environment and preferred cluster setup that will affect how I'm going to build the LAM-MPI binaries for tight integration:

  1. My network interconnect is nothing fancy, just plain Gigabit Ethernet. No exotic communication required.
  2. I don't have a Fortran compiler installed and don't care about Fortran support in my LAM-MPI environment
  3. I want to build and install these files into a dedicated location so that it does not interfere with existing installations of MPICH2 and a loosely integrated LAM-MPI installation. The path I've chosen to install into will be called "/opt/class/tight-lammpi" - this path is shared on all cluster nodes.

Time to build lam-7.1.2 from source and install it into its dedicated home:

cd ./lam-7.1.2
./configure --prefix=/opt/class/tight-lammpi --without-fc
make
make install

Note: Make sure you have not overridden the RSH binary command (substituting passwordless SSH for instance ...) -- leave the RSH method alone both when compiling and setting LAM related environment variables. This will allow us to invoke remote commands via SGE's 'qrsh' program instead. This is one reason why I have multiple LAM-MPI installations on this cluster -- a different installation is for non-SGE use and is hard-coded to use "ssh -q" as the underlying transport mechanism. I made this mistake while writing this Wiki entry, the first time I compiled LAM I added the "--with-rsh=ssh" line to the configure command. The end result was a LAM installation that used direct SSH calls instead of the SGE 'qrsh' based methods required for tight integraton with Grid Engine.

After building and installing, we now have a newly populated LAM-MPI folder:

[root@galaxy-demo lam-7.1.2]# ls -l /opt/class/tight-lammpi/
total 24
drwxr-xr-x  2 root root 4096 Jun  7 14:42 bin
drwxr-xr-x  2 root root 4096 Jun  7 14:42 etc
drwxr-xr-x  3 root root 4096 Jun  7 14:42 include
drwxr-xr-x  3 root root 4096 Jun  7 14:42 lib
drwxr-xr-x  8 root root 4096 Jun  7 14:42 man
drwxr-xr-x  3 root root 4096 Jun  7 14:42 share

Get and install the helper scripts mentioned in the HOWTO

The online HOWTO linked at the top of this page includes a download link for all of the scripts mentioned. We want to download those scripts and make them available. A sensible location would be somewhere inside the custom location where the newly built LAM-MPI binaries and files were installed. In this example, that directory was "/opt/class/tight-lammpi/":

# cd /opt/class/tight-lammpi
# mkdir helper-scripts
# cd ./helper-scripts
# wget http://gridengine.sunsource.net/howto/lam-integration/sge-lam-integration-scripts.tgz
# zcat sge-lam-integration-scripts.tgz | tar xvf -
# ls -l
total 24
-rwxr-xr-x  1  502 users  577 Feb 22  2005 lamd_wrapper
drwxr-xr-x  2  502 users 4096 Feb 24  2005 lam_loose_qrsh
drwxr-xr-x  2  502 users 4096 Feb 20  2005 lam_loose_rsh
drwxr-xr-x  2  502 users 4096 Feb 24  2005 lam_tight_qrsh
-rw-r--r--  1  502 users  358 Feb 22  2005 mpihello.c
-rw-r--r--  1 root root  3059 Mar 26  2005 sge-lam-integration-scripts.tgz

Following the HOWTO's suggestions for Additional changes to LAM/MPI we install the wrapper script in place of our newly built binary:

# cd /opt/class/tight-lammpi/helper-scripts
# mv /opt/class/tight-lammpi/bin/lamd /opt/class/tight-lammpi/bin/lamd_binary
# cp ./lamd_wrapper /opt/class/tight-lammpi/bin/
# chmod +x /opt/class/tight-lammpi/bin/lamd_wrapper
# cd /opt/class/tight-lammpi/bin
# ln -s lamd_wrapper lamd

Customize the startlam.sh and stoplam.sh scripts

The startlam.sh helper script that was downloaded needs to be adjusted to match where the helper scripts reside on the system. For startlam.sh look at the lines of code around #111:

   rsh_wrapper=$SGE_ROOT/lam_tight_qrsh/rsh
   rsh_wrapper=/home/reuti/lam_tight_qrsh/rsh
   if [ ! -x $rsh_wrapper ]; then
      echo "$me: can't execute $rsh_wrapper" >&2
      echo "     maybe it resides at a file system not available at this machine" >&2
      exit 1
   fi

Obviously you want to set "rsh_wrapper=" to point to the correct location on your system. If yo ... \n

Student Goes From Homeless to Harvard

Despite being abandoned to homelessness by her parents, Dawn Loggins worked as a high school custodian by day and studied hard by night to become the first person from her school to ever be admitted to Harvard.

[Student Goes From Homeless to Harvard]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

5 Principles for Inner Transformation at Work

People go to work to sustain themselves and produce value in the world. Yet work environments can also be stressful, filled with challenging responsibilities and personalities, and feel misaligned with our most deeply cherished values. Instead of sustaining us, the workplace can sometimes feel simply draining, and at worst, unwholesome for both ourselves and the world.

[5 Principles for Inner Transformation at Work]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

Your Life Is What You Make It

If you limit your choices only to what seems possible or reasonable, you disconnect yourself from what you truly want, and all that is left is compromise. Robert Fritz

[Your Life Is What You Make It]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

An Incredible Graduation Gift from Dad

Brenna Martins dad evidently doesnt like last-minute shopping. Bryan Martin a purchased a gift for his daughters high school graduation -- which happened earlier this month -- thirteen years ago. He managed to keep it hidden this whole time, and his "moving, touching, nostalgic, and thoughtful" present (her words) brought Brenna to tears when dad finally gave it to her last week.

[An Incredible Graduation Gift from Dad]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

Life Through A Camera Lens

"I couldnt believe how beautiful the ocean was, he commented. Ive never seen an ocean before, and then to get to see palm trees in person, and to even touch them. It was just amazing. He began thumbing through a series of photographs on his phone, each displaying an image of a palm tree." 
[Life Through A Camera Lens]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]