Difference between revisions of "Tight-HP-MPI-Integration-Notes"

From GridWiki
Jump to: navigation, search
m
(minor updates)
Line 207: Line 207:
  
 
The number of threads expanded by each qrsh depend of the shell from which it is called:
 
The number of threads expanded by each qrsh depend of the shell from which it is called:
* If the shell is recognised as “sh” then it expand 3 threads.
+
* If the shell is recognised as “sh” then it expa ... \n
* In other case, if the shell is NOT recognised as “sh”(for example, “bash”) then it expand 4 threads. In this case one of them is in charge of the shell and consume 30MB of memory extra. This problem could be avoided using the option ''–noshell''.
 
  
This way, if the stack is unlimited it consumes 73MB of virtual memory and if we have a stack limit of 256MB then each qrsh consumes more than 500MB of virtual memory.
+
== The Importance of Imagination ==
  
This could lead to problems when running jobs because GE kills them due to the fact that they surpass the memory limit of s_vmem in the master node just because the qrsh processes are consuming too much memory (one qrsh process is launched per slave node).
+
While growing up, Id never really considered how important it is to be imaginative. Its a childhood profession, you could say. It comes naturally. Then we hit an age when were presented with a scantron of bubble-in options, a template for a CV that we need to create, and Excel. At that point, our learning has to fit into certain parameters: within that little bubble, within the one page limit, and within a tiny digital graph. So, what happens to our imagination?It seems to fade.
  
For an optimal memory consumption in Itanium a stack value of 1MB can be used. In this case each qrsh consumes 10MB of virtual memory (4MB of them physical memory).  For that we set this limit inside the rsh script before qrsh is launched:
+
[[http://goodvillenews.com/The-Importance-of-Imagination-BvyL3n.html The Importance of Imagination]]
  
<pre>
+
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
...
 
ulimit -s 1024
 
...
 
  if [ $minus_n = 1 ]; then
 
      exec $me -n $rhost $cmd
 
  else
 
      exec $me $rhost $cmd
 
  fi
 
...
 
</pre>
 
  
= Additional Scripts Needed =
+
== Refugee Turned Entrepreneur Uplifts Women Abroad ==
  
* <YOUR_SGE_ROOT>/mpi/rsh
+
While some retailers have struggled in the recession, Amber Chand, an online retailer of items made by women living in war-torn countries, is experiencing success."As the economy was going into a downturn during the holiday season, which is my primary season, I noticed actually that my company was increasing in terms of sales and revenues, and we grew by 22 percent," says Chand.
  
<pre>
+
  [[http://goodvillenews.com/Refugee-Turned-Entrepreneur-Uplifts-Women-Abroad-w0ETyF.html Refugee Turned Entrepreneur Uplifts Women Abroad]]
#!/bin/sh
 
#
 
# Author: Sun, modifications by JLC
 
# Purpose:
 
#  Wrapper to run rsh commands through qrsh
 
#
 
#
 
#___INFO__MARK_BEGIN__
 
##########################################################################
 
#
 
# The Contents of this file are made available subject to the terms of
 
#  the Sun Industry Standards Source License Version 1.2
 
#
 
#  Sun Microsystems Inc., March, 2001
 
#
 
#
 
#  Sun Industry Standards Source License Version 1.2
 
#  =================================================
 
#  The contents of this file are subject to the Sun Industry Standards
 
#  Source License Version 1.2 (the "License"); You may not use this file
 
#  except in compliance with the License. You may obtain a copy of the
 
#  License at http://gridengine.sunsource.net/Gridengine_SISSL_license.html
 
#
 
#  Software provided under this License is provided on an "AS IS" basis,
 
#  WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING,
 
#  WITHOUT LIMITATION, WARRANTIES THAT THE SOFTWARE IS FREE OF DEFECTS,
 
#  MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE, OR NON-INFRINGING.
 
#  See the License for the specific provisions governing your rights and
 
#  obligations concerning the Software.
 
#
 
#  The Initial Developer of the Original Code is: Sun Microsystems, Inc.
 
#
 
#  Copyright: 2001 by Sun Microsystems, Inc.
 
#
 
#  All Rights Reserved.
 
#
 
##########################################################################
 
#___INFO__MARK_END__
 
  
# could be rsh or remsh
+
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
me=`basename $0`
 
# just_wrap=1
 
  
# remove path to wrapping rsh from path list
+
== 7 Ways to Deal With the Death of a Loved One ==
if [ "x$TMPDIR" != "x" ]; then
 
  PATH=`echo $PATH|tr : "\012"|grep -v $TMPDIR| tr "\012" :`
 
  export PATH
 
fi
 
  
# rehash
+
How do you deal with death, the loss of a loved one when the pain is so strong; how can you let go of the people you once loved and still love so much; how can you accept the fact that you will never see those people ever again? How can you accept the idea of loss, of death?
hash -r
 
  
if [ "x$JOB_ID" = "x" ]; then
+
[[http://goodvillenews.com/7-Ways-to-Deal-With-the-Death-of-a-Loved-One-K7YCOr.html 7 Ways to Deal With the Death of a Loved One]]
  exec $me $*
 
  echo command $me not found in PATH=$PATH
 
fi
 
  
# extract target hostname
+
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
if [ $# -lt 1 ]; then
 
  echo $me: missing hostname
 
  exit 1
 
fi 
 
  
# Handle hostname before options
+
== Worth Our Weight: The Taste of Compassion ==
rhost=
 
expr "$1" : "-*" >/dev/null 2>&1
 
  
if [ $? -ne 0 ]; then
+
Its a sunny Saturday morning, and my wife and I turn into a residential neighborhood of Santa Rosa, California the only sign of commerce is a beleaguered shopping mall a few hundred feet away. I glance down at my GPS to make sure this is where we want to be and in doing so I almost pass my destination. There it is tucked away to the left, sandwiched between two homes: Worth our Weight (W.O.W)
  rhost=$1
 
  shift
 
fi
 
  
ruser=
+
[[http://goodvillenews.com/Worth-Our-Weight-The-Taste-of-Compassion-qYcZwK.html Worth Our Weight: The Taste of Compassion]]
minus_n=0
 
  
# parse other rsh options
+
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
while [ "$1" != "" ]; do
 
  case "$1" in
 
      -l)
 
        shift
 
        if [ $# -lt 1 ]; then
 
            echo $me: option -l needs user name as argument
 
            exit 1
 
        fi 
 
        ruser=$1
 
        ;;
 
      -n)
 
        minus_n=1
 
        ;;
 
      -*)
 
        echo $me: Unsupported option - $1
 
        exit 1
 
        ;;
 
      *)
 
        break;
 
        ;;
 
  esac
 
  shift
 
done
 
  
# Handle hostname after options
+
== A Sisters Deathless Legacy of Love ==
if [ "x$rhost" = x ]; then
 
  if [ $# -lt 1 ]; then
 
      echo $me: missing hostname
 
      exit 1
 
  fi 
 
  rhost=$1
 
  shift
 
fi
 
  
# should the command to be started preceded with any starter command
+
After living through an experimental cancer treatment my sister Barb was left unable to work. When she was offered the opportunity to do a mission trip in India if she could come up with $3,000 - she was left thinking there was no way she could go. No way to raise the funds. She asked me to brainstorm with her as to ways she could raise money. "The only thing I can do is hug," she told me - and thus her adventure began.
if [ "x$RCMD_PREFIX" = x ]; then
 
  cmd="$*"
 
else
 
  cmd="$RCMD_PREFIX $*"
 
fi
 
  
# unset TASK_ID - it might be set if a task starts another tasks
+
[[http://goodvillenews.com/A-Sisters-Deathless-Legacy-of-Love-UvTY7u.html A Sisters Deathless Legacy of Love]]
#                and may not be exported in this case
 
if [ "x$TASK_ID" = x ]; then
 
  unset TASK_ID
 
fi
 
  
# CESGA's local variables
+
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
#
 
CESGA_VARS=""
 
if [ x$LOADEDMODULES != x ]; then
 
  CESGA_VARS="$CESGA_VARS -v LOADEDMODULES=$LOADEDMODULES"
 
fi
 
if [ x$OMP_NUM_THREADS != x ]; then
 
  CESGA_VARS="$CESGA_VARS -v OMP_NUM_THREADS=$OMP_NUM_THREADS"
 
fi
 
 
 
# qrsh from GE6.2 consumes 512MB with the default value of stack of 256MB
 
# reducing the stack limit to 1024 the memory usage is reduced to 43MB
 
ulimit -s 1024
 
 
 
if [ x$just_wrap = x ]; then
 
  if [ $minus_n -eq 1 ]; then
 
      echo $SGE_ROOT/bin/$ARC/qrsh $CESGA_VARS -inherit -nostdin $rhost $cmd 
 
      exec $SGE_ROOT/bin/$ARC/qrsh $CESGA_VARS -inherit -nostdin $rhost $cmd
 
  else
 
      echo $SGE_ROOT/bin/$ARC/qrsh $CESGA_VARS -inherit $rhost $cmd
 
      exec $SGE_ROOT/bin/$ARC/qrsh $CESGA_VARS -inherit $rhost $cmd 
 
  fi
 
else
 
  echo $me $rhost $*
 
  if [ $minus_n = 1 ]; then
 
      exec $me -n $rhost $cmd
 
  else
 
      exec $me $rhost $cmd
 
  fi
 
  echo $me not found in PATH=$PATH
 
fi
 
</pre>
 
 
 
= Testing the installation =
 
 
 
To test the installation you can use a script similar to the following one. In our case to use HP-MPI it is needed to load the hp-mpi and intel [http://modules.sourceforge.net/ modules], this of course depends on how you choose to install HP-MPI:
 
<pre>
 
#!/bin/bash
 
 
 
## HP-MPI
 
echo "== HP-MPI test =="
 
module load hp-mpi intel
 
 
 
# In case it is not done in the global/user environment
 
#export MPI_REMSH="$TMPDIR/rsh"
 
 
 
# You could need also the option "-hostfile $TMPDIR/machines" if it
 
# is not include in the global/user environment (MPIRUN_OPTIONS)
 
mpirun -np $NSLOTS my-hp-mpi-program
 
</pre>
 

Revision as of 19:25, 9 August 2012

Introduction

The integration has been tested both with GE 6.1 and 6.2.

  • GE 6.1 uses rsh:
    • Limitations:
      • There is a theoretical limit of 256 connections: Our tests indicate that only one connection is generated against each slave node per job.
      • No prompt in the session: not aware of any problem that this could cause in batch jobs
  • GE 6.2 uses an internal communication protocol:
    • Limitations:
      • No prompt
      • No X11 forwarding

HP-MPI Installation

We install HP-MPI in a shared directory (/opt/cesga) so all nodes can access it:

 cn141 # rpm -ivh --test --prefix /opt/cesga/hp-mpi-2.3/ hpmpi-2.03.00.00-20081120r.ia64.rpm 
 Preparing...                ########################################### [100%]
 cn141 # rpm -ivh --prefix /opt/cesga/hp-mpi-2.3/ hpmpi-2.03.00.00-20081120r.ia64.rpm 
 Preparing...                ########################################### [100%]
    1:hpmpi                  ########################################### [100%]

Local installation in each node is also possible.

GE Configuration

Set up the Grid Engine Parallel Environment (PE) using the command: qconf -ap mpi

  • mpi: Basic PE using $fill_up allocation rule (all slots allocated in a node before moving to the next one)
pe_name           mpi
slots             9999
user_lists        NONE
xuser_lists       NONE
start_proc_args   <YOUR_SGE_ROOT>/mpi/startmpi.sh -catch_rsh \
                  $pe_hostfile
stop_proc_args    <YOUR_SGE_ROOT>/mpi/stopmpi.sh
allocation_rule   $fill_up
control_slaves    TRUE
job_is_first_task FALSE
urgency_slots     min
accounting_summary FALSE
  • mpi_rr: PE with $round_robin allocation (one slot per node if possible)
pe_name            mpi_rr
slots              9999
user_lists         NONE
xuser_lists        NONE
start_proc_args    <YOUR_SGE_ROOT>/mpi/startmpi.sh -catch_rsh \
                   $pe_hostfile
stop_proc_args     <YOUR_SGE_ROOT>/mpi/stopmpi.sh
allocation_rule    $round_robin
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE
  • mpi_1p: 1 slot per node (mandatory)
pe_name            mpi_1p
slots              9999
user_lists         NONE
xuser_lists        NONE
start_proc_args    <YOUR_SGE_ROOT>/mpi/startmpi.sh -catch_rsh \
                   $pe_hostfile
stop_proc_args     <YOUR_SGE_ROOT>/mpi/stopmpi.sh
allocation_rule    1
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE
  • mpi_2p, mpi_4p, mpi_8p, mpi_16p, etc.: 2, 4, 8, 16, etc. slots per node (depending on how many different allocation rules you want to allow and how many CPUs each node has)

If you are using Modules to set up the hp-mpi environment then in the module use:

setenv          MPI_REMSH	"$env(TMPDIR)/rsh"

In other case in the global/user environment export the MPI_REMSH variable:

export MPI_REMSH=$TMPDIR/rsh

Modify the starter_method and pe_list options of the queue you want to use: qconf -mq hp-mpi-queue

starter_method        /opt/cesga/sistemas/sge/tools/util/job_starter.sh
pe_list               mpi mpi_rr mpi_1p mpi_2p mpi_4p mpi_8p mpi_16p

This is the script used as the job starter:

#!/bin/bash
# Author: JLC
# Purpose:
#   Wrapper to start the job script and the scripts it runs in the slave nodes using qrsh
#
# Changelog
#   05-11-2008 JLC
#     First version
#   11-11-2008 JLC
#     Added reading of user's profile
#   22-01-2009 JLC
#     Added debug functions and adapted to work with GE6.2
#   26-01-2009 JLC
#     Bug fix: set CPU variable in case profile has not done it
#   02-02-2009 JLC
#     Bug fix: export CPU variable in other case the sge's module does not see it
#

# Our queues are configured using shell_start_mode=posix_compliant and shell=/bin/bash
# We add the -l option to force the shell to re-read the profile.d scripts and set the correct env
# For qrsh sessions we want to load the right environment

#
# DEBUG
#
DEBUG="false" # Set it to "true" to get debug information
DEBUG_USER="uscfajlc"
# Place it on an NFS directory so you can easily get all the outputs
DEBUG_LOG="/home/usc/fa/jlc/mpi/job_starter_${HOSTNAME}_${JOB_ID}_$$.out" 

# Log output of command
debug_cmd (){
        if [[ $LOGNAME = $DEBUG_USER && $DEBUG = "true" ]]; then
                $* >> $DEBUG_LOG 2>&1
        fi
}
# Log function
debug_log (){
        if [[ $LOGNAME = $DEBUG_USER  && $DEBUG = "true" ]]; then
                echo "$*" >> $DEBUG_LOG 2>&1
        fi
}

debug_cmd env
debug_cmd module list

# We want to know the CPU type even in case /etc/profile has not been loaded
test -z "$CPU"  &&  CPU=`/bin/uname -m 2> /dev/null`
export CPU

# Export SGE_BINARY_PATH: zz-cesga.sh uses it to detect the right SGE module
export SGE_BINARY_PATH

if [[ $JOB_SCRIPT = QRSH || $JOB_SCRIPT = INTERACTIVE ]];then
        debug_log "QRSH|INTERACTIVE: Reading profile"
        # In GE6.2 PROFILEREAD=true is passed to the SLAVE. This prevents the re-reading of the profile scripts.
        # To avoid it:
        unset PROFILEREAD
        #
        # Read profile and profile.d scripts
        #
        if test -z "$PROFILEREAD" ; then
            test -r /etc/profile.d/sh.ssh   && . /etc/profile.d/sh.ssh
            test -r /etc/SuSEconfig/profile && . /etc/SuSEconfig/profile
            if test -z "$SSH_SENDS_LOCALE" ; then
                if test -r /etc/sysconfig/language -a -r /etc/profile.d/sh.utf8 ; then
                    tmp="$(. /etc/sysconfig/language; echo $AUTO_DETECT_UTF8)"
                    test "$tmp" = "yes" && . /etc/profile.d/sh.utf8
                    unset tmp
                fi
            fi
        fi
        if test -d /etc/profile.d -a -z "$PROFILEREAD" ; then
            for s in /etc/profile.d/*.sh ; do
                test -r $s && . $s
            done
            unset s
        fi

        if test -f $HOME/.profile ; then
                test -r $HOME/.profile && . $HOME/.profile
        fi

        if test -f $HOME/.bash_profile ; then
                test -r $HOME/.bash_profile && . $HOME/.bash_profile
        fi

        if test -f $HOME/.bash_login ; then
                test -r $HOME/.bash_login && . $HOME/.bash_login
        fi

        debug_log "After loading profile scripts"
        debug_cmd env
        debug_cmd module list

        # Finally we run the command
        echo $*
        $*
else
        echo /bin/bash -l $*
        /bin/bash -l $*
fi

Additional considerations in Itanium

The qrsh version included in GE6.2 that adds support for builtin communication (IJS) uses threads. Memory consumption is about 2 times the stack limit (set for example with ulimit). In case there is no stack limit it uses 32MB as a reference and multiplies it by 2.

The number of threads expanded by each qrsh depend of the shell from which it is called:

  • If the shell is recognised as “sh” then it expa ... \n

The Importance of Imagination

While growing up, Id never really considered how important it is to be imaginative. Its a childhood profession, you could say. It comes naturally. Then we hit an age when were presented with a scantron of bubble-in options, a template for a CV that we need to create, and Excel. At that point, our learning has to fit into certain parameters: within that little bubble, within the one page limit, and within a tiny digital graph. So, what happens to our imagination?It seems to fade.

[The Importance of Imagination]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

Refugee Turned Entrepreneur Uplifts Women Abroad

While some retailers have struggled in the recession, Amber Chand, an online retailer of items made by women living in war-torn countries, is experiencing success."As the economy was going into a downturn during the holiday season, which is my primary season, I noticed actually that my company was increasing in terms of sales and revenues, and we grew by 22 percent," says Chand.

[Refugee Turned Entrepreneur Uplifts Women Abroad]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

7 Ways to Deal With the Death of a Loved One

How do you deal with death, the loss of a loved one when the pain is so strong; how can you let go of the people you once loved and still love so much; how can you accept the fact that you will never see those people ever again? How can you accept the idea of loss, of death?

[7 Ways to Deal With the Death of a Loved One]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

Worth Our Weight: The Taste of Compassion

Its a sunny Saturday morning, and my wife and I turn into a residential neighborhood of Santa Rosa, California the only sign of commerce is a beleaguered shopping mall a few hundred feet away. I glance down at my GPS to make sure this is where we want to be and in doing so I almost pass my destination. There it is tucked away to the left, sandwiched between two homes: Worth our Weight (W.O.W)

[Worth Our Weight: The Taste of Compassion]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]

A Sisters Deathless Legacy of Love

After living through an experimental cancer treatment my sister Barb was left unable to work. When she was offered the opportunity to do a mission trip in India if she could come up with $3,000 - she was left thinking there was no way she could go. No way to raise the funds. She asked me to brainstorm with her as to ways she could raise money. "The only thing I can do is hug," she told me - and thus her adventure began.

[A Sisters Deathless Legacy of Love]

[GoodvilleNews.com - good, positive news, inspirational stories, articles]