Disabling direct ssh connection to the nodes

From GridWiki

Jump to: navigation, search

Contents

Introduction

To have a tight control of who is using the nodes and avoiding users connecting to the nodes and interfering with running jobs it is desirable to disable direct ssh connection to the nodes.

To accomplish this the following steps are required:

  • MPI Tight Integration
  • Qlogin: configure it to run over qrsh following also the tight integration way
  • Re-configuration of ssh to allow only certain administrative users to connect

In this way you also get proper accounting of all the processes run by the users through GE.

MPI Tight Integration

Check the corresponding pages for each MPI distribution:

Qlogin

Qlogin should be configured using the new IJS functionality available in GE6.2. That is (qconf -mconf command): 

qlogin_command               builtin
qlogin_daemon                builtin
rlogin_daemon                builtin
rsh_daemon                   builtin
rsh_command                  builtin
rlogin_command               builtin

To allow easier interactive use of the nodes (e.g. to compile programs and run interactive programs) we have created a wrapper over qlogin where the only required option is the memory needed.

Forwarding X11 in interactive sessions

Changes required in sshd_config in the login node: 

fs001:~ # diff /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
94,95c94
< X11UseLocalhost no
< AddressFamily inet
---
> #X11UseLocalhost yes

This allows to use directly a command like: 

qsh -l num_proc=1,s_rt=10:00:00,s_vmem=128M,h_fsize=20G,h_stack=256M

and get an xterm in your local display.

Other option is to run a gnome-terminal: 

qrsh -v DISPLAY=$DISPLAY -l num_proc=1,s_rt=10:00:00,s_vmem=256M,h_fsize=20G,h_stack=8M /opt/gnome/bin/gnome-terminal

NOTE.- In this case it is important to set the h_stack to a low value in order to reduce the memory consumption

A last option is to run a normal qlogin: 

qlogin -l num_proc=1,s_rt=10:00:00,s_vmem=512M,h_fsize=20G,h_stack=8M

an export the DISPLAY variable from the login node (fs001 in our case) in the session: 

export DISPLAY=fs001:29.0

If you need X11 functionality there are two options with the new wrapper:

  • Specifying the -X option a xterm is opened
  • Inside the interactive session you run source enable_X 
#!/bin/bash
#
# Authors: Javier Lopez Cacheiro (jlopez@cesga.es)
# Purpose: To provide users an interactive work environment limited by memory
# Usage:   compile --memory <MEM>
#	where <MEM> is the maximum memory needed in GB
# Return: 0 if success, 1 if error
#

CMDNAME=`basename $0`
USAGE="Usage: $CMDNAME --memory <MEM> [-X]
	where <MEM> is the maximum memory needed in GB
	-X: open an interactive session in a xterm with X11 environment
        Ex. Open an interactive session with a memory limit of 2GB
            $CMDNAME --memory 2
        The session is limit to use until 1 hour of CPU time
        (Don't confuse this time with the connection time)
	"
WARNING="
"

# Default memory limit (1 GB)
MEM=1

while :
do
	case $1 in
		--memory) shift
				MEM=$1
				shift
				;;
		-X) FORWARD_X11="yes"
				shift
				;;
		-*) echo "$USAGE" 1>&2
				exit 1
				;;
		*)   break
				;;
	esac
done

# Variables to pass through ssh
export INTERACTIVE="yes"
export INTERACTIVE_MEMORY="$MEM"

echo "$WARNING" 

# Alert about using mpi
echo "******************************************************************************************"
echo "***                                                                                    ***"
echo "***  Remember that it is not allow to run parallel jobs during an interactive session  ***"
echo "***                                                                                    ***"
echo "******************************************************************************************"
echo "***  If you want to use programs requiring an X11 graphics environment execute the     ***"
echo "***  following command:                                                                ***"  
echo "***   source enable_X                                                                  ***"
echo "******************************************************************************************"

# Option A: Create the interactive session using ssh
#NODE=cn001
#ssh $NODE

# Option B: Create the interactive session using qlogin
if [[ -n $DISPLAY ]]; then
	echo "export DISPLAY=$DISPLAY" > ~/.qlogin.display 
fi

if [[ $FORWARD_X11 = "yes" ]]; then
	qsh -l num_proc=1,s_rt=10:00:00,s_vmem=${MEM}G,h_fsize=20G,h_stack=256M
else
	qlogin -l num_proc=1,s_rt=10:00:00,s_vmem=${MEM}G,h_fsize=20G,h_stack=256M
fi

if [[ -n $DISPLAY ]]; then
	rm ~/.qlogin.display &> /dev/null
fi

/usr/bin/enable_X 

source ~/.qlogin.display

SSH Configuration

Modify /etc/ssh/sshd_config in the nodes to restrict direct access to only certain administrative users and restart sshd: 

UsePAM yes
X11Forwarding yes 
Subsystem	sftp	/usr/lib/ssh/sftp-server
AcceptEnv INTERACTIVE INTERACTIVE_MEMORY
AcceptEnv OMP_NUM_THREADS LOADEDMODULES
AcceptEnv TMPDIR
HostbasedAuthentication yes
IgnoreUserKnownHosts yes
AllowUsers root admin operator

Tracking/accounting of launched processes

In an normal job the additional GID is set up automatically, and in a parallel job that runs using a MPI PE with tight integration it is also set. To check it I simply run a job with an "id -a" command: 

uid=10946(jlopez) gid=113(cesga) groups=113(cesga),1046(gaussian),2516(vtune),20047

The additional GID is taken from the range gid_range=20000-21000 (it can be configured with qconf)

In this way all processes, including interactive jobs, are correctly accounted in the GE accounting file.

Retrieved from "http://wiki.gridengine.info/wiki/index.php/Disabling_direct_ssh_connection_to_the_nodes"
Personal tools