Olesen-FLEXlm-SGE5

From GridWiki
Jump to: navigation, search

Using the "Olesen" FLEXlm integration method on SGE 5.x

  • This document was last updated: -- --Dag 12:18, 26 February 2006 (EST)

Background

This is a companion document to the real Grid Engine 6 FlexLM License Integration (Olesen Method) document. You should be extremely familiar with that document (perhaps keeping it open in a different browser window) before starting to deal with the information on this page.

The purpose of this particular wiki entry is to document the steps involved in backporting Mark Olesen's qlicserver license monitoring tool so that it will support any version of Grid Engine 5.x (including SGE 5.3.x, SGE Enterprise Edition 5.x and Sun Grid Engine 5.x).

Supporting SGE 5.x is relatively easy but comes with some non-trivial limitations users should be familiar with.

Acknowlegements

  • Mark Olesen for providing code and hints
  • Computational chemistry software from Schrödinger was extensively used during the creation and testing of this document and the methods it describes. It is difficult to write about FLEXlm license integration without having licensed applications onhand to test with. The authors of this document would like to acknowledge and thank Schrödinger for their generous donation of software and licenses.

Current Status

  • A patch is available that can be applied to the official qlicserver distribution to enable support of Grid Engine 5.x.
  • The patch has currently only been tested on one cluster, using FLEXlm licensed applications that are not parallel so testing and feedback is needed.

Current Weaknesses

Summary of changes required

Altering the qlicserver code to support SGE 5.x systems is straightforward. There are only three changes that need to be made:

  1. In SGE 5.x one must explicitly work upon the global complex since unlike SGE 6 there are multiple complex object lists
  2. The formatting of fields in the complex is slightly different between SGE 5.x and 6.x requiring a slight change in how complex entries are created
  3. SGE 5.x does not support XML output options when using qstat. The qlicserver code written for SGE 6 based systems parses XML qstat output. To work properly on SGE 5.x based systems, a new qstat parser must be constructed that can handle the plaintext, non-XML output from the command "qstat -s prs -r"

Obtaining and applying the patch

The patch can be obtained from http://gridengine.info/files/SGE53-qlicserver-patch.txt


Download patch

[dag@bertha site]$ wget http://gridengine.info/files/SGE53-qlicserver-patch.txt
--09:34:34--  http://gridengine.info/files/SGE53-qlicserver-patch.txt
           => `SGE53-qlicserver-patch.txt'
Resolving gridengine.info... 66.92.70.114
Connecting to gridengine.info|66.92.70.114|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,266 (6.1K) [text/plain]

100%[======================================================>] 6,266         --.--K/s             

09:34:34 (1.30 MB/s) - `SGE53-qlicserver-patch.txt' saved [6266/6266]

[dag@bertha site]$

Apply the patch

[dag@bertha site]$ patch qlicserver < SGE53-qlicserver-patch.txt
patching file qlicserver

Once qlicserver has been patched, follow the general instructions for customization and testing described in Grid Engine 6 FlexLM License Integration (Olesen Method) document. Extra care will be needed in testing and troubleshooting, pay careful attention to the output of "./qlicserver -n" and "./qlicserver -n -D" when testing to make sure license accounting figures are accurate.

In addition, custom changes need to be made to the Qstat5 package - currently it is required that the user alter the regular expression used to capture the license-specific hard resource requests. This is the code that needs to be altered:

     my %regexp = (
         entries   => qr{Master queue:},
         states    => qr{^(r|s|qw)$},
         h_lic_req => qr{(i_glide=\d+|i\_main=\d+)},
     );

Adjust the value of h_lic_req to match the hard resource license request(s) seen via the "qstat -s prs -r" command. For this document, the hard resource requests will always be for "i_glide=N" or "i_main=N"

Technical: SGE Complex

For more information on the SGE complex, see the 'complex' man (5) manpage.

In all versions of SGE 5.x there are three distinct complex objects: global, queue and host. This is different in SGE 6.x where there is now a single unified complex object accessed via the standard sorts of "qconf -sc" and "qconf -mc" commands.

This means that step #1 in making the qlicserver code work on SGE 5.x based systems it is important to make sure that every command used to query or alter a complex resource entry is explicitly working on the global complex. This is very simple as it just means changing the "qconf -sc" command to "qconf -sc global" etc. etc.

The second change only slightly more complicated, in SGE 5 the actual format of resources listed within the complex object is slightly different than the format used in SGE 6.

Assuming one is dealing with the same licenses used in the Grid Engine 6 FlexLM License Integration (Olesen Method) document, this is what the complex would look like in SGE 6:

SGE 6.x complex

# name shortcut type    relop   requestable consumable default urgency
# --------------------------------------------------------------------------
i_glide i_glide INT     <=      YES         YES        0       0
i_main  i_main  INT     <=      YES         YES        0       0

And this is what the same user-requestable, consumable resources would look like in SGE 5.x global complex:

SGE 5.x complex

#name            shortcut   type   value      relop requestable consumable default
#---------------------------------------------------------------------------------
i_glide          i_glide    INT    0          <=    YES         YES        0    
i_main           i_main     INT    0          <=    YES         YES        0  

Note the minor changes in formatting of complex entries between SGE 6.x and 5.x based systems.

Modifying the original Olesen qlicserver code to support these changes is trivial. The specific changes can be examined within the patch.

Technical: qstat parsing

Much harder is the process of dealing with the output of "qstat -s prs -r" which is the command used to list running and pending jobs along with their particular resource requests. It is from this output that license monitoring code can figure out how many license dependent jobs are either active or pending.

This process is much easier in SGE 6.x because the "-xml" output option can be used. This leaves the original qlicserver Qstat package free to parse very structured data that looks something like this:

 <?xml version='1.0'?>
 <job_info  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <queue_info>
     <job_list state="running">
       <JB_job_number><b>934</b></JB_job_number>
       <JAT_prio>0.56000</JAT_prio>
       <JB_name>k-v20c_z04</JB_name>
       <JB_owner><b>stadler</b></JB_owner>
       <state>r</state>
       <JAT_start_time>11/30/2004 10:38:23</JAT_start_time>
       <queue_name><b>cfd@host.domain</b></queue_name>
       <slots>1</slots>
       <b><hard_request name="fire7">1</hard_request></b>
       <hard_req_queue>cfd</hard_req_queue>
     </job_list>
   </queue_info>
   <job_info>
   </job_info>
 </job_info>

With structured data, it is straighforward to pull out the following important bits of information:

  • JobID
  • User
  • Job state
  • All hard resource requests (where license tokens are requested)
  • Cluster queue name
  • Execution host name
  • Slot count (in PE environments)

Without XML output options, the plaintext output from SGE 5.x must be parsed. It looks like this:

[dag@bertha site]$ qstat -s prs -r
job-ID  prior name       user         state submit/start at     queue      master  ja-task-ID 
---------------------------------------------------------------------------------------------
      6     0 sge-1ett_d dag          r     02/24/2006 15:35:10 alpha.q    MASTER         
       Full jobname:     sge-1ett_dock
       Master queue:     alpha.q
       Hard Resources:   i_glide=4
                         i_main=1
      7     0 sge-1ett_d dag          qw    02/21/2006 14:20:29                           
       Full jobname:     sge-1ett_dock
       Hard Resources:   i_glide=4
                         i_main=1
      8     0 sge-1ett_d dag          qw    02/21/2006 14:20:32                           
       Full jobname:     sge-1ett_dock
       Hard Resources:   i_glide=4
                         i_main=1
      9     0 sge-1ett_d dag          qw    02/21/2006 14:23:59                           
       Full jobname:     sge-1ett_dock
       Hard Resources:   i_glide=4
                         i_main=1

Most of the information required is present in this output format, it just needs to be pulled out and collected via a new parsing mechanism.

The parsing is different enough that it makes more sense to just create a new Qstat5 perl package within the qlicserver program. This new package will parse the plaintext output and return information in the same way that the original xml-using Qstat package does.

The new Qstat5 package represents a majority of the patch that this document is discussing.

One potential problem

There is one major weakness that this patch creates for users of SGE 5.x based systems:

queue names are assumed to match execution host names!

This means the code will implicitly assume that a job running within the "alpha.q" queue instance is also executing on a machine named "alpha". This information is critically important and is used by the license monitoring code to discern between internal consumers of licenses (jobs running under the control of Grid Engine) and external license consumers (jobs running outside of Grid Engine, perhaps on a user workstation etc.). In order for the license monitoring code to correctly manage license consumables within Grid Engine, it needs account cleanly for both external and external users of floating license tokens.

How does qlicserver figure this out?

Well, the first info comes directly from the FLEXlm license server, when jobs are running and licenses are being consumed the server output will contain information lines that look like this:

Users of IMPACT_MAIN:  (Total of 20 licenses issued;  Total of 3 licenses in use)

  "IMPACT_MAIN" v44, vendor: SCHROD
  nodelocked license locked to NOTHING (hostid=ANY)

    dag bertha.bioteam.net /dev/tty (v35) (gw/27000 301), start Sat 2/25 10:56
    dag bertha.bioteam.net /dev/tty (v35) (gw/27000 109), start Sat 2/25 10:58
    dag bertha.bioteam.net /dev/tty (v35) (gw/27000 602), start Sat 2/25 10:59

Note that the license server output exactly specifies where license tokens are in use and who is using them.

This same information is easily available in the XML output from SGE 6.x:

  ...
<JB_owner>dag</JB_owner>
<queue_name>all.q@bertha.bioteam.net</queue_name>
...

By comparing the who and where data from both Grid Engine and the FLEXlm license server, the license management code can easily discern between internal and external consumers of license entitlement resources.

But ...

This is not so easy in SGE 5.x, the output of qstat does not contain any sort of explicit mention of the execution host name that can be compared back to the host names mentioned by the FLEXlm server.

The best info we have available is this line of "qstat -s rsp -r" output:

  Master queue:     alpha.q


This is where the weakness comes from. There is no conclusive way via qstat in SGE 5.x to learn the hostname of the execution host. This means we must rely instead on the common practice of naming SGE 5.x queue instances in accordance with the execution host's name.

If the queue name does not match the execution host name, the patched qlicserver code will still continue to operate. However, its internal accounting may be screwed up and it may treat the "mismatches" as external jobs rather than internal jobs. People currently in this situation are on their own -- they'll have to manually modify the code fix the mismatch -- a simple hash associating queue names with hostnames should be pretty simple to integrate into the Qstat5 package.


Technical: The new Qstat5 package

The vast majority of the patch consists of a new package written to parse non-XML qstat output. For the benefit of people who may easily be able to see mistakes, errors and optimization routes the code has been posted here. This is a modified version of the code Mark Olesen provided in this mailing list post: http://gridengine.sunsource.net/servlets/ReadMsg?list=users&msgNo=14906


# --------------------------------------------------------------------------

package Qstat5;
use vars qw( $query $timeout );

BEGIN {
    $timeout = 15;
    $query   = $Sge::binary_path . "qstat -r -s prs";
}

sub timeout {
    my ( $caller, $value ) = @_;
    ( $value ||= 0 ) > 0 or $value = 15;
    $timeout = $value;
}

# --------------------------------------------------------------------------
#              PARSE SGE 5.3.x qstat output that looks like this:
#
#job-ID  prior name user state submit/start at     queue      master  ja-task-ID 
#-------------------------------------------------------------------------------
#      6     0 sge-1ett_d dag          qw    02/21/2006 14:15:39                           
#       Full jobname:     sge-1ett_dock
#       Hard Resources:   i_glide=4
#                         i_main=1
#      7     0 sge-1ett_d dag          qw    02/21/2006 14:20:29                           
#       Full jobname:     sge-1ett_dock
#       Hard Resources:   i_glide=4
#                         i_main=1
#                         i_main=1
#     17  0 sge-1ett_d dag        r   02/25/2006 08:51:53 bertha.q   MASTER         
#       Full jobname:     sge-1ett_dock
#       Master queue:     bertha.q
#       Hard Resources:   i_glide=4
#                         i_main=1
#
# --------------------------------------------------------------------------

# extract
#   
# return:
# HASHREF => {
#    complex => {
#       waiting => {
#          "*user" => count,
#       },
#       jobid => {
#          "user@machine nlicense" => occurances,
#          "user@machine nlicense" => occurances,
#       },
#    },
# }
#
sub query {
    my $caller  = shift;
    my $license = {};

    my @lines = Shell->cmd( timeout => $timeout, cmd => "$query" )
      or return $license;

    my ( $hash_info, $width ) = ( {} );

    my %regexp = (
        entries   => qr{Master queue:},
        states    => qr{^(r|s|qw)$},
        h_lic_req => qr{(i_glide=\d+|i\_main=\d+)},
    );

    # skip header
    while (@lines) {
        ( shift @lines ) =~ /^-+\s*$/ and last;
    }

    # determine the width of the first column
    for ( $lines[0] ) {
        if ( $_ and /^(\s*\d+)\s*/ ) {
            $width = length $1;
        }
        else {
            warn "(WW):  qstat5 unknown problem\n";
            return $license;
        }
    }

    my ( $field, $jobid, $state, $user )     = ('');
    my ( $lic_name, $identifier, $host )     = undef;
    my ( $lic_count ) = 0;

    for (@lines) {
        ## capture jobID, username and job state
        if ( /^(\s*\d+)\s*/ and length($1) <= $width ) {
            $field = '';
            ( $jobid, $user, $state ) = (split)[ 0, 3, 4 ];
            
            ## reset license resource and license name counters for
            ## each new jobID ...
            ( $lic_count )  = 0;
            ( $lic_name, $identifier, $host ) = undef;

            next;
        } 
           ###
           # This is where we match "Master queue:"
           ### 
           if ( /^\s+($regexp{entries})\s+(.+?)\..*/ ) {
            #print STDERR "JobID $jobid: ENTRIES found --> $1 = $2 \n";
            $host = $2;
            next;
           }
           ###
           # This is where we match "<license name>=<count>"
           ###
           if ( /($regexp{h_lic_req})/ ) {
            #print STDERR "JobID $jobid: License data: $1 \n";
            ( $lic_name, $lic_count ) = split(/=/,$1);
            #print STDERR "JobID $jobid: (split) $lic_name = $lic_count \n";
           }
           ###
           # Now we know username, job state and any license hard requests
           # Lets put that info to use ...
           ###
           if ( $state =~ /$regexp{states}/ ) {
             if ( $state =~ /[qw]/  and ($lic_name) ) {
               # pending jobs get treated specially in the license hash               
               $license->{$lic_name}{waiting}{"*$user"} += $lic_count;
               next;   
             } else {
                ### 
                # if we got here we are dealing with active licenses
                ###
                $host or next;  # if we don't have $host, something is wrong
                $identifier = "\L$user\@$host";
              #print STDERR "Job:$jobid, $identifier $lic_name = $lic_count\n";
                $license->{$lic_name}{$jobid}{"$identifier $lic_count"}++;
                next;
             }
		   } else {
               # if we got here the job state was not something we
               # care about (ie not defined in $regex{states} 
               # this means we skip it and do nothing
               next;
           }
      }


    return $license;
}

1;

# --------------------------------------------------------------------------