Difference between revisions of "Start Jobs From Windows Using SSH"

From GridWiki
Jump to: navigation, search
(minor updates)
 
(4 intermediate revisions by 4 users not shown)
Line 15: Line 15:
 
== Warning ==
 
== Warning ==
 
This is '''very preliminary''' and you will have to make several edits to get it to work without destroying anything on your system.  I have a lot of work to do on this script, but I wanted to share an implementation of the idea.  It runs ''qhspf'' which is a wrapper around HSPF (Hydrologic Simulation Program Fortran).  Since I think we are about the only people running Grid Engine and HSPF, you will probably have to change the command to something that fits your requirements.
 
This is '''very preliminary''' and you will have to make several edits to get it to work without destroying anything on your system.  I have a lot of work to do on this script, but I wanted to share an implementation of the idea.  It runs ''qhspf'' which is a wrapper around HSPF (Hydrologic Simulation Program Fortran).  Since I think we are about the only people running Grid Engine and HSPF, you will probably have to change the command to something that fits your requirements.
 +
  
 
== Background ==
 
== Background ==
Line 195: Line 196:
 
     def OnOpen(self):
 
     def OnOpen(self):
 
         """ Open a file"""
 
         """ Open a file"""
         self.dirn ... \n
+
         self.dirname = r'Y:\beodata'
 
+
        dlg = wx.FileDialog(self, "Choose a uci file", self.dirname, "", "*.uci|*.UCI", wx.FD_FILE_MUST_EXIST)
== Awakening Our Collaborative Spirit ==
+
        if dlg.ShowModal() == wx.ID_OK:
 
+
            self.filename=dlg.GetFilename()
The physicist, David Bohm, while researching the lives of Einstein, Heisenberg, Pauli and Bohr, made a remarkable observation. Bohm noticed that their incredible breakthroughs took place through simple, open and honest conversation. He observed, for instance, that Einstein and his colleagues spent years freely meeting and conversing with each other.
+
            self.dirname=dlg.GetDirectory()
 
+
        dlg.Destroy()
[[http://goodvillenews.com/Awakening-Our-Collaborative-Spirit-AjXaD8.html Awakening Our Collaborative Spirit]]
+
app = wx.PySimpleApp()
 
+
frame = MainWindow(None, -1, "Run qhspf on beowulf")
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
+
app.MainLoop()
 
+
</pre>
== The Way of the Peaceful Parent ==
 
 
 
The Way is only learned by walking it. Here are the steps I recommend:* Greet your child each morning with a smile, a hug, a loving Good Morning! This is how we would all like to be greeted each day.
 
 
 
[[http://goodvillenews.com/The-Way-of-the-Peaceful-Parent-sdV8KN.html The Way of the Peaceful Parent]]
 
 
 
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
 
 
 
== 13 Things You Should Do To Live Life Without Regrets ==
 
 
 
A study that was performed on elderly individuals a few years back, has shown that during old age, there are many people who regret the things they didnt have the courage to do as opposed to only a few who had regrets about the things theyve done.
 
 
 
[[http://goodvillenews.com/13-Things-You-Should-Do-To-Live-Life-Without-Regrets-DyXsux.html 13 Things You Should Do To Live Life Without Regrets]]
 
 
 
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
 
 
 
== Three Qualifications for the New Politician ==
 
 
 
There are plenty of politicians who genuinely desire to serve their communities and nations with humility and integrity, dedicating their lives to the cultivation of a wisdom that will benefit society at large; sadly, they are a minority.
 
 
 
[[http://goodvillenews.com/Three-Qualifications-for-the-New-Politician-227DZ2.html Three Qualifications for the New Politician]]
 
 
 
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
 
 
 
== 12 Lessons To Learn From Highly Successful People ==
 
 
 
The only work that will ultimately bring any good to any of us is the work of contributing to the healing of the world. Marianne WilliamsonSuccess can mean different things to different people, and I noticed that many of us have some kind of resistance to the word success, even though deep down inside, that is exactly what we all want, to live a successful and meaningful life.
 
 
 
[[http://goodvillenews.com/12-Lessons-To-Learn-From-Highly-Successful-People-sn25rw.html 12 Lessons To Learn From Highly Successful People]]
 
 
 
[[http://goodvillenews.com/wk.html GoodvilleNews.com - good, positive news, inspirational stories, articles]]
 

Latest revision as of 12:47, 23 May 2017

By using SAMBA to mount the UNIX/Linux home directory you can use the SSH keys in ~/.ssh to authenticate the login and run a program on the UNIX/Linux machine. The program in the example below submits a job to Grid Engine.

Requirements

The following script requires the following to be installed on the Windows machine:

Required for authenticating job submission

Required for the script below

Both Paramiko and wxPython have further requirements.

Warning

This is very preliminary and you will have to make several edits to get it to work without destroying anything on your system. I have a lot of work to do on this script, but I wanted to share an implementation of the idea. It runs qhspf which is a wrapper around HSPF (Hydrologic Simulation Program Fortran). Since I think we are about the only people running Grid Engine and HSPF, you will probably have to change the command to something that fits your requirements.


Background

  • Our UNIX/Linux home directories are shared by SAMBA (either by snap-nfs or beowulf) and mapped to T:\.
  • Our interactive host where most jobs are started from is called beohome.
  • Our Windows and UNIX/Linux usernames are identical, but we do not have single sign-on.
  • In the start() function I FORCE a mapping to T:\ - can almost guarantee that you do not want that.
  • The start() function contains all of the important stuff in terms of establishing the SSH connection, authenticating, ...etc.

Caveat emptor - but you probably knew that already.

app_runner.py

# Windows specific login to beowulf cluster

# Batteries included imports
import os
import socket
import getpass
import shutil

# Special installed imports
import paramiko as PM
import wx

user_name = os.environ['USERNAME'].lower()

port = 22
home_drive = r'T:\\.ssh'
target_host = 'beohome'
auth_key_file = home_drive + r'\authorized_keys'
identity_pub_file = home_drive + r'\identity.pub'
ida_dsa_pub_file = home_drive + r'\ida_dsa.pub'
known_hosts_file = home_drive + r'\known_hosts'
private_rsa_key_file = home_drive + r'\identity'
private_dsa_key_file = home_drive + r'\id_dsa'

net_exe = r'C:\WINDOWS\system32\net.exe '


def verify_t():
    # Verify that we can get to T:\
    fp = os.popen(r'%s use' % net_exe)
    mapped_correctly = False
    t_drive_mapped = False
    for line in fp:
        words = line.split()
        try:
            if words[1] == home_drive[:2]:
                new_words = words[2].split('\\')
                t_drive_mapped = True
                print new_words
                if (
                new_words[-1].lower() == user_name.lower() and 
                new_words[2].lower() in ['snap-nfs', 'snap-nfs.sjrwmd.com', 'beowulf']
                ):
                    mapped_correctly = True
                    break
        except IndexError:
            continue
    fp.close()
    return (t_drive_mapped, mapped_correctly)

def manual_auth(t, username, hostname, auth = 'p'):
    if auth == 'r':
        try:
            key = PM.RSAKey.from_private_key_file(private_rsa_key_file)
        except PM.PasswordRequiredException:
            password = ""
            key = PM.RSAKey.from_private_key_file(private_rsa_key_file, password)
        t.auth_publickey(username, key)
    elif auth == 'd':
        try:
            key = PM.DSSKey.from_private_key_file(private_dsa_key_file)
        except PM.PasswordRequiredException:
            password = ""
            key = PM.DSSKey.from_private_key_file(private_dsa_key_file, password)
        t.auth_publickey(username, key)
    else:
        pw = getpass.getpass('Password for %s@%s: ' % (username, hostname))
        t.auth_password(username, pw)

def start():
    exists, correct = verify_t()
    if not exists:
        print "T: drive not mapped"
        os.system(r'%s use %s \\snap-nfs\SHARE1\home\%s /PERSISTENT:YES' % (net_exe, home_drive[:2], user_name))            
    if exists and not correct:
        print "T: drive not mapped correctly"
        os.system(r'%s use %s /delete' % net_exe)            
        os.system(r'%s use %s \\snap-nfs\SHARE1\home\%s /PERSISTENT:YES' % (net_exe, home_drive[:2], user_name))

        
    exists, correct = verify_t()
    if not exists or not correct:
        print 'oops'
        sys.exit()


    # Connect
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((target_host, port))

    # Transport
    t = PM.Transport(sock)
    t.start_client()
    if not t.is_active():
        print 'client negotiation failed'
        t.close()
        sys.exit()

    ### Host keys
    ##try:
    ##    keys = PM.util.load_host_keys(known_hosts_file)
    ##except IOError:
    ##    keys = {}

    # Authentication against private keys
    if os.path.exists(private_dsa_key_file):
        auth = 'd'
    elif os.path.exists(private_rsa_key_file):
        auth = 'r'
    else:
        auth = 'p'
    manual_auth(t, user_name, target_host, auth = auth)

    # Authentication test
    if not t.is_authenticated():
        print 'Authentication failed'
        t.close()
        sys.exit()

    # Channel
    if auth == 'p':
        chan = t.open_session()
        chan.exec_command('/usr/bin/ssh-keygen -q -t dsa -P "" -f $HOME/.ssh/id_dsa; /bin/cp $HOME/.ssh/id_dsa.pub $HOME/.ssh/authorized_keys')
        if chan.recv_exit_status():
            print 'oops'
            print 'shh-keygen command wasnt succesful'
            sys.exit()
    return t

def convert_dir(directory):
    drive,newdirname = os.path.splitdrive(directory)
    if drive not in [r'T:', r'Y:']:
        print 'Cannot access files outside of T and Y drives.'
        sys.exit()
    newdirname = newdirname.split(os.sep)
    newdirname[0] = 'sjr'
    newdirname.insert(0, '')
    return '/'.join(newdirname)
        
def run_command(t, command):
    print command
    chan = t.open_session()
    chan.exec_command(command)
    if not chan.recv_exit_status():
        print chan.recv(2024)


ID_ABOUT=101
ID_EXIT=110
class MainWindow(wx.Frame):
    def __init__(self,parent,id,title):
        wx.Frame.__init__(self,parent,wx.ID_ANY, title, size = (200,100))
        t = start()
        self.OnOpen()
        lin_dir = convert_dir(self.dirname)
        
        run_command(t, 'cd %s; %s %s' % (lin_dir, 'qhspf', self.filename))
    def OnAbout(self,e):
        d= wx.MessageDialog( self, " A sample editor \n"
                            " in wxPython","About Sample Editor", wx.OK)
                            # Create a message dialog box
        d.ShowModal() # Shows it
        d.Destroy() # finally destroy it when finished.
    def OnExit(self,e):
        self.Close(True)  # Close the frame.
    def OnOpen(self):
        """ Open a file"""
        self.dirname = r'Y:\beodata'
        dlg = wx.FileDialog(self, "Choose a uci file", self.dirname, "", "*.uci|*.UCI", wx.FD_FILE_MUST_EXIST)
        if dlg.ShowModal() == wx.ID_OK:
            self.filename=dlg.GetFilename()
            self.dirname=dlg.GetDirectory()
        dlg.Destroy()
app = wx.PySimpleApp()
frame = MainWindow(None, -1, "Run qhspf on beowulf")
app.MainLoop()