Multiple file support with scp

Share

Paramiko doesn’t provide a scp implementation, so I’ve been using my own for a while. http://blogs.sun.com/janp/entry/how_the_scp_protocol_works provides good documentation about the scp protocol, but it missed out on one detail I needed — how to send more than one file in a given session. In the end I implemented a simple scp logger to see what the protocol was doing during the copying of files. My logger said this:

    >>> New command invocation: /usr/bin/scp -d -t /tmp
    O: \0
    I: C0644 21 a\n
    O: \0
    I: file a file a file a\n\0
    O: \0
    I: C0644 21 b\n
    O: \0
    I: file b file b file b\n\0
    O: \0
    >>>stdin closed
    >>> stdout closed
    >>> stderr closed
    

It turns out its important to wait for those zeros by the way. So, here’s my implementation of the protocol to send more than one file. Turning this into paramiko code is left as an exercise for the reader.

    #!/usr/bin/python
    
    import fcntl
    import os
    import select
    import string
    import subprocess
    import sys
    import traceback
    
    def printable(s):
      out = ''
    
      for c in s:
        if c == '\n':
          out += '\\n'
        elif c in string.printable:
          out += c
        else:
          out += '\\%d' % ord(c)
    
      return out
    
    try:
      dialog = ['C0644 21 c\n',
                'file c file c file c\n\0',
                'C0644 21 d\n',
                'file d file d file d\n\0']
    
      proc = subprocess.Popen(['scp', '-v', '-d', '-t', '/tmp'],
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)
    
      r = [proc.stdout, proc.stderr]
      w = []
      e = [proc.stdout, proc.stderr]
    
      fl = fcntl.fcntl(proc.stdout, fcntl.F_GETFL)
      fcntl.fcntl(proc.stdout, fcntl.F_SETFL, fl | os.O_NONBLOCK)
      fl = fcntl.fcntl(proc.stderr, fcntl.F_GETFL)
      fcntl.fcntl(proc.stderr, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    
      stdin_closed = False
      while proc.returncode is None:
        (readable, _, errorable) = select.select(r, w, e)
    
        for flo in readable:
          if flo == proc.stdout:
            d = os.read(proc.stdout.fileno(), 1024)
            if len(d) > 0:
              sys.stdout.write('O: %s\n' % printable(d))
    
              if len(dialog) > 0:
                sys.stdout.write('I: %s\n' % printable(dialog[0]))
                os.write(proc.stdin.fileno(), dialog[0])
                dialog = dialog[1:]
    
              if len(dialog) == 0 and not stdin_closed:
                sys.stdout.write('>>> stdin closed\n')
                proc.stdin.close()
                stdin_closed = True
    
            else:
              sys.stdout.write('>>> stdout closed\n')
              r.remove(proc.stdout)
              e.remove(proc.stdout)
    
          elif flo == proc.stderr:
            d = os.read(proc.stderr.fileno(), 1024)
            if len(d) > 0:
              sys.stdout.write('E: %s\n' % printable(d))
            else:
              sys.stdout.write('>>> stderr closed\n')
              r.remove(proc.stderr)
              e.remove(proc.stderr)
    
          else:
            sys.stdout.write('>>> Unknown readable: %s: %s\n'
                             %(repr(flo), flo.read()))
    
        for flo in errorable:
          sys.stdout.write('>>> Error on %s\n' % repr(flo))
          r.remove(flo)
          e.remove(flo)
    
        proc.poll()
    
      print '#: %s' % proc.returncode
    
    except:
      exc = sys.exc_info()
      for tb in traceback.format_exception(exc[0], exc[1], exc[2]):
        print tb
        del tb
    
Share

Implementing SCP with paramiko

Share

Regular readers will note that I’ve been interested in how scp works and paramiko for the last couple of days. There are previous examples of how to do scp with paramiko out there, but the code isn’t all on one page, you have to read through the mail thread and work it out from there. I figured I might save someone some time (possibly me!) and note a complete example of scp with paramiko…

    #!/usr/bin/python
    
    # A simple scp example for Paramiko.
    # Args:
    #   1: hostname
    #   2: username
    #   3: local filename
    #   4: remote filename
    
    import getpass
    import os
    import paramiko
    import socket
    import sys
    
    # Socket connection to remote host
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((sys.argv[1], 22))
    
    # Build a SSH transport
    t = paramiko.Transport(sock)
    t.start_client()
    t.auth_password(sys.argv[2], getpass.getpass('Password: '))
    
    # Start a scp channel
    scp_channel = t.open_session()
    
    f = file(sys.argv[3], 'rb')
    scp_channel.exec_command('scp -v -t %s\n'
                             % '/'.join(sys.argv[4].split('/')[:-1]))
    scp_channel.send('C%s %d %s\n'
                     %(oct(os.stat(sys.argv[3]).st_mode)[-4:],
                       os.stat(sys.argv[3])[6],
                       sys.argv[4].split('/')[-1]))
    scp_channel.sendall(f.read())
    
    # Cleanup
    f.close()
    scp_channel.close()
    t.close()
    sock.close()
    
Share