Multiple file support with scp

Share

Paramiko doesn’t provide a scp implementation, so I’ve been using my own for a while. http://blogs.sun.com/janp/entry/how_the_scp_protocol_works provides good documentation about the scp protocol, but it missed out on one detail I needed — how to send more than one file in a given session. In the end I implemented a simple scp logger to see what the protocol was doing during the copying of files. My logger said this:

    >>> New command invocation: /usr/bin/scp -d -t /tmp
    O: \0
    I: C0644 21 a\n
    O: \0
    I: file a file a file a\n\0
    O: \0
    I: C0644 21 b\n
    O: \0
    I: file b file b file b\n\0
    O: \0
    >>>stdin closed
    >>> stdout closed
    >>> stderr closed
    

It turns out its important to wait for those zeros by the way. So, here’s my implementation of the protocol to send more than one file. Turning this into paramiko code is left as an exercise for the reader.

    #!/usr/bin/python
    
    import fcntl
    import os
    import select
    import string
    import subprocess
    import sys
    import traceback
    
    def printable(s):
      out = ''
    
      for c in s:
        if c == '\n':
          out += '\\n'
        elif c in string.printable:
          out += c
        else:
          out += '\\%d' % ord(c)
    
      return out
    
    try:
      dialog = ['C0644 21 c\n',
                'file c file c file c\n\0',
                'C0644 21 d\n',
                'file d file d file d\n\0']
    
      proc = subprocess.Popen(['scp', '-v', '-d', '-t', '/tmp'],
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)
    
      r = [proc.stdout, proc.stderr]
      w = []
      e = [proc.stdout, proc.stderr]
    
      fl = fcntl.fcntl(proc.stdout, fcntl.F_GETFL)
      fcntl.fcntl(proc.stdout, fcntl.F_SETFL, fl | os.O_NONBLOCK)
      fl = fcntl.fcntl(proc.stderr, fcntl.F_GETFL)
      fcntl.fcntl(proc.stderr, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    
      stdin_closed = False
      while proc.returncode is None:
        (readable, _, errorable) = select.select(r, w, e)
    
        for flo in readable:
          if flo == proc.stdout:
            d = os.read(proc.stdout.fileno(), 1024)
            if len(d) > 0:
              sys.stdout.write('O: %s\n' % printable(d))
    
              if len(dialog) > 0:
                sys.stdout.write('I: %s\n' % printable(dialog[0]))
                os.write(proc.stdin.fileno(), dialog[0])
                dialog = dialog[1:]
    
              if len(dialog) == 0 and not stdin_closed:
                sys.stdout.write('>>> stdin closed\n')
                proc.stdin.close()
                stdin_closed = True
    
            else:
              sys.stdout.write('>>> stdout closed\n')
              r.remove(proc.stdout)
              e.remove(proc.stdout)
    
          elif flo == proc.stderr:
            d = os.read(proc.stderr.fileno(), 1024)
            if len(d) > 0:
              sys.stdout.write('E: %s\n' % printable(d))
            else:
              sys.stdout.write('>>> stderr closed\n')
              r.remove(proc.stderr)
              e.remove(proc.stderr)
    
          else:
            sys.stdout.write('>>> Unknown readable: %s: %s\n'
                             %(repr(flo), flo.read()))
    
        for flo in errorable:
          sys.stdout.write('>>> Error on %s\n' % repr(flo))
          r.remove(flo)
          e.remove(flo)
    
        proc.poll()
    
      print '#: %s' % proc.returncode
    
    except:
      exc = sys.exc_info()
      for tb in traceback.format_exception(exc[0], exc[1], exc[2]):
        print tb
        del tb
    
Share

Broken Angels

Share

This book is very different from Altered Carbon, as noted by many other online reviews. I found it very slow going, for a few reasons: it is quite long; it is very different from Altered Carbon in a way that almost feel like a bait and switch (Altered Carbon is a film noir detective novel, this is a hard core combat book with an alien influence); and Morgan has an annoying habit of providing emphasis with. periods. in the middle. of sentences which makes his work sometimes hard to parse. Overall and ok book and I like the alien stuff, but not what I was expecting and not as good as Altered Carbon.

[isbn: 9780575081253;0575081252]

Share

Zoe’s Tale

Share

This is a retelling of the events of The Last Colony from the perspective of Zoe, the teenaged adopted daughter of the colony leaders. The story makes the teenaged female narrator quite believable, although that is a little annoying at times. There is a lot of introductory material, and then a relatively rapid wrap up of the story line, which is common across all of the Old Ma’s War books that I have read. Overall I enjoyed this book, but not as much as Old Man’s War or The Ghost Brigades.

[isbn: 0765356198]
[awards: nominee hugo 2009]

Share

The Sagan Diary

Share

I think I’d be a little bit peeved if I’d paid the recommended retail for this book, because its a 100 page book (in an unusually large font), and they want $45 for it. Sure its a limited edition, but that’s a lot of money for what is basically a short story. I paid $18, and I still think that’s a lot of money for this much book. This book is a series of reflections on the life of Jane Sagan. These are beautiful, but if you’re expecting a book in the same vein as the rest of the Old Man’s War books you’ll be disappointed.

[isbn: 9781596061033;1596061030]

Share

Snow Crash

Share

I’d read this book before, but a long time ago and I decided it was time for a re-read. Its a good book, although the exposition about Sumerian history feels like a first attempt at the style of exposition used in later books like Quicksilver and I think its not as well done here. The story is pretty good sci-fi, even if the plot feels a little dated today. Overall a good book, but not Stephenson’s best.

[isbn: 0140232923;0553562614]

Share

Ender’s Game

Share

I’ve read this book before, many years ago. I figured I should re-read it, given how much I love the short story. Unfortunately, I think the short story is better than the novelization. The novel tends to try to explain too much, although the last chapter is a worthy addition. I’m sure I’ll still read the rest in the series though, as there is more to see in this universe.

[award: winner nebula 1985; winner hugo 1986; locus_short_fiction nominee 1978; locus_novel nominee 1987]
[isbn: 0812550706;185723720X]

Share

The Last Colony

Share

All of the Old Men’s War books (Old Man’s War and The Ghost Brigades) have started slowly and built up to a climax. That’s been good because its given some time for background which makes the second phase of the book all the more fully formed. This book is the same, although I think the introduction is more long winded than previously, and the whole thing gets wrapped up surprisingly quickly. Overall a good book, but not as good as the previous two.

[award: nominee hugo 2008]
[isbn: 0765316978]

Share

Registered

Share

I’m registered for LCA 2011! I wasn’t expecting to be able to come due to a clash with a work event, but then they moved that event. So, I’m pretty happy about that. Have you registered yet? Early bird closes soon…

Share

The Ghost Brigades

Share

This book is a really good sequel, and just as good as Old Man’s War. While some of the characters reappear, the story stands on its own and is quite entertaining. I enjoyed this book a lot. There’s something about bright green genetically engineered super soldiers killing aliens that makes me happy.

[award: nominee prometheus 2007]
[isbn: 0765354063]

Share