Tag: paramiko

Multiple file support with scp

Paramiko doesn’t provide a scp implementation, so I’ve been using my own for a while.

http://blogs.sun.com/janp/entry/how_the_scp_protocol_works (link now unfortunately dead) provides good documentation about the scp protocol, but it missed out on one detail I needed — how to send more than one file in a given session. In the end I implemented a simple scp logger to see what the protocol was doing during the copying of files. My logger said this:

>>> New command invocation: /usr/bin/scp -d -t /tmp
O: \0
I: C0644 21 a\n
O: \0
I: file a file a file a\n\0
O: \0
I: C0644 21 b\n
O: \0
I: file b file b file b\n\0
O: \0
>>>stdin closed
>>> stdout closed
>>> stderr closed

It turns out its important to wait for those zeros by the way. So, here’s my implementation of the protocol to send more than one file. Turning this into paramiko code is left as an exercise for the reader.

#!/usr/bin/python

import fcntl
import os
import select
import string
import subprocess
import sys
import traceback

def printable(s):
  out = ''

  for c in s:
    if c == '\n':
      out += '\\n'
    elif c in string.printable:
      out += c
    else:
      out += '\\%d' % ord(c)

  return out

try:
  dialog = ['C0644 21 c\n',
            'file c file c file c\n\0',
            'C0644 21 d\n',
            'file d file d file d\n\0']

  proc = subprocess.Popen(['scp', '-v', '-d', '-t', '/tmp'],
                          stdin=subprocess.PIPE,
                          stdout=subprocess.PIPE,
                          stderr=subprocess.PIPE)

  r = [proc.stdout, proc.stderr]
  w = []
  e = [proc.stdout, proc.stderr]

  fl = fcntl.fcntl(proc.stdout, fcntl.F_GETFL)
  fcntl.fcntl(proc.stdout, fcntl.F_SETFL, fl | os.O_NONBLOCK)
  fl = fcntl.fcntl(proc.stderr, fcntl.F_GETFL)
  fcntl.fcntl(proc.stderr, fcntl.F_SETFL, fl | os.O_NONBLOCK)

  stdin_closed = False
  while proc.returncode is None:
    (readable, _, errorable) = select.select(r, w, e)

    for flo in readable:
      if flo == proc.stdout:
        d = os.read(proc.stdout.fileno(), 1024)
        if len(d) > 0:
          sys.stdout.write('O: %s\n' % printable(d))

          if len(dialog) > 0:
            sys.stdout.write('I: %s\n' % printable(dialog[0]))
            os.write(proc.stdin.fileno(), dialog[0])
            dialog = dialog[1:]

          if len(dialog) == 0 and not stdin_closed:
            sys.stdout.write('>>> stdin closed\n')
            proc.stdin.close()
            stdin_closed = True

        else:
          sys.stdout.write('>>> stdout closed\n')
          r.remove(proc.stdout)
          e.remove(proc.stdout)

      elif flo == proc.stderr:
        d = os.read(proc.stderr.fileno(), 1024)
        if len(d) > 0:
          sys.stdout.write('E: %s\n' % printable(d))
        else:
          sys.stdout.write('>>> stderr closed\n')
          r.remove(proc.stderr)
          e.remove(proc.stderr)

      else:
        sys.stdout.write('>>> Unknown readable: %s: %s\n'
                         %(repr(flo), flo.read()))

    for flo in errorable:
      sys.stdout.write('>>> Error on %s\n' % repr(flo))
      r.remove(flo)
      e.remove(flo)

    proc.poll()

  print '#: %s' % proc.returncode

except:
  exc = sys.exc_info()
  for tb in traceback.format_exception(exc[0], exc[1], exc[2]):
    print tb
    del tb

Calculating a SSH host key with paramiko

I needed to compare a host key from something other than a known_hosts file with what paramiko reports as part of the SSH connection today. If you must know, the host keys for these machines are retrieved a XMLRPC API… It turned out to be a lot easier than I thought. Here’s how I produced the host key entry as it appears in that API (as well as in the known_hosts file):

    #!/usr/bin/python

# A host key calculation example for Paramiko.
# Args:
#   1: hostname

import base64
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
key = t.get_remote_server_key()

print '%s %s' %(key.get_name(),
                base64.encodestring(key.__str__()).replace('\n', ''))

t.close()
sock.close()

Note that I could also have constructed a paramiko key object based on the output of the XMLRPC API and then compared those two objects, but I prefer the human readable strings.

paramiko exec_command timeout

I have a paramiko program which sshs to a large number of machines, and sometimes it hits a machine where Channel.exec_command() doesn’t return. I know this is a problem with the remote machine, because the same thing happens when I try to ssh to the machine from the command line. However, I don’t have any way of determining which machines are broken beforehand.

Paramiko doesn’t support a timeout for exec_command(), so I am looking for a generic way of running a function call with a timeout. I can see sample code which does this using threads, but that’s pretty ugly. I can’t use SIGALARM because I am not running on the main thread.

Can anyone think of a better way of doing this?

Weird paramiko problem

I had a strange paramiko problem the other day. Sometimes executing a command through a channel (via the exec_command() call) would result in an exit code being returned, but no stdout or stderr. This was for a command I was absolutely sure always returns output, and it wasn’t consistent — I’d run batches of commands and about 10% of them would fail, but not always on the same machine and not always at the same time. I spent ages looking at my code, and the code for the command running at the other end of the channel.

Then it occurred to me that this seemed a lot like a race condition. I started looking at the code for the paramiko Channel class, and ended up deciding that the answer was to check that the eof_received member variable was true before trying to close the channel.

It turns out this just works. I’ve my code running commands for a couple of days now and have had zero more instances of the “no output, but did exit” error. So, there you go. Its a shame that member variable doesn’t have accessors and isn’t documented though. I guess that makes my code a little more fragile than I would be happy with.

Executing a command with paramiko

I wanted to provide a simple example of how to execute a command with paramiko as well. This is quite similar to the scp example, but is nicer than executing a command in a shell because there isn’t any requirement to do parsing to determine when the command has finished executing.

    #!/usr/bin/python

# A simple command example for Paramiko.
# Args:
#   1: hostname
#   2: username
#   3: command to run

import getpass
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
t.auth_password(sys.argv[2], getpass.getpass('Password: '))

# Start a cmd channel
cmd_channel = t.open_session()
cmd_channel.exec_command(sys.argv[3])

data = cmd_channel.recv(1024)
while data:
  sys.stdout.write(data)
  data = cmd_channel.recv(1024)

# Cleanup
cmd_channel.close()
t.close()
sock.close()

Implementing SCP with paramiko

Regular readers will note that I’ve been interested in how scp works and paramiko for the last couple of days. There are previous examples of how to do scp with paramiko out there, but the code isn’t all on one page, you have to read through the mail thread and work it out from there. I figured I might save someone some time (possibly me!) and note a complete example of scp with paramiko…

#!/usr/bin/python

# A simple scp example for Paramiko.
# Args:
#   1: hostname
#   2: username
#   3: local filename
#   4: remote filename

import getpass
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
t.auth_password(sys.argv[2], getpass.getpass('Password: '))

# Start a scp channel
scp_channel = t.open_session()

f = file(sys.argv[3], 'rb')
scp_channel.exec_command('scp -v -t %s\n'
                         % '/'.join(sys.argv[4].split('/')[:-1]))
scp_channel.send('C%s %d %s\n'
                 %(oct(os.stat(sys.argv[3]).st_mode)[-4:],
                   os.stat(sys.argv[3])[6],
                   sys.argv[4].split('/')[-1]))
scp_channel.sendall(f.read())

# Cleanup
f.close()
scp_channel.close()
t.close()
sock.close()