Python: How to distinguish last loop of iteration

Sometimes you want to treat the last element in a list differently when you're iterating through. Here's how to do that:

prev = None
for x in range(10):
    if prev is not None:
        print "not last:", prev
    prev = x
print "last:", prev

Output:
not last: 0
not last: 1
not last: 2
not last: 3
not last: 4
not last: 5
not last: 6
not last: 7
not last: 8
last: 9

Python Padding with PKCS7

Here's the definition of PKCS7 padding (from RFC 2315):

RFC 2315, section 10.3, note #2:
     2.   Some content-encryption algorithms assume the
          input length is a multiple of k octets, where k > 1, and
          let the application define a method for handling inputs
          whose lengths are not a multiple of k octets. For such
          algorithms, the method shall be to pad the input at the
          trailing end with k - (l mod k) octets all having value k -
          (l mod k), where l is the length of the input. In other
          words, the input is padded at the trailing end with one of
          the following strings:

                   01 -- if l mod k = k-1
                  02 02 -- if l mod k = k-2
                              .
                              .
                              .
                k k ... k k -- if l mod k = 0

          The padding can be removed unambiguously since all input is
          padded and no padding string is a suffix of another. This
          padding method is well-defined if and only if k < 256;
          methods for larger k are an open issue for further study.

And here's how to implement it in python:

class PKCS7Encoder():
    """
    Technique for padding a string as defined in RFC 2315, section 10.3,
    note #2
    """
    class InvalidBlockSizeError(Exception):
        """Raised for invalid block sizes"""
        pass

    def __init__(self, block_size=16):
        if block_size < 2 or block_size > 255:
            raise PKCS7Encoder.InvalidBlockSizeError('The block size must be ' \
                    'between 2 and 255, inclusive')
        self.block_size = block_size

    def encode(self, text):
        text_length = len(text)
        amount_to_pad = self.block_size - (text_length % self.block_size)
        if amount_to_pad == 0:
            amount_to_pad = self.block_size
        pad = chr(amount_to_pad)
        return text + pad * amount_to_pad

    def decode(self, text):
        pad = ord(text[-1])
        return text[:-pad]

Example use:

>>> # basic use
>>> encoder = PKCS7Encoder()
>>> padded_value = encoder.encode('hi')
>>> padded_value
'hi\x0e\x0e\x0e\x0e\x0e\x0e\x0e\x0e\x0e\x0e\x0e\x0e\x0e\x0e'
>>> len(padded_value)
16
>>> encoder.decode(padded_value)
'hi'

>>> # empty string
>>> padded_value = encoder.encode('')
>>> padded_value
'\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10'
>>> len(padded_value)
16
>>> encoder.decode(padded_value)
''

>>> # string that is longer than a single block
>>> padded_value = encoder.encode('this string is long enough to span blocks')
>>> padded_value
'this string is long enough to span blocks\x07\x07\x07\x07\x07\x07\x07'
>>> len(padded_value)
48
>>> len(padded_value) % 16
0
>>> encoder.decode(padded_value)
'this string is long enough to span blocks'

>>> # using the max block size
>>> encoder = PKCS7Encoder(255)
>>> padded_value = encoder.encode('hi')
>>> len(padded_value)
255
>>> encoder.decode(padded_value)
'hi'

In django, formsets (formset_factory) allow you to list out a bunch of forms one after the other.  This is useful when you want to allow the user to add/remove objects.  For example, if you have a recipe application where the user can create new recipes, you could have an ingredient formset that allows the user to add more ingredients to a recipe.  In this case, you could also use an inline formset (inlineformset_factory) that is linked with the recipe that you're adding the ingredients for.

In my case, I had a number of questions that I wanted the user to answer.  There were a finite set of questions, and I wanted each question/answer to be listed out on a separate line on the page.  For my models I have a Question table and an Answer table.  The answers are linked to the customer's asset (don't worry about what an asset really is, it doesn't really matter) and are created at the same time the asset is created and from the same "create asset" page, so I used an inline formset to link the answers to the asset.  I thought this would be easy, but I ran into a number of problems and eventually just hand-wrote the forms.  But I'll go through the process here really for my own sake of remembering later what the limitations of inline formsets are.

models.py:

class Question(models.Model):
    question = models.CharField(max_length=250, editable=False)

    def __unicode__(self):
        return unicode(self.question)

class Answer(models.Model):
    asset = models.ForeignKey(Asset)
    question = models.ForeignKey(Question)
    answer = models.BooleanField(default=False)

    def __unicode__(self):
        return unicode('%s: %s' % (self.question, self.answer))

My tables actually had more to them, but for the sake of this example I've simplified them.

forms.py

class AnswerForm(ModelForm):
    to_save = False
    question = forms.ModelChoiceField(queryset=Question.objects.all(),
                                      widget=forms.HiddenInput())

    class Meta:
        model = Answer
        exclude = ('asset')

    def set_question(self, question):
        self.fields['answer'].label = question

views.py (there was a separate method for saving the formset, but I've unfortunately lost that code)

def get_answer_formset(asset, data):
    questions = StageQuestion.objects.all()
    extra = 0
    if not asset:
        # This is a brand new set, so we should pre-populate the questions.
        extra = len(questions)

    AnswerFormSet = inlineformset_factory(
            Asset,
            Answer,
            form=AnswerForm,
            can_delete=False,
            extra=extra,
            )

    formset = AnswerFormSet(
            data,
            prefix='que',
            instance=asset,
            )

    for form, question in zip(formset.forms, questions):
        form.set_question(question.question)
        form.initial = {'question': question.id}

    return formset

This code probably doesn't work too well anymore, since I changed it so many times, but here's a list of issues I ran into. The way a formset works is you tell it how many extra empty entries you want to display. When you have a brand new asset, extra would simply equal the number of Answers total, since I don't allow the user to actually add or remove Answers. So at first I set `extra` to equal the number of questions I wanted the user to answer, and I pre-populated each answer form with a unique question. This worked for creation, but when I wanted to edit an existing asset, it showed a list of all the questions and then a list of all the questions again, because it turned out `extra` is the number of (duh) *extra* empty questions to show. So then I set `extra` to zero, thinking that I don't want any *extra* empty questions, I just want the set of original questions, but you can't pre-define objects before putting them in the formset, so my icky solution was to set `extra` to zero only when an asset already exists and to set it to the number of questions otherwise. This worked, but it's certainly not clean.

By the way, look at that last part with the zip. This is pretty cool. I needed to loop through each form and seed it with the question. The `form.set_question` part sets the answer label so it correctly displays the question in the html. Django doesn't seem to have a label form, just a label attribute of each type of form, so I had to set the answer label to the question text and then hide the question field, otherwise the question would show up as a dropdown list of questions to select from. The `form.initial` part is my way of forcing initial data into the inline formset, since the init method doesn't allow it. This bothers me, actually, because regular formsets allow initial data, so why not inline formsets (especially since inline formsets subclass regular formsets)?

Back to the final issue. Formsets are "smart" enough to know not to add empty objects to the database, and an unchecked checkbox looks exactly like an empty object. This meant that only the True answers were saved in the database. This wouldn't seem so bad, after all it's saving DB space, but then if the user wants to edit their asset and maybe change some of their answers, you need to add back in the questions that they previously answered False on, and that's just ugly. Alternatively, you could change the saving of the formset to save the empty forms too, but I tried this and started running into issues with the forms missing key data, and finally I decided that I was hacking formsets so much that they were no longer useful. So I changed it to a list of forms. Trying to get formsets working took me a couple of days. The list of forms was done quickly and cleanly. When you look at it, it's clear what it's doing, and no more ugly hacks!

In conclusion, while it's possible none of this was clearly written enough to really understand, just know that formsets are great for what they were made for: allowing the user to add multiple objects of the same type; they are not good for a static list of objects that need to be created.

.screenrc

Here is my .screenrc file:

escape ^Jj
startup_message off
defscrollback 10000
vbell off
autodetach on
hardstatus alwayslastline
#hardstatus string "%{=b}%{G} %{b}%w%=%{kG}%C%A"
hardstatus string '%{= kG}[%{G}%{y}%H%{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(
%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B}%m/%d %{W}%c%{g}]'
shell -$SHELL

.vimrc

Here is my .vimrc file:


" Whitespace
set expandtab           " tabs -> spaces
set tabstop=4           " tab = 4 spaces
set shiftwidth=4        " used with << and >> to shift text over

set showmode            " when in INPUT mode, show INPUT text at all times
set ruler               " show current position at all times
set showcmd             " show incomplete commands at bottom of screen

set title               " set title to name of open file
set showmatch           " show matching brace/parenthese/bracket
" make the matching parenthesis not stand out so much
hi MatchParen cterm=underline ctermbg=black

" after leaving VIM, remove that annoying 'Thanks for flying Vim' title
let &titleold="~"

set visualbell          " flash screen instead of beep when error happens

" Search
set incsearch           " start searching as you type the search
set hls                 " highlight search matches
set ignorecase          " ignore case while searching
set smartcase           " if search string has uppercase, don't ignore case
set infercase           " if using auto-completion, this makes the word found have an appropriate case
" use spacebar to clear search highlighting and any message already displayed
nnoremap :silent nohecho

syntax on               " turn on syntax coloring
"syntax match Test "p"
"highlight Test ctermbg=red guibg=red
"syntax match Tab "\s"
"highlight Tab ctermbg=red guibg=red

au BufReadPost,FileReadPost * syntax match OverLength "\%>79v.\+"
hi OverLength ctermbg=blue ctermfg=white guibg=blue

au BufReadPost,FileReadPost * syntax match TrailingSpaces "\s\+$"
hi TrailingSpaces ctermbg=red guibg=red

" highlight text past 79 characters
match OverLength /\%>79v.\+/

" highlight trailing spaces at end of line
"2match TrailingSpaces /\s\+$/

"highlight TrailingSpaces ctermbg=red guibg=red
highlight OverLength ctermbg=blue ctermfg=white guibg=blue

" Function to remove superfluous white space from the end of a line
function! RemoveWhiteSpace()
    :%s/\s*$//g
    :'^
    "`.
endfunction

Cygwin Tweak

to get a cygwin terminal that actually uses bash and allows you to paste stuff into the window, edit /cygdrive/c/cygwin/Cygwin.bat to have this:


@echo off

C:
chdir C:\cygwin\bin

start rxvt -sr -sl 10000 -fg grey -bg black -fn courier -tn cygwin -e /bin/bash --login -i

pexpect and subprocess

If you want to run a program from your python script, you have two main options: subprocess and pexpect.  Recently I wanted to run ssh-keygen from a python script and I tried it both ways, so you can see a comparison.

pexpect:

import pexpect

def get_fingerprint_pexpect(key_file_name):
    command = 'ssh-keygen -lf %s' % (key_file_name,)
    child = pexpect.spawn(command)
    result_id = child.expect([
            pexpect.TIMEOUT,
            'No such file',
            'fail',
            'error',
            'not a public key',
            '[0-9a-f:]+',])
    if result_id == 0:
        raise Exception, 'Timeout occurred.'
    if result_id == 1:
        raise Exception, 'File "%s" does not exist.' % (key_file_name,)
    if result_id in (2, 3, 4):
        raise ValueError, 'Improperly formatted key.'

    fingerprint = child.match.group()
    return fingerprint

subprocess:

import re
import subprocess

NO_SUCH_FILE_ERROR = re.compile('No such file')
KEY_FORMAT_ERROR = re.compile('(fail|error|not a public key)')
FINGERPRINT = re.compile('[0-9a-f]{2}:[0-9a-f:]+')

def get_fingerprint_subprocess(key_file_name):
    command = ['ssh-keygen', '-lf', key_file_name,]
    result = subprocess.Popen(command,
            stdout = subprocess.PIPE,
            stderr = subprocess.STDOUT,
            stdin = subprocess.PIPE).communicate()[0]

no_such_file_error = NO_SUCH_FILE_ERROR.search(result)
    if no_such_file_error:
        raise Exception, 'File "%s" does not exist.' % (key_file_name,)

    key_format_error = KEY_FORMAT_ERROR.search(result)
    if key_format_error:
        raise ValueError, 'Improperly formatted key.'

    fingerprint = FINGERPRINT.search(result)
    if fingerprint:
        return fingerprint.group()

    raise Exception, 'Error generating key fingerprint.'

What the pexpect code is doing
With pexpect, you hand it a list of responses you expect to see.  In this case we are expecting to either see a timeout, a few potential error strings, or the fingerprint.  The content of this list is automatically converted into regular expressions by pexpect.

When we call "child.expect" with the list of responses we expect, we get back an id that refers to the index of the matching item in this list.  So when we say "if result_id == 1...", we're saying "if the resulting response was the item at index 1 (i.e. 'No such file')...".

We use the following regex to represent the fingerprint:

[0-9a-f]{2}:[0-9a-f:]+

Then later we say "child.match.group()" to get the actual content that was matched.

What the subprocess code is doing
First off, we split up the command into its component parts ("ssh-keygen", "-lf", "...", "...").  We redirect stdout, stderr, and stdin so that we can read it easily.  When we say "subprocess.PIPE", we're telling it to use subprocess to communicate with stdout and stdin.  When we set stderr equal to "subprocess.STDOUT", we're saying to send errors to stdout.

Calling subprocess.Popen(...).communicate() gives you a tuple of responses for stdout and stderr.  We choose to only look at stdout by adding the "[0]" to the end.

Subprocess doesn't do any matching for us like pexpect does.  It simply returns the response, so you much find out what it means on your own.  In this example, we have some regular expressions that we match against to find out what the result was.

Lastly, if the response was none of these, we treat it as an unexpected error, much like the pexpect timeout.

Conclusion
You may be thinking now that subprocess and pexpect are interchangeable, but they're not.  If you want to do something quick and easy, subprocess is nice.  But if you're dealing with more complicated things (for example, the "ssh" command requires a password, and it can't be passed in via stdin), you should stick with pexpect.  It can be tricky working out all of the regular expressions, but there's a pexpect log you can look at to see everything that pexpect is seeing.