The Occasional Occurence

I've really been enjoying working with Python iterators lately. They make working with iterative tasks so flexible. Lately I've been working on adding progress tracking to an application so that we can give our users some feedback on where the application is at in various long-running tasks.

Since the long-running tasks in question are coded as iterators, I'm thinking about using something like the following:

def progtracker(iterator, name, store, workunits, step=1):
    """Store the progress of a named iterator in a dictionary-like store."""
    count = 0
    workunits = float(workunits)
    for part in iterator:
        yield part
        count += 1
        if count % step == 0:
            store[name] = count/workunits

That iterator can be used to track the progress of any bounded iterator. I guess it has to be more than a bounded iterator - you have to know the upper bound as well. Here's a semi-complete example:

progress = {} # some dict-like store
total = len(records) + 1 # account for all the rows plus the header row
docgen = create_document(records, schema)
p_docgen = progtracker(docgen, "docgen-task-x", progress, total, 5)
f = open('outfile.fmt', 'w')
for line in p_docgen():
    f.write(line)
    update_progress_display(progress)
f.close()

I've got another iterator recipe that uses the processing package to offload the processing of an iterator to a subprocess. I'll see if I can get that posted here soon.

cw

UPDATE: corrected error in example (didn't pass the store to the progtracker - thanks Michael)