The Occasional Occurence
Reading Chunked HTTP/1.1 Responses
April 02, 2008 at 12:35 AM | categories: Python, work, computing, cherrypy, GeneralFor work today I wanted a way to iterate over an HTTP response with chunked transfer-coding on a chunk-for-chunk basis. I didn't see a builtin way to do that with httplib. It supports chunked reads but you have to specify the amount that you want to read if you don't want it to buffer. I just wanted it to read and yield each chunk that it received from the server.
For my first crack at it I really just tried to use the httplib basics:
import httplib conn = httplib.HTTPConnection('localhost:8080') conn.request('GET', '/') r = conn.getresponse() data = r.read(10) while data: print data data = r.read(10)
That worked but since I won't know the chunk size in real-life, I would probably get output similar to this:
Chunk 0 Ch unk 1 Chun k 2 Chunk 3 Chunk 4 ...
I really wanted that chunk-for-chunk iteration. After taking a look at the very readable httplib source this evening, it wasn't very hard to accomplish. I basically just took the httplib.HTTPResponse._read_chunked method and modified it to be a generator. I subclassed HTTPResponse and stuck my generator in an __iter__ method. Behold; now you can do this sort of thing:
if __name__ == "__main__": import httplib import iresponse conn = httplib.HTTPConnection('localhost:8080') conn.response_class = iresponse.IterableResponse conn.request('GET', '/') r = conn.getresponse() for chunk in r: print chunk
With nice results like this:
Chunk 0 Chunk 1 Chunk 2 Chunk 3 Chunk 4 ...
You can download the iresponse module from my projects site. There is also a small CherryPy application that serves some data with chunked transfer-coding in case any of you want to fiddle with it.
cw