How to stream an HttpResponse with Django
PythonDjangoStreamingPython Problem Overview
I'm trying to get the 'hello world' of streaming responses working for Django (1.2). I figured out how to use a generator and the yield function. But the response still not streaming. I suspect there's a middleware that's mucking with it -- maybe ETAG calculator? But I'm not sure how to disable it. Can somebody please help?
Here's the "hello world" of streaming that I have so far:
def stream_response(request):
resp = HttpResponse( stream_response_generator())
return resp
def stream_response_generator():
for x in range(1,11):
yield "%s\n" % x # Returns a chunk of the response to the browser
time.sleep(1)
Python Solutions
Solution 1 - Python
You can disable the ETAG middleware using the condition decorator. That will get your response to stream back over HTTP. You can confirm this with a command-line tool like curl. But it probably won't be enough to get your browser to show the response as it streams. To encourage the browser to show the response as it streams, you can push a bunch of whitespace down the pipe to force its buffers to fill. Example follows:
from django.views.decorators.http import condition
@condition(etag_func=None)
def stream_response(request):
resp = HttpResponse( stream_response_generator(), content_type='text/html')
return resp
def stream_response_generator():
yield "<html><body>\n"
for x in range(1,11):
yield "<div>%s</div>\n" % x
yield " " * 1024 # Encourage browser to render incrementally
time.sleep(1)
yield "</body></html>\n"
Solution 2 - Python
A lot of the django middleware will prevent you from streaming content. Much of this middleware needs to be enabled if you want to use the django admin app, so this can be an annoyance. Luckily this has been resolved in the django 1.5 release. You can use the StreamingHttpResponse to indicate that you want to stream results back and all the middleware that ships with django is aware of this and acts accordingly to not buffer your content output but send it straight down the line. Your code would then look like the following to use the new StreamingHttpResponse object.
def stream_response(request):
return StreamingHttpResponse(stream_response_generator())
def stream_response_generator():
for x in range(1,11):
yield "%s\n" % x # Returns a chunk of the response to the browser
time.sleep(1)
Note on Apache
I tested the above on Apache 2.2 with Ubuntu 13.04. The apache module mod_deflate which was enabled by default in the setup I tested will buffer the content you are trying to stream until it reaches a certain block size then it will gzip the content and send it to the browser. This will prevent the above example from working as desired. One way to avoid this is to disable mod_deflate by putting the following line in your apache configuration:
SetEnvIf Request_URI ^/mysite no-gzip=1
This is discussed more in the https://stackoverflow.com/q/1922934/1699750 question.