Sunday, December 19, 2010

The Rights and Wrongs of Dynamic Pages

I was recently reading this article over at The Economist. The content of the article aside, it made one other thing come to mind: with great power comes great responsibility.

As I began reading the article, and thus scrolling, all of a sudden my screen began to look a cluttered mess. A full-screen-width bar dropped down from the top with please-for-the-love-of-God-share-this-article-on-all-your-social-networking-sites buttons and a search box. Besides being redundant since those features are all already embedded on the page, it was distracting. About the same time a square box slides up from the bottom telling me I need to subscribe to the magazine, again covering up the content of the article. Again, there is already an advertisement-like area near the top of the page offering four free issues and telling you to subscribe.

The passive versions of these features I am fine with, but the two slide-in boxes are too much because they happen as one has already begun to read, thus distracting your attention and covering up the content you're there to see in the first place. It is a very in-your-face type of pressure that most people do not approve, just like extremely loud commercials.

Okay, so if The Economist is the Comcast of internet news, what's an example of dynamic pages done right in that area? I think the New York Times does it right. Go read an article (this one I chose at random) or simply scroll through it. Nothing pops up to annoy you, everything happens on page load. The one exception is when you reach the bottom of the article a box slides in--and not over the content you're reading!--letting you know of related articles you may be interested in. This is actually helpful rather than self-serving like The Economists's dynamic content.

So, while we are all enamored with the eye candy of modern Ajax development remember to take a critical eye to it and note those who are using it well. I think I'll go read some more NYT.

Wednesday, December 1, 2010

AJAX Uploads in Django (with a little help from jQuery)

*** April 4, 2010: This post is a bit outdated and does not work with Django 1.3 final's stricter CSRF enforcement. I have a new post that is much more up-to-date, cleaner, and easier to follow, especially because it uses the github repo I created that holds the changes that need to be made to the file uploader's javascript.***

Part of my current project is creating an area where files need to be uploaded in a snazzier way that the normal "browse for a single file" sort of forms. The idea is to have a download button with the same OS chrome file dialog, but one that allows multiple files to be selected. Additionally, the even snappier part, is that HTML5's drag-and-drop functionality should also be available where supported.

After some searching I found a couple of sites that pointed me in the right direction.

  • AJAX Upload handles the client-side end of the upload process, including the required multiple-select and drag-and-drop (for browsers that support it). It even handles graceful fallback to iframe uploads for browsers that do not support those advanced features (Opera, IE, etc.).
  • On the Django side, I found this site, which gives some pointers to get it working with Django 1.2's CSRF token. The problem I encountered is simply passing the CSRF token to Ajax Upload via its params will not work, because Ajax Upload sends it in the querystring and Django expects it as POST data.

Because neither of those gave me the whole picture, I had to piece things together on my own. Here is the straight-skinny on my findings for getting Ajax Upload to work with Django.


Ajax Upload handles the client-side very seamlessly and only gives one challenge to the programmer: it passes the file either as the raw request, for the "advanced" mode, or as the traditional form file for the "basic" mode. Thus, on the Django side, the receiving function must be written to process both cases. In the raw request version, reading the request without Django blowing memory was a bit of a challenge. I at first tried reading the data into Django's SimpleFileUpload. It is an in-memory class though, so it runs into issues with large files. Next I tried reading/writing the data through Python functions, which had similar problems. The "works all the time" solution requires Django 1.3, which is to use its new "http request file-like interface" to read from the request. If you're using Django 1.2 and figure out another way to read the request data for all file sizes please comment! I discuss these solutions a little more in depth at this Stack Overflow question of mine.


First is to get AJAX Upload installed by getting the JS and CSS files wherever is appropriate and linking to them in your Django templates. Also make sure you have set up Django's file upload handlers. Next is setting it up on the site.

The Web (Client) Side


This is the HTML code that will house the upload button/drag area so place it appropriately.

<div id="file-uploader">       
        <p>Please enable JavaScript to use file uploader.</p>
        <!-- or put a simple form for upload here -->


You probably want to dump this in the same HTML/template file, but it is up to you.

function generate_ajax_uploader( $url, $csrf_token, $success_func )
  var uploader = new qq.FileUploader( {
    action: $url,
    element: $('#file-uploader')[0],
    onComplete: function( id, fileName, responseJSON ) {
      /* you probably want to handle the case when responseJSON.success is false,
         which happens when the Django view could not save the file */
      if( responseJSON.success )
        $success_func( responseJSON ) ;
    params: {
      'csrfmiddlewaretoken': $csrf_token, /* MUST call it csrfmiddlewaretoken to work with my later changes to Ajax Upload */
  } ) ;

A little explanation is probably needed here.

  • I have wrapped the code inside of a function that generates the uploader because I use it on a couple different pages that require different URLs. You can easily strip off the function part and simply place it in a regular <script> block for use on a single page.
  • It is probably simplest to use the url template tag to fill in the action, which is the URL that gets the ajax data and does the server-side processing. I use the url template tag to construct the $url parameter to this function, but if you are removing the function part put the tag directly with action: or you can hard-code the URL you want as a string.
  • I use the onComplete callback to pass returned json to another "success function" that parses the json and add information to a table on the page. Again, this is not necessary, but I thought it would be useful to show how this could work. The upload plugin itself will say whether the file upload was a success based on the returned json.
  • jQuery is used to grab the appropriate part of the div. If you are not using jQuery use whatever method is appropriate for your system to get the file-uploader DOM element. Using regular Javascript you could do document.getElementById('file-uploader'), as Valum uses in the examples on his site.

Ajax Upload Modifications

I found the easiest way to get the CSRF token piece going was to modify Ajax Upload itself, unfortunately (I hate editing libraries since they need to be re-updated at each new release). Around line 1100 of fileuploader.js you will find the line "var form = ..." within UploadHandlerForm's _createForm method. Replace this line with the following:

var form = null ; 
if( params.csrfmiddlewaretoken )
  var csrf = '<div style="display:none"><input type="hidden" name="csrfmiddlewaretoken" value="' + params.csrfmiddlewaretoken + '" /></div>' ;
  form = qq.toElement('<form method="post" enctype="multipart/form-data">' + csrf + '</form>');
  delete params.csrfmiddlewaretoken
  form = qq.toElement('<form method="post" enctype="multipart/form-data"></form>');

All this code does is search for the CSRF token, and if it is present insert it into the form in the way Django expects to receive it.

The Server (Django) Side

Django URLs

It is best to have two views for this setup to work: one to display the upload page and one to process the upload file. First, the URLs

url( r'/project/ajax_upload/$', ajax_upload, name="ajax_upload" ),
url( r'/project/$', upload_page, name="upload_page" ),


First is the upload_page view, which is going to display the page with which the user interacts. This is a simple skeleton, add whatever your template needs.

from django.middleware.csrf import get_token
def upload_page( request ):
  ctx = RequestContext( request, {
    'csrf_token': get_token( request ),
  } )
  return render_to_response( 'upload_page.html', ctx )

Next is the view to handle the upload. Remember that this code must handle two situations: the case of an AJAX-style upload for the "advanced" mode and a form upload for the "basic" mode.

def save_upload( uploaded, filename, raw_data ):
  ''' raw_data: if True, upfile is a HttpRequest object with raw post data
      as the file, rather than a Django UploadedFile from request.FILES '''
    from io import FileIO, BufferedWriter
    with BufferedWriter( FileIO( filename, "wb" ) ) as dest:
      # if the "advanced" upload, read directly from the HTTP request 
      # with the Django 1.3 functionality
      if raw_data:
        foo = 1024 )
        while foo:
          dest.write( foo )
          foo = 1024 ) 
      # if not raw, it was a form upload so read in the normal Django chunks fashion
        for c in uploaded.chunks( ):
          dest.write( c )
  except IOError:
    # could not open the file most likely
    return False

def ajax_upload( request ):
  if request.method == "POST":    
    # AJAX Upload will pass the filename in the querystring if it is the "advanced" ajax upload
    if request.is_ajax( ):
      # the file is stored raw in the request
      upload = request
      is_raw = True
        filename = request.GET[ 'qqfile' ]
      except KeyError: 
        return HttpResponseBadRequest( "AJAX request not valid" )
    # not an ajax upload, so it was the "basic" iframe version with submission via form
      is_raw = False
      if len( request.FILES ) == 1:
        # FILES is a dictionary in Django but Ajax Upload gives the uploaded file an
        # ID based on a random number, so it cannot be guessed here in the code.
        # Rather than editing Ajax Upload to pass the ID in the querystring, note that
        # each upload is a separate request so FILES should only have one entry.
        # Thus, we can just grab the first (and only) value in the dict.
        upload = request.FILES.values( )[ 0 ]
        raise Http404( "Bad Upload" )
      filename =
    # save the file
    success = save_upload( upload, filename, is_raw )

    # let Ajax Upload know whether we saved it or not
    import json
    ret_json = { 'success': success, }
    return HttpResponse( json.dumps( ret_json ) )

And that's it, go have some fun!

***Edit: A few errors in the source have been fixed, thank you for your comments!