Monday, April 4, 2011

Ajax file uploads and CSRF (in Django 1.3, or possibly other frameworks)

To begin, this is an update of my old post AJAX Uploads in Django (with a little help from jQuery). This guide is specific to Django, but my version of the file uploader can (theoretically, it is untested) be used with other web frameworks that use CSRF, like Ruby on Rails. You should be able to follow along with the guide and make adjustments as appropriate for your framework.

Required Software

  • My version of Valum's file upload
  • Python 2.6+
  • Django 1.3+

If you are on an older version of Python and/or Django, reading the prior version of this post and especially this Stack Overflow question of mine may provide some help in adjusting the code. The only part that requires updated Python and Django is the save_upload function. The code uses buffered readers/writers and the 'with' keyword from Python 2.6+ (these parts can easily be changed I suspect) and reads from the raw HttpRequest, which comes with Django 1.3+. The Stack Overflow question has code I tried before moving up to requiring these newer software versions. It worked for small uploads below CD ISO size (700MB) and can probably be fixed to work with all uploads, I just found the Django 1.3+ solution easier and quicker at the time.

Overview

Ajax Upload handles the client-side very seamlessly and only gives one challenge to the programmer: it passes the file either as the raw request, for the "advanced" mode, or as the traditional form file for the "basic" mode. Thus, on the Django side, the receiving function must be written to process both cases. As the old post discusses, reading this raw request was a bit of trouble, and that is why I went with Django 1.3 as a requirement for my code.

Setup and Settings

First is to get AJAX Upload installed by downloading the latest version from my Github repo. This fork of Valum's original includes my changes as well as improvements from other forks that I need. As of this writing, I have added correct awareness in FileUploader of FileUploaderBasic's 'multiple' parameter and included David Palm's onAllComplete trigger. Once downloaded, grab fileuploader.js and fileuploader.css out of the client folder and place them wherever is appropriate for your setup. Finally, link them in your HTML via your Django templates.

The Web (Client) Side

HTML

This is the HTML code that will house the upload button/drag area so place it appropriately.

<div id="file-uploader">       
    <noscript>          
        <p>Please enable JavaScript to use file uploader.</p>
    </noscript>         
</div>

Javascript

You probably want to dump this in the same HTML/template file as the above, but it is up to you of course.

var uploader = new qq.FileUploader( {
    action: "{% url ajax_upload %}",
    element: $('#file-uploader')[0],
    multiple: true,
    onComplete: function( id, fileName, responseJSON ) {
      if( responseJSON.success )
        alert( "success!" ) ;
      else
        alert( "upload failed!" ) ;
    },
    onAllComplete: function( uploads ) {
      // uploads is an array of maps
      // the maps look like this: { file: FileObject, response: JSONServerResponse }
      alert( "All complete!" ) ;
    },
    params: {
      'csrf_token': '{{ csrf_token }}',
      'csrf_name': 'csrfmiddlewaretoken',
      'csrf_xname': 'X-CSRFToken',
    },
  } ) ;
}

Now, let's make some sense of that.

  • It is probably simplest to use the url template tag to fill in the action as I did above, but it could also be a hard-coded URL as a string. It is set here to match the URL config covered later in this guide.
  • The multiple option is not something that is not discussed in Valum's documentation that I found. Its purpose is to limit the uploader to allow you to determine whether it supports selecting/dragging multiple files for upload at a time. A value of true allows multiples, false will let it only do one at a time. In Valum's, this option is available to FileUploaderBasic, but not FileUploader, which is the class most people use. For my repo I chose to update FileUploader to be aware of the multiple option.
  • The onAllComplete callback is something added to my repo over Valum's that I got from David Palm's fork. It is called whenever the queue of uploads becomes empty. For example, if you drag/select 4 uploads, this will fire once all 4 have finished. If you then drag/select 2 more files for upload, this will fire again when those 2 are completed.
  • The params are set up so the uploader can interact with Django's CSRF framework properly. csrf_token is obviously the token itself, while csrf_name is the name of the input expected by Django for form submissions and csrf_xname is the HTTP header parameter it reads for AJAX requests. Why did I bother with making these last two parameters? Well, theoretically my version of the file uploader should work with other frameworks, which may expect different names for these. For example, Ruby on Rails will expect 'X-CSRF-Token' for AJAX requests and 'authenticity_token' for forms (I think).
  • jQuery is used to grab the appropriate part of the div. If you are not using jQuery use whatever method is appropriate for your system to get the file-uploader DOM element. Using regular Javascript you could do document.getElementById('file-uploader'), as Valum uses in the examples on his site.

The Server (Django) Side

Django URLs

It is best to have two views for this setup to work: one to display the upload page and one to process the upload file. The URLs need to be set in urls.py of course.

url( r'/project/ajax_upload/$', ajax_upload, name="ajax_upload" ),
url( r'/project/$', upload_page, name="upload_page" ),

Note that these may require some adjustments depending on how your urls.py is coded.

Views

First is the upload_page view, which is going to display the page with which the user interacts. This is a simple skeleton, add whatever your template needs.

from django.middleware.csrf import get_token
def upload_page( request ):
  ctx = RequestContext( request, {
    'csrf_token': get_token( request ),
  } )
  return render_to_response( 'upload_page.html', ctx )

Including the csrf_token in the context is very important, as earlier code depends on having this variable available. For some reason Django does not give you access to the token automatically in templates.

Next is the view to handle the upload. Remember that this code must handle two situations: the case of an AJAX-style upload for the "advanced" mode and a form upload for the "basic" mode. I split this code up into two functions: one to actually save the upload and the other the view.

def save_upload( uploaded, filename, raw_data ):
  ''' 
  raw_data: if True, uploaded is an HttpRequest object with the file being
            the raw post data 
            if False, uploaded has been submitted via the basic form
            submission and is a regular Django UploadedFile in request.FILES
  '''
  try:
    from io import FileIO, BufferedWriter
    with BufferedWriter( FileIO( filename, "wb" ) ) as dest:
      # if the "advanced" upload, read directly from the HTTP request 
      # with the Django 1.3 functionality
      if raw_data:
        foo = uploaded.read( 1024 )
        while foo:
          dest.write( foo )
          foo = uploaded.read( 1024 ) 
      # if not raw, it was a form upload so read in the normal Django chunks fashion
      else:
        for c in uploaded.chunks( ):
          dest.write( c )
      # got through saving the upload, report success
      return True
  except IOError:
    # could not open the file most likely
    pass
  return False

def ajax_upload( request ):
  if request.method == "POST":    
    if request.is_ajax( ):
      # the file is stored raw in the request
      upload = request
      is_raw = True
      # AJAX Upload will pass the filename in the querystring if it is the "advanced" ajax upload
      try:
        filename = request.GET[ 'qqfile' ]
      except KeyError: 
        return HttpResponseBadRequest( "AJAX request not valid" )
    # not an ajax upload, so it was the "basic" iframe version with submission via form
    else:
      is_raw = False
      if len( request.FILES ) == 1:
        # FILES is a dictionary in Django but Ajax Upload gives the uploaded file an
        # ID based on a random number, so it cannot be guessed here in the code.
        # Rather than editing Ajax Upload to pass the ID in the querystring,
        # observer that each upload is a separate request,
        # so FILES should only have one entry.
        # Thus, we can just grab the first (and only) value in the dict.
        upload = request.FILES.values( )[ 0 ]
      else:
        raise Http404( "Bad Upload" )
      filename = upload.name
    
    # save the file
    success = save_upload( upload, filename, is_raw )

    # let Ajax Upload know whether we saved it or not
    import json
    ret_json = { 'success': success, }
    return HttpResponse( json.dumps( ret_json ) )

The first thing you probably want to edit here is the use of filename in either ajax_upload or save_upload. The saving function as it stands assumes filename is a path. In my actual usage, I combine filename with a constant from settings.py that represents the path to where uploads should be saved. So, at the beginning of save_upload you could have something like filename = settings.UPLOAD_STORAGE_DIR + filename where UPLOAD_STORAGE_DIR it set to something like "/data/uploads/". Or, of course, you could skip the constant and hard code your path string, but that's bad right?

And that's it, go have some fun!

I have had many people ask me via comments or email about providing a demo of this system. From a user standpoint it looks/works no different from the demo on Valum's site. As of right now I cannot provide my own demo because my web host does not provide a Django environment. I'm trying to work with them on getting it available though. I will also work on getting my code in as the Django example with the uploader code in my github repo. When either of those happen I will update this post.

Thanks to everyone who commented on the last post, they helped immensely in creating my github repo and in fixing bugs on the post itself. If you find any mistakes here please comment or contact me directly (contact info can be found on my site).

31 comments:

  1. tnx for it, that's brilliant\
    but how about if we wanna upload image for a model
    e.g for Article model.
    if images will upload with ajax, so how we can associate images with that Article model?

    ReplyDelete
  2. @Alir3z4 I would use my code to save the file and then use the following post as a guide for associating it with an ImageField field in your Article model. Basically you want to open the uploaded file as a Django File object, then save to the ImageField with something like my_article.the_image.save( "/path/to/upload/image.png", file_object )

    http://stackoverflow.com/questions/1308386/programmatically-saving-image-to-django-imagefield

    ReplyDelete
  3. Have you ever run into issues with BufferedWriter, specifically
    "with BufferedWriter( FileIO( filename, "wb" ) ) as dest:"
    raising
    "ValueError: invalid mode: wb"? Can't figure this one out

    ReplyDelete
  4. Got some help on Stack Overflow, I found that FileIO( filename, 'b') instead of 'wb' for me on Mac.

    http://stackoverflow.com/questions/5628575/invalid-mode-wb-when-trying-to-upload-file-using-django-and-ajax

    ReplyDelete
  5. I want to use this for a field which allows only one file to be uploaded; do you happen to know if this is possible or should I find another solution?

    ReplyDelete
  6. @adrian In the same place you pass the csrf params, there is a parameter called 'multiple' you can use which, if given false, will allow only one at a time.

    ReplyDelete
  7. How can I see thumbnail image after upload?

    ReplyDelete
  8. Alex, thanks for the great integration. I've built out a class-based view that streams the files to S3, using your views as a base.

    Code's up on github - https://github.com/GoodCloud/django-ajax-uploader

    Thanks for the inspiration and readable code!

    ReplyDelete
  9. hi! good guide!
    I'm trying to make it all working ;-)
    ...in the meantime I found 2 typos:
    Javascript line 2: missing quotes around url;
    in save_upload: in the try block return True is missing

    ReplyDelete
  10. @Babu corrected, thanks for the bug hunting!

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. how to use this in the django admin?

    ReplyDelete
  13. Hi,
    The request.method on the view side is unexpectedly GET for me(instead of post).

    I have followed your instructions to the T. What might I be doing wrong?

    ReplyDelete
  14. @Eli I have never been big on the Django admin so I never bothered with any integration there. What functionality would you like?

    @siddharthsarda I am not sure what the problem is, no one has ever brought that one up. Could be a browser issue? Without being able to replicate it I can't determine if there is a bug in my code.

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete
  16. Hmm, everything's working well, except that it says "Multiple file uploads are disabled" even though it's set to true, just as in your example. Any ideas there? Using Django 1.4.

    ReplyDelete
  17. @shacker I would suspect you have an error somewhere in your javascript but it is hard to tell. For this particular problem your Django version shouldn't matter.

    ReplyDelete
  18. Hi Alex - I haven't altered the Javascript from the example in your distribution, except to change the div ID, the reversed URL, and to change the alert! Here's what I'm using:

    http://dpaste.com/722551/

    Any suggestions welcome - hoping to give a demo of this tomorrow.

    Thanks for the great tutorial by the way.

    ReplyDelete
  19. This comment has been removed by the author.

    ReplyDelete
  20. @shaker I had same problem. I found that this is wrong parameter name in file js/fileuploader.js line #604 - "if( !self.multiple && e.dataTransfer.files.length > 1 )" instead of "if( !self._options.multiple && e.dataTransfer.files.length > 1 )"

    ReplyDelete
  21. Hi Alex, thanks a bunch for putting this together. I did find one typo: in the 'save_upload' python function I think there should be another 'return True' on line 18, so that success is reported when it's uploaded as raw data.

    ReplyDelete
  22. Dear sir, I've been flailing around trying to figure out how to do something like this, and your solution is exactly what I was looking for. +1000 points.

    ReplyDelete
  23. Thank you everyone for posting fixes, I will be incorporating these into the version posted on Github soon.

    @David thanks for taking the time to comment, glad my code was of use

    ReplyDelete
  24. Help! I have a problem: the file is uploaded successfully through ajax, but it during my tests it shows the alert "upload failed!". After uploading a file, the file is uploaded in my webserver, but the alert message that the upload failed is shown. In the chrome developer console, it showed an INTERNAL SERVER ERROR message. It's weird because I already double checked the code for the server-side script. Any help will be greatly appreciated.

    Thanks a lot for this very good stuff!

    ReplyDelete
  25. @pat It is pretty hard to track down since a 500 can be thrown in various places for various reasons. I have never had very sophisticated debugging techniques, but you can try throwing errors, returning other statuses (403 maybe), or print statements if you are using the django server to try to track down the specific line(s) that are causing the 500.

    ReplyDelete
  26. Hey dude, thanks for replying, really appreciate it! Somehow, the problem I mentioned was fixed when I did it all over again using another project. I have questions though:
    1. Can this have a remove attachment functionality? I want to add an ajax-powered link beside the name of the file that was successfully uploaded, that will remove the file from my database.

    2. What will I edit if I want to remove the adding of a list item below the button if the upload was successful? Actually, I only want non-image files to be shown in the list item (then add a link to remove the file as mentioned in Question #1)

    Thank you very much for taking time to read comments on your blog! :)

    ReplyDelete
  27. Any Ideas how to adjust this if I am using Amazon S3 as my backend storage server.
    Thanks a lot in advance

    ReplyDelete
  28. @Salma I've not dealt with Amazon S3 so I have no idea how the interaction would happen. Sorry I can't be of more help!

    ReplyDelete
  29. I have a problem. I get 'success' and 'All complete' alerts but with the file that is uploaded 'Failed' is written on my modal and file is nowhere to be seen.

    I have the div on the modal.

    ReplyDelete
  30. correction i m getting the file copied but Failed is always written

    ReplyDelete