5 Ways to Speed Up Your Django Tests

If you use Test Driven Development or test your code regularly during development, running your test suite can start to become very time consuming. Here’s some tips for speeding up your tests. All of these changes can go in a test_settings.py file, which your can import into your settings.py file only when a test is being run.

  1. Use Sqlite
    I’ve written about this before. When using Sqlite, the Django test runner will create the DB in memory instead of on disk, which is faster than any other database available.
  2. Skip South Migrations
    If you’re using South, migrations will be run during the set up for every test, which can add a lot of time to your tests. Set “SOUTH_TESTS_MIGRATE = False” to use the normal syndb behaviour for tests.
  3. Use an In-Memory Storage Class
    There’s a project called django-inmemorystorage, which stores files in memory instead of on disk. This is not only faster but eliminates the problem of having to clean up files created during testing. Unfortunately, it doesn’t currently support the url() method, so using the “.url” attribute of a FileField or ImageField will throw an error, which is why I created a fork to solve this issue. After installing set “DEFAULT_FILE_STORAGE = ‘inmemorystorage.InMemoryStorage’” in your test settings.
  4. Use Weaker Password Hashing
    Prior to Django 1.4, Django used SHA1 to hash passwords. Starting with 1.4, Django can be configured to use a number of password hashers. The default setting is:

    PASSWORD_HASHERS = (
    'django.contrib.auth.hashers.PBKDF2PasswordHasher',
    'django.contrib.auth.hashers.PBKDF2SHA1PasswordHasher',
    'django.contrib.auth.hashers.BCryptPasswordHasher',
    'django.contrib.auth.hashers.SHA1PasswordHasher',
    'django.contrib.auth.hashers.MD5PasswordHasher',
    'django.contrib.auth.hashers.CryptPasswordHasher',
    )

    Django will encrypt passwords using the first one in this list, PBKDF2PasswordHasher, which was designed to take a really long time to execute in order to be harder to break. This is an important security feature that should be left in place on your live site, but for running a test suite it’s really not needed. You can set it to the following in order to go back to using SHA1:

    PASSWORD_HASHERS = (
    'django.contrib.auth.hashers.SHA1PasswordHasher',
    'django.contrib.auth.hashers.PBKDF2PasswordHasher',
    'django.contrib.auth.hashers.PBKDF2SHA1PasswordHasher',
    'django.contrib.auth.hashers.BCryptPasswordHasher',
    'django.contrib.auth.hashers.MD5PasswordHasher',
    'django.contrib.auth.hashers.CryptPasswordHasher',
    )

    I have one project whose tests took 50 seconds to complete before I made this change, and 7 seconds afterwards.

  5. Mock Network Calls
    Mocking network calls is not only a best practice, but is also a lot faster than making real requests. I recommend using Mock. It gets the job done, it’s widely used, and it’s even being merged into the standard library of Python 3.3. Here’s a condensed example of how to use it:

Please note, aside from using mock objects, these techniques should only be used while testing during development. On a regular basis (such as nightly, or after every commit on a CI server, and/or before pushing your changes to your code repo), you should run your test suite using settings as close to your live site settings as possible (same DB type, same storage, run migrations, etc.) in order to weed out any bugs.

If you have any tips feel free to share!

Returning the Correct HTTP Status Code in TastyPie: 400 Bad Request

Another problem I’ve encountered with TastyPie is that a 500 Internal Server error is thrown when the user sends bad input. One of my clients was using hurl.it to try out an API that I wrote, using hand written JSON. Ocassionally there would be mistakes, such as the use of single quotes instead of double quotes, or the inclusion of a trailing comma, which would cause JSON deserialization to fail. Instead of catching this error and returning a 400 Bad Request response, TatsyPie just lets the error propagate, resulting in a 500 Internal Server Error. Here’s some code I wrote to fix that:

Returning the Correct HTTP Status Code in TastyPie: 401 Unauthorized

In TastyPie, a library for creating REST APIs for Django projects, the recommended way to limit the objects available to a user is through the apply_authorization_limits() function. This method is extremely handy and easy to use, but has a serious flaw; by filtering out objects from the queryset that a user should not access, TastyPie acts as if that object doesn’t exist at all. This causes 2 problems:

  1. When trying to access an object which a user doesn’t have access to, a 404 Not Found status code is returned instead of the more semantic (and less misleading) 401 Unauthorized. Normally you wouldn’t care about misleading users trying to access other users’ data, but this caused a bit of confusion with one of my clients who has another team writing a mobile app using the API I wrote. When trying to access an object while logged in as the wrong user the API tells them the object doesn’t exist, instead of telling them they don’t have access.
  2. Sometimes your get a 500 Internal Server error. If a user tries to update (PUT) an existing resource for another user, instead of returning a 401 status code, TastyPie thinks the object doesn’t exist, tries to insert a new row in the database with an already used primary key, and an IntegrityError is thrown. Even though this only happens during incorrect usage, it still looks bad.

To solve this problem I created a CustomModelResource class with an overridden obj_get() method.

Awesome Web App, Codecademy, Teaches You to Program

I just found out about this cool web app called Codecademy that teaches programming to complete novices. As soon as you visit the site you can start learning to program without having to create an account, and as of now it looks like it’s completely free. If you or anyone you know is interested in learning to program I recommend trying it out.

Twitter Web Intents Javascript Events Not Firing

If you’re trying out Twitter’s Web Intents Javascript Events and it’s not working, be aware that they only work if your HTML page is loaded from an HTTP server. If you open your page locally off your hard drive it won’t work, though you can fire up an Apache instance on your localhost and it’ll work from there.

Source: https://dev.twitter.com/comment/reply/304/491

Update: It looks like this can also happen with the Facebook JavaScript API: http://forum.developers.facebook.net/viewtopic.php?pid=362167#p362167. I guess it’s because they use iFrames, and using them locally can cause security problems.

Saved Web Pages from Firefox Have Scripts Removed

So, I was just in the middle of a programming test for a job application where I had to download an HTML page, edit the JavaScript to perform a desired task, and email it back. I downloaded the page, but the JavaScript had been replaced by the text, “Script removed by snapshot save”. Luckily the removed JS was only one line and I was able to figure out what it was.

Afterwards, I went searching for the cause and found this forum post, which explains that the problem is caused by the Mozilla Archive Format extension, which has a setting that even affects pages not saved as .maff. To fix the problem, go into the MAF extension settings and change “When saving complete web page contents:” from “Take a faithful snapshot of the page” to one of the other options.

Recruiter SPAM

Got a job posting mass emailed to me today, here’s an excerpt:

“Our records show based on your resume, which was imported into our system recently that you are an experienced IT professional whose background may possibly align to the senior level position we have been retained to work on…” [Emphasis added by me.]

Wow, they didn’t even try to pretend that they read my resume or sent me a personal email, pretty ballsy, and of course the job had nothing to do with my skill set. It’s also hard to imagine any quality senior level candidates responding to such a faceless, generic email. To any recruiters who might be reading, this sort of email gets marked as SPAM.

Cookies Not Saved In Internet Explorer for Facebook iframe Apps

I’m making a Facebook game using Django and was having a problem with Internet Explorer.

The Problem: After login the page would render correctly as if the user was logged in, but when navigating to another page it was as if the user wasn’t logged in. Upon investigation I could see that the cookies weren’t being saved.

The Solution: A quick Gogole search brought me to this blog post: IE Blocking iFrame Cookies. It’s an easy to fix P3P issue, and the author describes how to add the necessary header for various languages/frameworks.

Rather than add the header to each Django view or write custom Django middleware, I just added the header to my Apache configuration. The following goes in your VirtualHost Directive:

Header add P3P ‘CP=”IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT”‘

In order to work you have to enable mod_headers (run “sudo a2enmod headers”).

CSS Meta Languages and Compilers

So, I’ve finally decided to add a CSS framework into my workflow, namely Blueprint. I’m not very skilled when it comes to graphic design (though I’m trying to learn) and the CSS frameworks out there give you some nice looking web pages. The only problem is they require you to add extra classes to each HTML element you want to use them with. The whole point of CSS is you can modify it without changing your markup. If I could set an id for each element (#content, #footer) and have each id style definition simply inherit the CSS framework classes I want to use my problem would be solved, but that can only be done using a CSS meta language and compiler.

I was looking at django-css, a project which would integrate CSS compilation into my Django projects, and it mentioned 4 popular languages: HSS, Sass, CleverCSS, and LESS.

Here’s a spreadsheet comparing the features:

Popularity
I think SASS is by far the most popular. I’m secretly rooting for it, since it’s probably in higher demand and would therefore be a more valuable skill to have, though it would be awkward introducing Ruby dependencies in a Python project.

Inheritance
The main reason I want to use a CSS meta language is for the inheritance. When looking at the documentation for CleverCSS I couldn’t see any inheritance implementation, which is a shame because it’s the only one implemented in Python and would’ve been easier to integrate into my Django project.

HASS example:

var nomargin = { margin : 0px; padding : 0px; }

pre {
    $nomargin;
    color : #FF0000;

}

The inherited styles can’t be a normal CSS class, so this can’t be used to inherit classes from a CSS framework.

SASS Mixin example:

@mixin table-base {
  th {
    text-align: center;

    font-weight: bold;
  }
  td, th {padding: 2px}
}

@mixin left($dist) {
  float: left;
  margin-left: $dist;
}

#data {
  @include left(10px);
  @include table-base;
}

The inherited definition needs to be declared as a mixin, and I’d rather not modify any CSS frameworks. There is a project called Compass that provides CSS frameworks rewritten in SASS, but I’d rather not add yet another dependency into my project, especially a Ruby one (nothing against Ruby, it’s just undesirable within the context of a Python project).

SASS Inheritance example:

.error {
  border: 1px #f00;
  background: #fdd;
}
.error.intrusion {
  font-size: 1.3em;
  font-weight: bold;
}

.badError {
  @extend .error;

  border-width: 3px;
}

Just what I want, but it generates more code than needed:

/* CSS */

.error, .badError {
  border: 1px #f00;
  background: #fdd;
}

.error.intrusion,

.badError.intrusion {
  font-size: 1.3em;
  font-weight: bold;
}

.badError {
  border-width: 3px;
}

LESS example:

.bordered {
  border-top: dotted 1px black;

  border-bottom: solid 2px black;
}

#menu a {
  color: #111;

  .bordered;
}
.post a {
  color: red;
  .bordered;
}

Generates:

#menu a {
  color: #111;
  border-top: dotted 1px black;

  border-bottom: solid 2px black;
}
.post a {
  color: red;
  border-top: dotted 1px black;

  border-bottom: solid 2px black;
}

Exactly what I need. The documentation says, “Any CSS class, id or element ruleset can be mixed-in that way.” And since it’s written in JavaScript, you can load your LESS files and compile them on the client side, which should make designing and debugging a lot easier. LESS best suits my needs, so it’s the one I’ll be going with.

Update: It turns out LESS doesn’t eliminate the inherited classes, they’re still present in the output. Even so, I still think it best suits my needs.

Update 2: It looks like django-css isn’t being maintained, so you should probably use django_compressor.