Programming « Senko's Blog

Programming

Quickstart your Django project in 60 seconds

Sane-default boilerplate to avoid repeating yourself each time you start on a new Django project.

Django comes with a lot of batteries included, but it takes some time to set them up. And usually, the initial few steps are always the same. We can refactor these into a boilerplate empty Django project, and use it as a base for new projects, instead of starting from scratch each time.

That’s what I did with DJ Skeletor. It’s an empty, relocatable (ie. uses no hardcoded paths) Django project, set up to my liking and with a few Django apps that I use in virtually all projects (impatient? jump directly to the examples).

Django settings

Default Django settings organisation is rather simplistic – it’s just one file. But as soon as you deploy the project to somewhere else than the computer you’re writing it on, you’re going to have at least two sets of settings: development and production.

There are several ways to handle this in Django. I prefer the following:

Settings module is split into several submodules and lives in settings/ directory.
Settings that are the same for both development and production live in settings.base module.
Development-specific settings are provided by settings/dev.py, and production-specific settings are in settings/prod.py; a symbolic link settings/local.py points to the one that should be used in a specific environment.
The settings module imports settings.base and settings.local

This allows me to have all of the settings code (even per-environment settings) in the git repository. By using the symlink (instead of host name or IP address as some do), I can activate the variant I need without regards to the rest of the environment. This allows me to eg. have several variants of the same project running on the same machine.

Database

A test SQLite database is set up for the development environment. The database file is configured to be test.db in the project root by default.

South

South is an awesome Django app for handling database schema migrations (ie. table/field changes when you modify your models). If you’re using a database with Django, you want to use South as well.

Django Debug Toolbar

Django Debug Toolbar is very handy for inspecting what happens when you request a page from Django. It lists things such as SQL queries executed (including how long did they take and why they were executed), signals, logging, exception, request params, etc. I use it all the time for finding and fixing suboptimal database queries.

The app usually defines a white-list of client IPs for which to be shown. As I’m not on a static IP, I find it more useful to have the toolbar show all the time when in development environment, and never when in production.

Sentry

Sentry helps with exception logging and viewing for your Django project. It can handle multiple Django installs where the logs can be managed from a single place, or it can be used on a per-project basis. The latter is how it’s preconfigured in DJ Skeletor.

Besides logging the exceptions, Sentry can also catch normal logs you create with Python’s logging system. This is also preconfigured in DJ Skeletor.

Example

Enough talk, let’s see some action. First, we’ll create a virtual environment (you do use virtualenv, right? if not, you should) and install prerequisite packages:

    virtualenv --no-site-packages myenv
    cd myenv
    source bin/activate
    pip install django django-debug-toolbar south django-sentry

Next, we’ll clone the boilerplate project:

    git clone https://github.com/senko/dj-skeletor.git myproject
    cd myproject

Let’s activate the development environment and prepare the database:

    cd settings
    ln -s dev.py local.py
    cd ..
    python manage.py syncdb
    python manage.py migrate sentry

All done, let’s run it!

    python manage.py runserver

See? Piece of cake. With boring initial set-up out of our way, we can focus on building an awesome web site or app.

DJ Skeletor is open source and is available on GitHub. Feel free to use it or base your own boilerplate on it – if you do, please share your thoughts in the comments.

Bonus: HTML boilerplate

If you’re a programmer and couldn’t design if your life depended on it, it’s useful to have the user interface boilerplate handy as well.

If you need a clean, well designed (but definitely not unique or artistic) user interface for your HTML app, I heartily recommend Bootstrap, created and open sourced by the fine folks at Twitter.

If you do need to make a proper, unique design, again no need to start from zero: use HTML5 boilerplate which takes care of a myriad little gotchas for you; and there’s a mobile option as well.

Git / Programming — 4 Comments
1
Sep 11

On generic coding style

.. or, much ado about whitespace

Every programming language has at least one standard or preferred code style – most have several, and they’re an endless inspiration for bikeshedding.

I’m about to do just that.

Instead of talking about a specific language or community, I’ll look at some general guidelines that are applicable to every programming language. Of course, there are exceptions, but there has to be a good reason to make the exception.

These are just things I picked along the way and found that they’re useful in general. I’ll try to explain why I think each of the guidelines is important, but YMMV – code style is largely a subjective issue, and I’m not here to preach my way is in any way superior for you. It just works best for me.

Indentation: spaces versus tabs

Spaces and tabs make an ugly mix. In some languages (like Python), mixing them can easily lead to hard-to-find errors. In others, the effect is purely visual, but still ugly.

People have various tab length settings in their editors. Yes, I agree we should all standardize on tab width of 4 (or 8, or 2, or ..), but at this point I don’t think we’ll ever do that. So if you share (as in show or view) code with anyone not following the same style, it’s going to be a mess.

Spaces or tabs? When you’re mainly using tabs, there are still places where a space or two are going to be needed. You’re effectively going to mix them. So don’t, use only spaces.

Okay, that was easy. I also intentionally ignored the fact that some tools require mixing them – make, for example.

Line endings: Unix or Windows?

If you’re only working in and for Windows environment, and you’re not going to need to share code with developers on other operating systems, you can get by by using Windows-style line ending (CR-LF). In fact, it’s probably going to be easier than the alternative.

Otherwise, use Unix line ending (LF) everywhere. It’s the default on both Linux and OS X, and if your (web app) code is going to be served by some cloud server, it’s probably going to be Linux or some other variant of Unix as well.

Whatever you do, don’t ever mix the two. If you have a hybrid team, some people on Windows and some on OSX or Linux, standardize now.

Trailing whitespace

Trailing whitespace on each line can be trouble, because otherwise identical lines can actually be different. This can wreak havoc with your version control, since it’s easy to change them, without changing the actual content of the line, and the version control will hapilly treat them as code changes.

Fortunately, most editors have the option of automatically trimming the white space when saving a file. No reason not to use it.

Line width

This is a tough one. I try to have the content fit in 80 columns. It’s easier to read, as it avoids wrapping (easily mistaken for multiple lines) and horizontal scrolling (downright cumbersome).

This is easier for some languages than for others. I’ve pretty much given up on doing it in HTML. It’s just too verbose for 80 columns to be enough.

So, I try to follow the 80-column rule unless working around it would make the code uglier than if I just let it go beyond col 80. In those cases, I prefer wrapping instead of horizontal scroll.

Terminating newline

A file should end with a single newline. That means, no empty lines after the content, but also not having the last line of the code dangling without an ending newline.

This makes it easier on the editors, nicer when using cat or patch, and makes git not complain. Good enough for me.

File encoding

With everyone supporting it these days (and it being a default on a lot of systems), there’s just no reason not to use UTF-8 encoding. I try to stick to the ASCII part whenever possible, but when I do need it, UTF-8 is the only sane choice.

YMMV

I’m pretty opinionated with regards to these things, as I’m sure you are as well, dear reader. So please do share your preferred settings for the above in the comments (we’ll agree to disagree, thus avoiding any potential flamewars).

Also, if you have an interesting example where some of these MUST be broken (eg. in Makefile), please do share.

Linux / Programming — 8 Comments
18
Aug 11

Auto rsync local changes to remote server

When developing software that’s intended to run on a server, I like to edit the code directly on my laptop, but don’t want to run the development server locally.

Instead, I have a dev setup on a remote server, and copy the local changes as needed, using rsync to copy only the changed bits.

Running rsync is manual work, prone to errors and easy to forget. So I automated it by writing a shell script that runs it whenever I change the directory.

The Script

The script is pretty simple. It uses inotifywait (part of inotify tools on Linux) to detect changes in the current directory and runs a command when that happens. To avoid running multiple commands on a related sequence of events, the script waits a second to see if there are more events coming.

The script itself doesn’t assume which command needs to be run, so it’s useful beyond just triggering rsync. You can find the script on GitHub.

Example usage

When working on a project, I start the watch script with something like:

    onchange.sh rsync -avt --delete . server:/path/

Speeding up rsync+ssh

Since I’m running rsync frequently, I want it to be as quick as possible (even though it’s happening in the background). Since the changes are very small, most of the time spent is in initiating SSH connection to the remote server.

To minimise this, I’m using SSH connection multiplexing. I just ssh into the server, even if I don’t need to do anything there. This keeps the connection open. Subsequent rsyncs over ssh just reuse it.

Portability

Unfortunately, as this uses inotify tools directly, it’s not very portable to non-Linux systems. It’s easy to write the equivalent tool in, say, python using watchdog.

That’s left as an excercise to the reader ;-)

Update: there’s also lsyncd, a full-blown tool for live syncing local with remote folders.

Business / Programming — 1 Comment
13
Aug 11

Developer litmus test

Here’s the situation: you need to hire a new developer. You’ve put up a job posting, you’ve got ton of CVs, and even after you discard the ones obviously not suited to the job, you still have a lot more potential candidates than you’d like to interview in person.

One way to trim this even further is to give them a task and see how well they execute it. Unfortunately, this only tests the programming abilities of the candidate. While important, these are not the only skills they need to have. Also, the assignment can easily be gamed – the candidate can ask (or even pay) someone to solve it for them.

The “patches welcome” test

Open source contributors usually start by making a small tweak or fix in the existing project code, not by developing all new codebase. Similarly, new hires usually first work on the existing code, before being put in charge of a new component of the system.

Can we apply the same principle to candidate assignments? I think we can.

We can make up a standalone project that uses the same languages or framework that the new hire will use – something that can be done in a day. We actually implement it, but leave it full of errors (syntax, logic, or valid but obviously poorly performant code).

The setup

Instead of writing corret code and inventing errors, it’s better to just write the whole thing in one go, never bothering to check and fix any syntax or logic errors (in fact, it’s best if you disable syntax highlighting and other code validation checks that your editor might have).

We put the project on a code hosting site and give the candidates no further instructions than “this needs fixing, documenting and unit tests – patches welcome.” We leave the code wide open, so candidates don’t think we’re abusing the test to have the “real” work done.

Then, we wait, see what they come up with.

Benefits

This tests a lot:

whether the candidate can use git (or hg or your preferred vcs) or, failing that, that they can (and do) learn the very basics
whether they can read and understand others’ code and modify/refactor it without making it into a pasta
that the candidate can actually program and fix the code in question
that they know the frameworks or libraries you need them to know
whether they can follow the code/style conventions
that they write unit tests (if applicable to the assignment)
whether they’ve identified valid but poorly performant parts of the code
can they write documentation
whether they split their work in smaller chunks (commits/patches) or give one big code dump at the end

Conveniently, it also serves as a quick intro for your coding practices for the candidate you do end up hiring.

While this does take a bit more involvement up front (lose a day to write the test), I believe this is far more accurate test across various skills needed for a good developer, than just a simple fizzbuzz.

Python — 1 Comment
9
Jul 11

A CLI runner module for Python

Recently I had to write several Python scripts to be run from command line or cron. They didn’t need elaborate parameter parsing, so I didn’t bother with optparse or argparse modules. I just needed something to get the strings from the command line, interpret the first one as a command, and the rest as the parameters.

I happen to use this pattern for CLI utilities a lot, so at some point I abstracted it in a small module that does it for you automatically: clirunner.

The module exports just one class, CLIRunner, which you need to extend and add methods to it – one method for each command. The runner will take care of exposing them as commands, making sure they’re called with the right number of arguments, and provide help to the user. Class and method docstrings are used for the help info.

CLIRunner does this by using the standard inspect module, which makes it easy to have code poke at itself, look up methods, docstrings and various other interesting introspection bits.

An example

Here’s a simple example:

from clirunner import CLIRunner
class HelloWorld(CLIRunner):
    "A simple Hello World application."
    def hello(self):
        "Print standard greeting."
        print "Hello World!"

if __name__ == '__main__':
    HelloWorld()

gives:

$ python hello.py
HelloWorld - A simple Hello World application.
Usage: hello.py COMMAND [ARGS...]
Commands:
    hello - Print standard greeting.

The code is open source, available on GitHub. Use it, fork it, send pull requests.

Senko's BlogProgramming, free software and the Web