Huy Nguyen

A guide to analyzing Python performance

2012-07-19T00:00:00-07:00

While it’s not always the case that every Python program you write will require a rigorous performance analysis, it is reassuring to know that there are a wide variety of tools in Python’s ecosystem that one can turn to when the time arises.

Analyzing a program’s performance boils down to answering 4 basic questions:

How fast is it running?
Where are the speed bottlenecks?
How much memory is it using?
Where is memory leaking?

Below, we’ll dive into the details of answering these questions using some awesome tools.

Coarse grain timing with time

Let’s begin by using a quick and dirty method of timing our code: the good old unix utility time.

$ time python yourprogram.py

real    0m1.028s
user    0m0.001s
sys     0m0.003s

The meaning between the three output measurements are detailed in this stackoverflow article, but in short

real - refers to the actual elasped time
user - refers to the amount of cpu time spent outside of kernel
sys - refers to the amount of cpu time spent inside kernel specific functions

You can get a sense of how many cpu cycles your program used up regardless of other programs running on the system by adding together the sys and user times.

If the sum of sys and user times is much less than real time, then you can guess that most your program’s performance issues are most likely related to IO waits.

Fine grain timing with a timing context manager

Our next technique involves direct instrumentation of the code to get access to finer grain timing information. Here’s a small snippet I’ve found invaluable for making ad-hoc timing measurements:

timer.py

import time

class Timer(object):
    def __init__(self, verbose=False):
        self.verbose = verbose

    def __enter__(self):
        self.start = time.time()
        return self

    def __exit__(self, *args):
        self.end = time.time()
        self.secs = self.end - self.start
        self.msecs = self.secs * 1000  # millisecs
        if self.verbose:
            print 'elapsed time: %f ms' % self.msecs

In order to use it, wrap blocks of code that you want to time with Python’s with keyword and this Timer context manager. It will take care of starting the timer when your code block begins execution and stopping the timer when your code block ends.

Here’s an example use of the snippet:

from timer import Timer
from redis import Redis
rdb = Redis()

with Timer() as t:
    rdb.lpush("foo", "bar")
print "=> elasped lpush: %s s" % t.secs

with Timer as t:
    rdb.lpop("foo")
print "=> elasped lpop: %s s" % t.secs

I’ll often log the outputs of these timers to a file in order to see how my program’s performance evolves over time.

Line-by-line timing and execution frequency with a profiler

Robert Kern has a nice project called line_profiler which I often use to see how fast and how often each line of code is running in my scripts.

To use it, you’ll need to install the python package via pip:

$ pip install line_profiler

Once installed you’ll have access to a new module called “line_profiler” as well as an executable script “kernprof.py”.

To use this tool, first modify your source code by decorating the function you want to measure with the @profile decorator. Don’t worry, you don’t have to import anyting in order to use this decorator. The kernprof.py script automatically injects it into your script’s runtime during execution.

primes.py

@profile
def primes(n): 
    if n==2:
        return [2]
    elif n<2:
        return []
    s=range(3,n+1,2)
    mroot = n ** 0.5
    half=(n+1)/2-1
    i=0
    m=3
    while m <= mroot:
        if s[i]:
            j=(m*m-3)/2
            s[j]=0
            while j<half:
                s[j]=0
                j+=m
        i=i+1
        m=2*i+3
    return [2]+[x for x in s if x]
primes(100)

Once you’ve gotten your code setup with the @profile decorator, use kernprof.py to run your script.

$ kernprof.py -l -v fib.py

The -l option tells kernprof to inject the @profile decorator into your script’s builtins, and -v tells kernprof to display timing information once you’re script finishes. Here’s one the output should look like for the above script:

Wrote profile results to primes.py.lprof
Timer unit: 1e-06 s

File: primes.py
Function: primes at line 2
Total time: 0.00019 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2                                           @profile
     3                                           def primes(n): 
     4         1            2      2.0      1.1      if n==2:
     5                                                   return [2]
     6         1            1      1.0      0.5      elif n<2:
     7                                                   return []
     8         1            4      4.0      2.1      s=range(3,n+1,2)
     9         1           10     10.0      5.3      mroot = n ** 0.5
    10         1            2      2.0      1.1      half=(n+1)/2-1
    11         1            1      1.0      0.5      i=0
    12         1            1      1.0      0.5      m=3
    13         5            7      1.4      3.7      while m <= mroot:
    14         4            4      1.0      2.1          if s[i]:
    15         3            4      1.3      2.1              j=(m*m-3)/2
    16         3            4      1.3      2.1              s[j]=0
    17        31           31      1.0     16.3              while j<half:
    18        28           28      1.0     14.7                  s[j]=0
    19        28           29      1.0     15.3                  j+=m
    20         4            4      1.0      2.1          i=i+1
    21         4            4      1.0      2.1          m=2*i+3
    22        50           54      1.1     28.4      return [2]+[x for x in s if x]

Look for lines with a high amount of hits or a high time interval. These are the areas where optimizations can yield the greatest improvements.

How much memory does it use?

Now that we have a good grasp on timing our code, let’s move on to figuring out how much memory our programs are using. Fortunately for us, Fabian Pedregosa has implemented a nice memory profiler modeled after Robert Kern’s line_profiler.

First install it via pip:

$ pip install -U memory_profiler
$ pip install psutil

(Installing the psutil package here is recommended because it greatly improves the performance of the memory_profiler).

Like line_profiler, memory_profiler requires that you decorate your function of interest with an @profile decorator like so:

@profile
def primes(n): 
    ...
    ...

To see how much memory your function uses run the following:

$ python -m memory_profiler primes.py

You should see output that looks like this once your program exits:

Filename: primes.py

Line #    Mem usage  Increment   Line Contents
==============================================
     2                           @profile
     3    7.9219 MB  0.0000 MB   def primes(n): 
     4    7.9219 MB  0.0000 MB       if n==2:
     5                                   return [2]
     6    7.9219 MB  0.0000 MB       elif n<2:
     7                                   return []
     8    7.9219 MB  0.0000 MB       s=range(3,n+1,2)
     9    7.9258 MB  0.0039 MB       mroot = n ** 0.5
    10    7.9258 MB  0.0000 MB       half=(n+1)/2-1
    11    7.9258 MB  0.0000 MB       i=0
    12    7.9258 MB  0.0000 MB       m=3
    13    7.9297 MB  0.0039 MB       while m <= mroot:
    14    7.9297 MB  0.0000 MB           if s[i]:
    15    7.9297 MB  0.0000 MB               j=(m*m-3)/2
    16    7.9258 MB -0.0039 MB               s[j]=0
    17    7.9297 MB  0.0039 MB               while j<half:
    18    7.9297 MB  0.0000 MB                   s[j]=0
    19    7.9297 MB  0.0000 MB                   j+=m
    20    7.9297 MB  0.0000 MB           i=i+1
    21    7.9297 MB  0.0000 MB           m=2*i+3
    22    7.9297 MB  0.0000 MB       return [2]+[x for x in s if x]

Where’s the memory leak?

The cPython interpreter uses reference counting as it’s main method of keeping track of memory. This means that every object contains a counter, which is incremented when a reference to the object is stored somewhere, and decremented when a reference to it is deleted. When the counter reaches zero, the cPython interpreter knows that the object is no longer in use so it deletes the object and deallocates the occupied memory.

A memory leak can often occur in your program if references to objects are held even though the object is no longer in use.

The quickest way to find these “memory leaks” is to use an awesome tool called objgraph written by Marius Gedminas. This tool allows you to see the number of objects in memory and also locate all the different places in your code that hold references to these objects.

To get started, first install objgraph:

pip install objgraph

Once you have this tool installed, insert into your code a statement to invoke the debugger:

import pdb; pdb.set_trace()

Which objects are the most common?

At run time, you can inspect the top 20 most prevalent objects in your program by running:

(pdb) import objgraph
(pdb) objgraph.show_most_common_types()

MyBigFatObject             20000
tuple                      16938
function                   4310
dict                       2790
wrapper_descriptor         1181
builtin_function_or_method 934
weakref                    764
list                       634
method_descriptor          507
getset_descriptor          451
type                       439

Which objects have been added or deleted?

We can also see which objects have been added or deleted between two points in time:

(pdb) import objgraph
(pdb) objgraph.show_growth()
.
.
.
(pdb) objgraph.show_growth()   # this only shows objects that has been added or deleted since last show_growth() call

traceback                4        +2
KeyboardInterrupt        1        +1
frame                   24        +1
list                   667        +1
tuple                16969        +1

What is referencing this leaky object?

Continuing down this route, we can also see where references to any given object is being held. Let’s take as an example the simple program below:

x = [1]
y = [x, [x], {"a":x}]
import pdb; pdb.set_trace()

To see what is holding a reference to the variable x, run the objgraph.show_backref() function:

(pdb) import objgraph
(pdb) objgraph.show_backref([x], filename="/tmp/backrefs.png")

The output of that command should be a PNG image stored at /tmp/backrefs.png and it should look something like this:

The box at the bottom with red lettering is our object of interest. We can see that it’s referenced by the symbol x once and by the list y three times. If x is the object causing a memory leak, we can use this method to see why it’s not automatically being deallocated by tracking down all of its references.

So to review, objgraph allows us to:

show the top N objects occupying our python program’s memory
show what objects have been deleted or added over a period of time
show all references to a given object in our script

Effort vs precision

In this post, I’ve shown you how to use several tools to analyze a python program’s performance. Armed with these tools and techniques you should have all the information required to track down most memory leaks as well as identify speed bottlenecks in a Python program.

As with many other topics, running a performance analysis means balancing the tradeoffs between effort and precision. When in doubt, implement the simplest solution that will suit your current needs.

Refrences

Modifying Python's SimpleHTTPServer to accept directory aliases

2012-05-28T00:00:00-07:00

I often use python’s SimpleHTTPServer as an adhoc means for sharing files. As it turns out it’s also a useful tool for dynamically viewing the contents of static site generators like Jekyll (which is what this site is built upon).

The command below is a quick way for one to spawn a webserver that will serve the contents of the current directory.

python -m SimpleHTTPServer

And since python comes pre-installed on most linux distros and on MacOSX, this is a no-nonsense way for serving files.

No dependencies, no installation steps, it just works.

Given this, there are times however, when I wish it had just one extra piece of functionality: the ability to route different url prefixes to different directories.

The script below patches a single function in SimpleHTTPServer’s request handler and makes this possible without adding in any extra dependencies.

server.py

import os
import posixpath
import urllib
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler

# modify this to add additional routes
ROUTES = (
    # [url_prefix ,  directory_path]
    ['/media', '/var/www/media'],
    ['',       '/var/www/site']  # empty string for the 'default' match
) 

class RequestHandler(SimpleHTTPRequestHandler):
    
    def translate_path(self, path):
        """translate path given routes"""

        # set default root to cwd
        root = os.getcwd()
        
        # look up routes and set root directory accordingly
        for pattern, rootdir in ROUTES:
            if path.startswith(pattern):
                # found match!
                path = path[len(pattern):]  # consume path up to pattern len
                root = rootdir
                break
        
        # normalize path and prepend root directory
        path = path.split('?',1)[0]
        path = path.split('#',1)[0]
        path = posixpath.normpath(urllib.unquote(path))
        words = path.split('/')
        words = filter(None, words)
        
        path = root
        for word in words:
            drive, word = os.path.splitdrive(word)
            head, word = os.path.split(word)
            if word in (os.curdir, os.pardir):
                continue
            path = os.path.join(path, word)

        return path

if __name__ == '__main__':
    BaseHTTPServer.test(RequestHandler, BaseHTTPServer.HTTPServer)

While not as convenient as the former solution, you can run this script by simply invoking python server.py

Having great conversations, in spite of being a programmer

2012-03-21T00:00:00-07:00

Listening is the most important skill for having great conversations. I’ve had to learn and relearn this skill so many times through out my career. As a programmer, I’ve come to realize that the mode of thinking required to be good at my job and the mode of thinking needed to have great conversations are completely at odds with one another.

Programming is one of those activities where in order to achieve a high level of productivity you have to shut out the world. You put on blinders and dive into an alternate reality where bit by bit you’re constructing a delicate mental model of the problem at hand.

On difficult projects, the concepts can be quite slippery to grasp, and ideas often need several iterations of refinement. Some of the best programmers I know revisit a concept dozens of times before it’s ready to implement.

To a certain extent, I believe all programmers possess this craving to refactor and refine ideas. And it’s this exact same quality that impedes so many of us when it comes time to have conversations with other people.

Programmers end up missing out a lot on conversations because our minds so readily want to explore and refine the dozens of ideas as they come up. Unfortunately, the tempo and pace of a typical conversation is so rapid that ideas relevant in one moment may be irrelevant in the next. Our tendency to explore and refine leads us down rabbit holes and before you know it, the other person has already moved on to the next subject.

Being explicit about switching modes

The trick to overcoming these bad habbits is to be explicit when you’re switching modes between programming and having conversations with people. You have to mentally recognize the change in tempo and that your mode of thinking needs to adapt.

As a physical trigger, I wiggle my toes during these transitions, and it lets me know that okay … it’s time to shut off my inner voice and actively listen to what the other person is saying.

Once this is done, the rest of mechanics fall into place. By actively listening, I don’t get distracted by minor details and can truly absorb what the other person is trying to convey. With this understanding I can deliver my comments and questions when they are most relevant.

I’ve gotten better at having good conversations over the years, and most of that has come through conscious repeated practice of this simple little trick.

Could testing in Python be simplified?

2011-05-28T00:00:00-07:00

Edited on May 29 : After writing this article, Christian Heimes pointed out to me that the debug method on unttest.TestCase allows you to run tests interactively. The idiot in me didn't do enough research before writing this rant. I'm leaving this post up to provide some context for future visitors/googlers who are looking for a quick way debug their test cases interactively.

One lesson I've learned over many years of observation and programming in Python is that the easier I make writing and running tests for my programs, the faster I can produce bug-free software. That is why I start most projects nowadays with two pre-baked files: main.py and tests.py.

With this setup and tools like Nose, testing code in Python for me is nearly pain free. Recently however, I notice one aspect of the unittest package in Python's standard libary drives me absolutely crazy:

Why is there no simple method to run a single test case from within an interactive python shell?

As developers, this is important because if we are going to make writing tests an integral part of our development workflow, one needs some way to actually run the damn test without dropping out of our Python sessions and breaking out of our mental flow.

Being forced to run tests solely from the commandline also means we can't take advantage of features like ipython's ability to automatically drop into a pdb debugging session when an error occurs. This is extremely useful for when you want to introspect one or two variables to determine why a test failure occurred.

In general, it seems like the unittest library's lack of a simple function to run single test cases in an interactive shell dis-incentivizes programmers from writing tests and detracts from python's whole "rapid iteration" ethos.

After googling around and failing to find any good alternatives, I've finally settled on writing my own utility function to run single test cases:

def runtest(testCase, methodName):
    """ Runs a test case from within interactive shell """

    tc = testCase(methodName)
    getattr(tc, "setUp", lambda:None)()
    try:
        getattr(tc, methodName)()  
    finally:
        getattr(tc, "tearDown", lambda:None)()

I add this function to every single project that I create now, and the qualitative feel of writing tests just feels so much more natural now. I also find myself writing more tests and running them more often with this function in my projects.

Give it a try, and let me know whether this has any effect for you. I'd love to hear your feedback on how it affects your development workflow.

Textmate hack: A keyboard shortcut for opening files in a new window

2011-05-11T00:00:00-07:00

Textmate has no split windows, and you can't open the same file within a project in a new window without resorting to using the mouse. Here's what I mean:

Situations often arise where you have to refrence one file while editing another. Maybe you want to view a header file while editing its implementation code. Or maybe you want a TODO list in one window while working on the rest of your code base.

In these scenarios tabbed editing becomes extremely painful, and the alternative of mousing around to open a new window has always felt like a clunky solution to me. Here's a hack to create a keyboard shortcut for opening files in new windows. Oh what a hack it is:

#!/bin/sh

tof=`echo $TM_FILEPATH`
echo $tof

mate "$tof" .RANDOM_STRING_DOES_NOT_MATTER && osascript <<EOF
tell application "System Events"
    delay 0.2
    tell application "TextMate" to activate
    keystroke "t" using {command down}
    keystroke return
    keystroke "d" using {command down, control down, option down}
end tell
EOF

To use this, create a new bundle command, add this code to it, and associate your keyboard shortcut to the command.

A poorman's Excel using sqlite, apsw, and csv2sql

2011-04-15T00:00:00-07:00

Excel has always been my go-to tool for exploring the abundance of csv-formatted data that I often find on the internet. However, every once in awhile large datasets like these H1B salary figures show up¹, where the shear amount of records exceeds Excel's 65k row limit.

In these scenarios I turn to sqlite and a recently discovered project, apsw for my data analysis-fu.

Importing data into sqlite3

To start, I load up the data using a script I created a while back called csv2sql.py ². Although you can use sqlite to directly load in csv files, I often find the functionality chokes on slightly malformed csv files. So here's my short script that reinvents the wheel a little better using Python:

Usage: csv2sql CSVFILE

Options:
  -h, --help  show this help message and exit
  -o DBNAME   output sqlite3 database file
  -t TABLE    default table name for data import

#!/usr/bin/env python
import sys
import csv
import re
import codecs
import optparse as op
import sqlite3 as db
from itertools import islice

parser = op.OptionParser()
parser.usage = "%prog CSVFILE"
parser.add_option('-o', action="store", dest="dbname",default="data.db", help="output sqlite3 database file")
parser.add_option('-t', action="store", dest="table", default="Records", help="default table name for data import")

def main(fpath, table, dbname):

    with codecs.open(fpath, encoding='utf-8', errors='ignore') as fh:
        rows    = csv.reader(fh)
        fields  = next(rows)
        slugify = lambda s: re.sub(r'\W+',"_", s).upper()
        schema  = "\n\t\t" + ",\n\t\t".join("%s text" % k for k in map(slugify,fields))

        print("Creating table: %s\n" % dbname)
        conn   = db.connect(dbname)
        c      = conn.cursor()

        # create table with schema
        stmt   = "CREATE TABLE %s (%s);" % (table, schema)     
        c.execute(stmt)

        print("Schema")
        print("======")
        print("\n%s\n" % stmt)

        # insert
        print("Adding records to database")
        stmt   = "INSERT INTO %s (%s) VALUES (%s);" 
        stmt   = stmt % (table, ", ".join(fields), ", ".join(['?'] * len(fields)))
        to_add = list(islice(rows, 2000))
        while to_add:
            sys.stdout.write('.')
            c.executemany(stmt, to_add)
            to_add = list(islice(rows, 2000))
            conn.commit()

if __name__ == '__main__':
    opts, args = parser.parse_args()
    main(fpath=args[0], table=opts.table, dbname=opts.dbname)

Exploring the data with apsw

After importing the csv file, I typically run SQL queries against the database in a sqlite shell to explore the data. Apsw is a Python wrapper for sqlite3, which also includes an enhanced shell with features like tab completions and output modes for json and Python tuples. These two things combined make interactive data exploration extremely pleasant.

Once you have apsw installed ³, create a short alias in your .bashrc file so that you can invoke the enhanced sqlite shell from the commandline:

alias sqlite='python -c "import apsw;apsw.main()"'

With this in place, you can invoke the shell by typing sqlite data.db on your terminal prompt. Those familiar with the sqlite shell know that you can have it output the results of a query in several different formats (i.e. column, csv, line etc) . And to set the output mode you simply type .mode <MODENAME> into the prompt before executing your queries.

The two most useful output formats that apsw provides is the "json" and "python" modes. Here's what the following SQL query looks like after setting up "json" and "python" modes respectively.

SELECT job_title, wage_from, avg(wage_from)
  FROM main 
  WHERE job_title like "%SOFTWARE%" COLLATE NOCASE 
  GROUP BY job_title 
  ORDER by wage_from
  LIMIT 10;

As you can see in the above example, this data is readily consumable in any standard python shell. Cut and paste the output into a python interpreter for further analysis, and use matplotlib to further visualize the data.

Mentioned on Hacker News ↩
This script assumes that the csv file's first row contains the field names for the data. This is how it automatically creates the table schema for you ↩
To install apsw, you'll need to make sure that the sqlite development libraries are in your library and include paths. ~~I finally was able to install apsw after using brew install sqlite and setting the libraries variable in setup.py to ["/usr/local/lib"]~~ See Roger's comment below for a way to automatically retrieve all of the dependencies: ↩

So that's how tracing JITs work ...

2011-04-07T00:00:00-07:00

From PyPy's blog: Adding a JIT to your interpreter

When it (the jit compiler) detects a loop of code in the target language that is executed often, the loop is considered "hot" and marked to be traced. The next time that loop is entered, the interpreter gets put in tracing mode where every executed instruction is logged.

When the loop is finished, tracing stops. The trace of the loop is sent to an optimizer, and then to an assembler which outputs machine code. That machine code is then used for subsequent loop iterations.

(The generated machine code) depends on several assumptions about the code. Therefore, the machine code will contain guards, to validate those assumptions. If a guard check fails, the runtime falls back to regular interpreted mode.

This is one of the best explanation of JIT compilers I've found. I finally understand how they work after hearing the term tossed around so much lately.

Sane color scheme for Matplotlib

2011-02-08T00:00:00-08:00

John Hunter, creator of MatPlotlib, originally designed it’s color scheme to be familiar to Matlab users. As it turns out, the color scheme works well for publication material but doesn't work so great for viewing visualizations on the web.

I find the default styling for graphs produced using ggplot2 aesthetically pleasing for this purpose, so I spent some time over the weekend to refine the default colors and settings for my matplotlib installation. The result of this work is embodied in this .matplotlibrc color theme file. If you want graphs that look like the ones below by default, download it and place the file under ~/.matplotlib/matplotlibrc.

Under the hood: An HTTP request with multipart/form-data

2011-02-06T00:00:00-08:00

Check yourself, do you know what an HTTP request looks like coming over the wire? No, not the raw bits or the flowing electrons. Just try to picture the actual HTTP request body as text. Now imagine what it looks like if the HTTP request has an image attached. Here it is. This is what the internet is built on.

/
User-Agent: curl/7.21.2 (x86_64-apple-darwin)
Host: localhost:8080
Accept: */*
Content-Length: 1143
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------83ff53821b7c

------------------------------83ff53821b7c
Content-Disposition: form-data; name="img"; filename="a.png"
Content-Type: application/octet-stream

?PNG

IHD?wS??iCCPICC Profilex?T?kA?6n??Zk?x?"IY?hE?6?bk
Y?<ߡ)??????9Nyx?+=?Y"|@5-?M?S?%?@?H8??qR>?׋??inf???O?????b??N?????~N??>?!?
??V?J?p?8?da?sZHO?Ln?}&???wVQ?y?g????E??0
 ??
   IDAc????????-IEND?B`?
------------------------------83ff53821b7c
Content-Disposition: form-data; name="foo"

bar
------------------------------83ff53821b7c--

You don't always need to know what's under the hood. But sometimes, it's just magical to realize that the thing you're using everyday is nothing more than this. What other protocols do you take for granted yet find completely amazing?

To watch raw HTTP requests yourself

Get reflect.py -- an echo server for HTTP requests
Run this in terminal #1 python reflect.py
Run this in terminal #2 curl -X POST -F foo=bar -F img=@a.png localhost:8080

Django custom form widget for dictionary and tuple key-value pairs

2010-12-20T00:00:00-08:00

Here's a code snippet to ease your pains when asking for user input in the form of key-value pairs of data. I've been using this widget in more and more applications as Redis and other lightweight databases displace some of my traditional SQL storage solutions.

I had to create this widget because none of the builtin django widgets could generate arbitrarily long lists of key-value pairs for the user to modify at runtime. It's especially useful for storing and displaying dictionary data. Here's an example of what it looks like after being rendered:

To use this, set the JsonPairInputs as the default widget on any form field. Here's a simple example below:

examplejsonfield = forms.CharField(label  = "Example JSON Key Value Field", required = False,
                                   widget = JsonPairInputs(val_attrs={'size':35},
                                                           key_attrs={'class':'large'}))

You may also pre-populate the input boxes if you pass a json-encoded list of key-value pairs to the
"initial" argument of the above CharField's form.

widgets.py

import simplejson 
from django.forms import Widget
from django.utils.encoding import force_unicode
from django.utils.safestring import mark_safe
from django.forms.widgets import flatatt

class JsonPairInputs(Widget):
    """A widget that displays JSON Key Value Pairs
    as a list of text input box pairs

    Usage (in forms.py) :
    examplejsonfield = forms.CharField(label  = "Example JSON Key Value Field", required = False,
                                       widget = JsonPairInputs(val_attrs={'size':35},
                                                               key_attrs={'class':'large'}))

    """

    def __init__(self, *args, **kwargs):
        """A widget that displays JSON Key Value Pairs
        as a list of text input box pairs

        kwargs:
        key_attrs -- html attributes applied to the 1st input box pairs
        val_attrs -- html attributes applied to the 2nd input box pairs

        """
        self.key_attrs = {}
        self.val_attrs = {}
        if "key_attrs" in kwargs:
            self.key_attrs = kwargs.pop("key_attrs")
        if "val_attrs" in kwargs:
            self.val_attrs = kwargs.pop("val_attrs")
        Widget.__init__(self, *args, **kwargs)

    def render(self, name, value, attrs=None):
        """Renders this widget into an html string

        args:
        name  (str)  -- name of the field
        value (str)  -- a json string of a two-tuple list automatically passed in by django
        attrs (dict) -- automatically passed in by django (unused in this function)
        """

        if value is None or value.strip() is '': value = '{}'
        twotuple = simplejson.loads(force_unicode(value))

        ret = ''
        if value and len(value) > 0: 
            for k,v in twotuple: 
                ctx = {'key':k,
                       'value':v,
                       'fieldname':name,
                       'key_attrs': flatatt(self.key_attrs),
                       'val_attrs': flatatt(self.val_attrs) }
                ret += '<input type="text" name="json_key[%(fieldname)s]" value="%(key)s" %(key_attrs)s> <input type="text" name="json_value[%(fieldname)s]" value="%(value)s" %(val_attrs)s><br />' % ctx
        return mark_safe(ret)

    def value_from_datadict(self, data, files, name):
        """
        Returns the simplejson representation of the key-value pairs
        sent in the POST parameters

        args:
        data  (dict)  -- request.POST or request.GET parameters
        files (list)  -- request.FILES
        name  (str)   -- the name of the field associated with this widget

        """
        if data.has_key('json_key[%s]' % name) and data.has_key('json_value[%s]' % name): 
            keys     = data.getlist("json_key[%s]" % name) 
            values   = data.getlist("json_value[%s]" % name) 
            twotuple = []
            for key, value in zip(keys, values): 
                if len(key) > 0:
                    twotuple += [(key,value)] 
            jsontext = simplejson.dumps(twotuple) 
        return jsontext

Gathering Leads using MTurk

2010-06-13T00:00:00-07:00

I want to share a technique with you that I have been using to streamline my customer development process. It's experimental, but so far it's saving me a lot of time and producing good results.

At the core, my technique involves using MTurk to collect targeted leads for some business ideas that have been brewing in my head. By using MTurk I was able to gather contact information from a huge list of websites in a very little amount of time. Sure, I could've written a scraper to find emails on most of these sites, but after weighing the cost/benefit ratio, I figured it just wasn't worth my time.

The Task

To give you some context, I had a list of around 2000 urls pointing to local chamber of commerce sites scattered across the United States. Quite a few of my business ideas involve these organizations, so I wanted to validate/gauge interest by sending out targeted email inquiries to each organization's leader.

For those of you unfamiliar with MTurk, it is a crowd-sourcing service from Amazon that allows you to submit small, well-defined tasks to thousands of people around the world. You basically construct a HTML form with a special templating syntax and then submit a CSV file that will populate into that form when Amazon displays it to their crowd of workers.

You can see a sample of the task that I created below.

Aside: I generated the above form using Shpaml which I highly recommend for writing short-hand HTML. It significantly cut down on my typing and development time.

The Results

I ran a test trial using 100 urls. The task was set up so that every url would be sent to at least 3 workers. In total, 300 turkers worked on my task. On Sunday morning I had submitted the task to Amazon Mturk. When I came back 4 hours later, it was already completed! Although there were a few spammers, the overall quality was great for what I was trying to achieve. Check out a small sample of my mturk results.

Out of the batch of 300, fourty of the answers came from spammers. I know that 13 percent sounds like a lot! However, since I had 3 workers work on each URL, I quickly filtered out the spam by accepting only the answers where two or more mturkers intersected.

For those of you trying to capture the local marketplace, or even if you're just interested in getting this data, send me a quick email! I will be running through the rest of my batch in the upcoming days.

Daily Signal (May 10, 2010)

2010-05-10T00:00:00-07:00

Pandas - R's dataframes in Python
- Pandas is a python package for working with timeseries data. I've been looking a long time for an equivalent to R's dataframe functionality within python and this is it. What makes these dataframe structures so special is their ability to quickly slice and dice a table of data.
node.js, jsdom and jQuery
- JSdom allows you emulate the entire document object model of a webpage without running a browser. I just found out today that node.js now runs jsdom and jquery seamlessly. This is nothing short of a webscraping revolution.
CryptoJS
- A cryptographic library for Javascript. Perfect use case: Interactive API documentation. Got an api service? Want developers to adopt quicker? Allow them to submit signed API calls within your webpages.
The MarketPlace Play
- A detailed analysis of the marketplace business model.

Exceptions are your Friends

2010-02-09T00:00:00-08:00

Robust code cries often and loudly as soon as something is not right. It does not cower away in corners of obscurity hoping that no one will notice, until one day, shit hits the fan.

Any serious python code contains proper use of exceptions, errors, and asserts. In fact, I would argue that their presence defines the difference between one-off throwaway scripts and robust library code. Imagine a python interpreter without these facilities. What would the scenario look like? When you write to a non-existent or write-protected file, instead of the sys module alerting you to the fact, you'd get .. nothing. Not a single sign that your data is gone forever. Exceptions are you friends. They're your canary in the coal mine. They let you know that something is wrong at the point of inception rather than pretending that everything is fine until two months later when your client pops a vein because their files were not backed-up. Use exceptions to annotate your code. Use them to communicate to your future colleagues that what they're trying to do is seriously wrong. Use them to put yourself in check before you try to modify that class attribute. Exceptions keep you sane.

Handling Exceptions

Some would argue that robust code has no exceptions in it. They would be happy to throw a giant all-inclusive try ... except statement around every piece of code that raises an exception. Don't do it, please. That's like ignoring a crying kid without knowing why they are crying in the first place. Here's a simple rule of thumb for dealing with exceptions -- I guess you can call it a best-practice. Catch specific exceptions and errors only if you know what to do with them. Otherwise, let some other piece of code higher in the call stack with enough context deal with it. Robust code cries often and loudly as soon as something is not right. It does not cower away in corners of obscurity hoping that no one will notice, until one day, shit hits the fan. At the same time, when it knows how to deal with an exception, it'll do so. But it will never suppress an exception when it has no idea. Exceptions and Exception Handlers. These are the two pillars of robust, high-quality, and fault-tolerant code.

Don’t Hash Your Secrets, Here’s why in Python

2010-02-01T00:00:00-08:00

Ben Adida suggests that you don't hash your secrets.

That means that if you know SHA1(secret || message), then you can compute SHA1(secret || message || ANYTHING), which is a valid signature for message || ANYTHING. So to break this system, you just need to see one signature.

Not being a cryptography expert, I was blown away by his article. At the core of his post is the idea that given a hash digest of a message, one could compute the hash of message + appended_message without even knowing the original message.

I had to see this for myself. Was it that easy to extend an MD5 or SHA1 hash? Below, you'll find working python code and an explanation for spoofing signatures signed with the MD5 algroithm.

Implementation

To generate a hash from a message, algorithms like MD5 and SHA1 iterate through the message block by block. For each block, the algorithm runs a transformation function where the input is a seed state and a message block . The output of this transformation is then fed back as the seed state for the transformation of the next message block (see the above diagram).

After the hashing function has digested the entire message, it then appends some padding and runs the transformation function one more time. The final state of this transformation becomes the digest.

from hashlib import md5
signature = md5("secret" + "hello world").digest()

print(repr(signature))
"O'Q\xa8\xb8\x9d\x81%\xd7\x13'\xe0\xfb_2\xde"

In the code above, the signature represents the state output of the final transformation function.

AHA! We now have a strategy to extend the hash. If we can seed the transformation function with the state(AKA signature) of the original message, we can essentially extend the hash without even knowing the original message.

There is one problem however. I mentioned before that the MD5 algorithm adds a piece of padding to the original message before it gives us the hash. That means whenever we see a signature it's really the hash of the message + padding. Fortunately, the padding is only dependent upon the length of the original message. With that in mind, we can easily generate both the new signature and padding. Here's some pseudocode

state = decode(signature)
padding = calculate_padding(original_message_len)
new_signature = transform(state, "appended message")

# This should be True
new_signature == md5(original_message + padding + "appended message").digest()

Now here is the real code.

def spoof_digest(originalDigest, originalLen, spoofMessage=""):
    # first decode digest back into state tuples
    state = Decode(originalDigest, 16)

    # generate a seed md5 object
    spoof = md5()

    # seed the count variable for calculation of index, padLen, and bits
    spoof.count += originalLen << 3

    # calculate some variables to generate the original padding
    index = int((spoof.count >> 3) & 0x3f)
    padLen = (56 - index) if index < 56 else (120 - index)
    bits = Encode((spoof.count & 0xffffffffL, spoof.count>>32), 8)

    # construct the original padding
    padding = PADDING[:padLen]

    # augment the count with the new padding and trailing bits
    spoof.count += len(padding) << 3
    spoof.count += len(bits) << 3
    spoof.state = state

    # run an update
    spoof.update(spoofMessage)

    # We now have a digest of the original secret + message + some_padding
    return (spoof.digest(), padding + bits)

The code has a dependency on a pure-python implementation of the md5 algorithm that I've packaged it together with the source code. If you want to try it out, download the file and run this test function (also included in the file):

def test_spoofing():
    originalMsg = "secret" + "my message"
    appendedMsg = "my message extension"

    # This is the signature that a legitimate user sends
    # over the wire in clear text. 
    originalSignature = md5(originalMsg).digest()

    # This is how an attacker would spoof the signature where,
    # the message ==  originalMsg + padbits + appendedMsg .
    # Notice that this method implies that the attacker
    # knows the original length of the "secret" ... 
    # Most apis such as Flickr assign secrets that are of
    # uniform length for all of their api users.
    spoofSignature, padbits = spoof_digest(originalSignature, len(originalMsg), appendedMsg)

    # This is how a legitimate user would construct the
    # a signature when message == originalMsg + padbits + appendedMsg
    testSignature = md5(originalMsg + padbits + appendedMsg).digest()

    # make sure the spoof signature and the test signature match.
    # if, this passes, we've successfully constructed a spoofed message
    # of the form: secret + orginal_message + padding + appended_message
    # without actually knowing the secret.
    assert testSignature == spoofSignature

Information in this blog is meant for educational purposes only!

Quick Bash Tip : Directory Bookmarks

2009-09-10T00:00:00-07:00

EDIT 2010-07-01 : I've packaged up a shell script to allow you to save and jump to commonly used directories. It's called bashmarks and it has tab completion functionality built-in. Learn more about bashmarks here.

Entirely new and improved version

Do not use the stuff below

Before I wrote this script, It felt like I spent half of my time in terminal cd-ing around to various directories. If you're like me, placing this snippet into your .bashrc file will save you tons of time each and every single day:

# Bash Directory Bookmarks
alias m1='alias g1="cd `pwd`"'
alias m2='alias g2="cd `pwd`"'
alias m3='alias g3="cd `pwd`"'
alias m4='alias g4="cd `pwd`"'
alias m5='alias g5="cd `pwd`"'
alias m6='alias g6="cd `pwd`"'
alias m7='alias g7="cd `pwd`"'
alias m8='alias g8="cd `pwd`"'
alias m9='alias g9="cd `pwd`"'
alias mdump='alias|grep -e "alias g[0-9]"|grep -v "alias m" > ~/.bookmarks'
alias lma='alias | grep -e "alias g[0-9]"|grep -v "alias m"|sed "s/alias //"'
touch ~/.bookmarks
source ~/.bookmarks

Directory Bookmark Usage

With this in place, your bash shell will have the ability to set and retrieve directory bookmarks. Let's say you're in a folder that you visit a hundreds of times per day. Run one of the "m" (a.k.a mark) commands inside the directory to create a bookmark. Here's an example:

# This will create a bookmark for the /var/www directory
user@host[/var/www/] : m1

Now whenever you want to cd into that directory, you can run the corresponding "g" (a.k.a goto mark) command.

# This will cd into /var/www
user@host[/etc/apache2] : g1

In other words the m1 command will set the g1 bookmark, the m2 command will set the g2 bookmark, and so on ... If you don't want to keep track of these bookmarks in your head, you'll be glad to hear that the "lma" (a.k.a "list marks ") command can show you all of your current bookmarks like so:

user@host[/usr/local/] : lma
g1='cd /var/www/'
g2='cd /etc/'

Persisting the Bookmarks

If you want to preserve your bookmarks for the next time you log in, execute the mdump command which will store the bookmarks into a file called .bookmarks under your HOME directory. Keep in mind that if you do not run this command your bookmarks will be forgotten once you log out of the shell.

How I Located a Camera in your Back Yard

2009-08-29T00:00:00-07:00

I found a webcam in your neighborhood. As I type, I see your dog easing out a steady stream of its steady-stream onto your neighbor's freshly lacquered patio. Don't believe me? see the results for yourself:

(Unsecure) Webcams Around the World

How?

It first started with this link, And then this, And finally this.

These three simple google searches reveal security cameras around the world that are ironically left unsecure for the entire internet to see. While I used Google for illustrative purposes above, Yahoo works pretty much the same when it comes to these particular searches. More importantly, there is a nice python interface to Yahoo called pYsearch, which means that you can use it to programatically harvest urls to thousands of these cameras. Below, I use this module to locate the first 40 webcam urls on a Yahoo Search Page:

from yahoo.search import web

# Do the search
query_string = 'inurl:"viewerframe?mode=motion"'
s = web.WebSearch("YahooDemo", query=query_string, results=40, start=0)
result_list =  s.parse_results().results

# Extract the urls
webcam_urls = [ result['Url'] for result in result_list ]

Geolocating the Cameras

By itself, this is already cool, but to make things a little more interesting, I wanted to identify the exotic locations these cameras were spying on day-in and day-out. There happens to be a wonderful web service that does geolocation on IP addresses, and they have an open API available at ipinfodb.

In order to use it, we must first translate the webcam URLs into IP addresses if it's not done for us already. Below is a utility function I wrote to accomplish this.

Extract IP Address from URL

def url2ip(url):
    import socket
    from urlparse import urlparse
    basename = urlparse(url)[1].split(':')[0]
    return socket.gethostbyname(basename)

# obtain a list of ip addresses
ip_list = [ url2ip(url) for url in webcam_urls ]

After extracting the IP address from all of these urls, you're now ready to geolocate the IP addresses using ipinfodb, The function below, given the IP address, will return the longitude, latitude and various other pieces of geographical information associated with the IP address.

Geolocate IP addresses

def ip2geo(ip_addr):
    from urllib import urlopen
    from xml.dom import minidom

    # setup url and request
    url =  u"http://ipinfodb.com/ip_query.php?ip=%s" % ip_addr
    response = urlopen(url)

    # Convert response into a dictionary with key:value pairs
    results = {}
    doc = minidom.parse(response)
    for node in doc.firstChild.childNodes:
        if node.firstChild: 
            results[node.nodeName] = node.firstChild.nodeValue
    return results

Example output

In [2]: ip2geo("74.125.45.100")

Out[2]: 
{u'City': u'Mountain View',
 u'CountryCode': u'US',
 u'CountryName': u'United States',
 u'Dstoffset': u'-7.0',
 u'Gmtoffset': u'-8.0',
 u'Ip': u'74.125.45.100',
 u'Latitude': u'37.4192',
 u'Longitude': u'-122.057',
 u'RegionCode': u'06',
 u'RegionName': u'California',
 u'Status': u'OK',
 u'ZipPostalCode': u'94043'}

Plotting the Locations on a Google Map

Google has a fairly simple mechanism that allows developers to create custom google maps. You first load their map widget onto your webpage using some boiler-plate javascript/html:

Google Maps Basic Setup

These two lines -

var geoXml = new GGeoXml("http://code.google.com/apis/kml/documentation/KML_Samples.kml" );
map.addOverlay(geoXml);

- in the boilerplate code will then load an external KML file, which is where you can put all of the anotations to the map such as placemarkers, shape boundaries and etc. Note that since you will be generating your own KML file for pinpointing the camera locations, you must point the boiler plate code to reflect this change.

At this point we have a list of longitude/latitude coordinates obtained from geolocating IP addresses associated with webcam URLs. To plot these camera locations on a Google map, we will generate a KML file using a small utility that I wrote:

pygmap - A KML Generator

It essentially gives you a pythonic interface for generating KML files like these:

Sample KML - A placemark over Mountain View,CA

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2">
  <Placemark>
    <name>Simple placemark</name>
    <description>
        A longer description for the placemark. It can include <b>HTML</b>
    </description>
    <Point>
      <coordinates>39,39,0</coordinates>
    </Point>
  </Placemark>
</kml>

All we have to do now is to loop over our list of longitudes and latitudes, and plot them onto the Google Map using KML. You can see this in action from the code below:

from pygmap import GMap
m = GMap()

for ipaddr in ip_list:

    # Geolocate ipaddress
    geo = ip2geo(ipaddr)
    lon = float( geo['Longitude'] )
    lat = float( geo['Latitude'] )
    m.add_placemark(lon=lon, lat=lat, title="Camera", desc="this is a <b>Camera</b>")

# Output the result KML
print m.renderKML()

So there you have it, a KML file that has all of the camera locations plotted. Just make sure to point the boilerplate google-map html page to your own generated KML file and you should see results like the map at the beginning of this post:

Webcams Around the World

Source Code

pygmap - A KML Placemark Generator
Google Maps HTML Boilerplate

Python logging from multiple processes

2009-08-13T00:00:00-07:00

Had a rough day today. I just wanted a log. Not just any log mind you, but one that could handle writes from multiple processes running at the same time. A naive me would have put python’s basic log handler within each process and then watch as each of the processes crash and burn because of conflicting disk write access.

But I’ve learned from my past lessons - now I use the python SocketHandler for my logging needs. Here’s a basic snippet below to get you started.

# ============== Socket Logger =============== #
import logging
import logging.handlers # you NEED this line
logger = logging.getLogger("%s_%s" % (os.getpid(), sys.argv[1]) )
logger.setLevel(logging.DEBUG)
socketHandler = logging.handlers.SocketHandler('localhost',
                    logging.handlers.DEFAULT_TCP_LOGGING_PORT)
logger.addHandler(socketHandler)
# ============== Socket Logger =============== #

In a nutshell, it gets a logger instance, sets the logging level to DEBUG, and then attaches a SocketHandler to itself. This means that whenever any piece of your code calls logger.debug(“this is my log message”) , it will send that message through the pipes to your server sitting on the other end which will handle writing your log messages to disk. On most machines the DEFAULT_TCP_LOGGING_PORT will be 9020 and the server will be sitting on localhost:9020 .

The Log Server

I was surprised that python didn’t already have a canonical implementation of a logging server to interface “SocketHandler” available in the standard library. Luckily there’s a pretty neat project that does just that: python-loggingserver. As a bonus it comes with a web interface to view your logs. On most systems, you can access the website by going to http://localhost:9021/ once you’ve started the server.

It requires the twisted networking library so if you haven’t had that installed do a

sudo easy_install twisted

Then svn checkout the project:

svn checkout http://python-loggingserver.googlecode.com/svn/trunk/ logserver

Start the server:

cd logserver
twistd --pidfile=loggingserver.pid --logfile=logginserver.log --python=loggingserver.py

Now you’re ready to log. You can now use the code snippet posted at the beginning of this article or just use the prepackaged testing script to send messages to the server:

python loggingtest.py this_is_one_process

View the results at http://localhost:9021

Happy logging!

Save Your Hands

2009-07-21T00:00:00-07:00

Here’s a brief tip: Rebind your ctrl key to capslock.

Save yourself from the pain. The contorted positions we developers strain our hands into will eventually break them - Emacs & Textmate users, you know what I’m talking about. Do yourself a favor and rebind your control key to capslock. You can do this under MacOSX by going to System Preferences > Keyboard & Mouse > Keyboard > Modifier Keys. Change the settings to look like the following and you’re set.

Franchising: Running Multiple Sites from one Django Codebase

2009-07-14T00:00:00-07:00

While surveying the modularity and reusability of code written with Django, I’ve observed that projects usually travel down two divergent paths of evolution. Either they start small and grow into behemoths as developers add feature-after-feature, or they stop growing after a certain point and are packaged into reusable components. Today I propose a third route that weaves between these two extremes and allows you to reuse your existing code by running multiple websites from a single codebase. In essence, I want you to feel like you're developing one site, when in reality you're making multiple. For lack of a better term, I’ll call this process Franchising your Django project.

Why franchise your project?

When my organization began licensing our research-tracking software to multiple clients, each of them wanted to make minor adjustments to various aspects of our software suite. Our initial response to this situation was to branch the original code into several different versions for each site. In this way, when a client wanted minor changes, we would make the modification to their respective code branch. This approach worked fine as long as we weren’t adding any new features to our software. The problem, however, was that we we were. As more clients began licensing our software and as we continued to make improvements to it, the process of syncing those changes between all the different code branches became unmanageable. Despite heroic efforts, our svn-fu was not up to par. Franchising was our answer to this tangled mess.

Defining “Franchise”

The goal was to develop a system capable of sharing common components between multiple sites from a single code-base while keeping it flexible enough for us to make minor changes to each specific site. The critical question that we had to answer was, “What did we need to keep in common between all sites, and what were the things we wanted to change from site to site?” The table below is our attempt at redefining the above question in terms of components common to a Django project.

Django Components in a Franchised Project
	Globally-Shared	Site-Specific
database		x
models	x
urls	x	x
views	x	x
settings	x	x
templates	x	x

From this table, three important pieces of information stand out. One, most Django components - the settings, views, urls, and templates - will have both a globally shared aspect and a site-specific aspect. Two, all sites, despite their differences in templates and views, will have a globally shared model schema. Why? because this allows us to develop our Django project as if we were running only one site. Finally, each site will have its own unique database since each site will have its own unique data. With this in mind, a typical directory layout of a “franchised” Django project might look something like this: Franchised Django Project - Directory Layout
In the top level directory sits all of the globally shared settings, applications, and urls. Under that, a “site_overloads” folder will contain all of the site specific components such as databases, urls, and templates.

Implementing a Franchised Django Site

Now the trickiest part about franchising your django project is getting the globally shared components and the site-specific components working seamlessly with each other. Inherent within this problem is the challenge of figuring out where to route users once they reach one of your sites. Going along with our research-tracker example, imagine that a user has typed “http://research.nasa.gov/” into their browser. If it was possible to dynamically configure Django for a specific site’s settings before responding, we could essentially run many sites from a single codebase by switching between the different configurations. Luckily for us, Apache and most web-servers allow you to dynamically set environment variables depending on the url of the incoming request. Here’s how you do it within apache’s /etc/apache2/httpd.conf file /etc/apache2/httpd.conf

<VirtualHost *:80>
        SetHandler python-program
        PythonHandler django.core.handlers.modpython
        SetEnv DJANGO_SETTINGS_MODULE settings
        SetEnv OVERLOAD_SITE nasa
        PythonPath "['/var/www/data/research_tracker'] + sys.path"
        ServerName research.nasa.com
</VirtualHost>

Using the SetEnv directive, we set a variable called “OVERLOAD_SITE” to “nasa”. Now, anytime someone requests a page from “http://research.nasa.com” Apache will automatically set the variable before handing off the request to Django. By appending the following snippet to our top-level settings.py file, we can use the environment variable to dynamically load the desired site-specific settings. globally-shared settings.py

import os
OVERLOAD_SITE = os.environ.get('OVERLOAD_SITE')
OVERLOAD_SITE_MODULE ="site_overloads" + "." + OVERLOAD_SITE
exec "from %s.settings import *" % (OVERLOAD_SITE_MODULE)

The os.environ.get() function in the above code extracts the value of our desired variable, “OVERLOAD_SITE”, from the operating environment. Using this value, it determines the correct import path to the site-specific settings. Finally, with the exec() function, it imports the site specific settings. To reiterate our general approach: 1. Use Apache to pass off requests to Django, but before doing so, set an environment variable dependent upon the request url. 2. Read this environment variable from within the project’s globally shared settings.py and determine the desired site. 3. From within the globally shared settings.py, import the desired site-specific settings module.

Configuring the site-specific settings to override global settings

Django Components in a Franchised Project
	Globally-Shared	Site-Specific
database		x
models	x
urls	x	x
views	x	x
settings	x	x
templates	x	x

Earlier, we defined the boundaries between globally shared components and site-specific components in a franchised Django project(see above table). With our dynamic configuration setup in place, we can now force site-specific components such as the database, urls, views, and templates to override the default global components. The key to doing so is defining our intentions in the site-specific settings.py. From our NASA research-tracker example, that file would be site_overloads/nasa/settings.py

Setting The Site-Specific Database

By adding these few lines, we can specify to Django the exact database to use for our NASA site. site_overloads/nasa/settings.py

# Overload Default database
DATABASE_NAME = "site_overloads/nasa/",'database.sqlite3'

Overriding Global Templates with Site-Specific Templates To make sure that templates in the site_overloads directory override the globally shared templates, we have to modify both the globally-shared and site-specific settings.py files like so: site_overloads/nasa/settings.py

# Overload Default TEMPLATE_DIRS
SITE_TEMPLATE_DIRS = [
    "site_overloads/nasa/templates",
]

globally-shared settings.py

# Prepend a list of site-specific template directories to the TEMPLATE_DIRS
if SITE_TEMPLATE_DIRS:
    TEMPLATE_DIRS = SITE_TEMPLATE_DIRS + TEMPLATE_DIRS

As a result of these settings, when rendering templates, files in the site_overloads/nasa/templates folder will always be chosen over files in the globally shared templates folder, even if they share the same name.

Adding Site-Specific URLs

In the globally-shared u{urls.py} add these few lines of code to dynamically import the correct site-specific urls: urls.py

# import site specific urls
from django.conf import settings
if hasattr(settings, "OVERLOAD_SITE_MODULE"):
   exec "from %s import urls as site_urls" % (settings.OVERLOAD_SITE_MODULE)
   urlpatterns = site_urls + urlpatterns

With this in place you can now add url patterns as normal under your site-specific urls.py.

Testing the setup

After making all of those adjustments, you’ll most likely want to try out your new setup without having to startup Apache. A good way to test our solution is to manually set the environment variable before running “manage.py runserver” or “manage.py shell”. Type the following on the commandline to run a shell using the site-specific setup. Once inside the Django shell we can poke around at the various site-specific settings and verify that they are what we expect:

OVERLOAD_SITE=nasa; manage.py shell
> from django.conf import settings
> print( settings.OVERLOAD_SITE )
nasa

Summary

In this article, I outlined how to franchise your Django project. As our main approach, we used Apache’s ability to set environment variables to dynamically configure Django before responding to any requests. Because of this, we could dynamically route users to site-specific views, urls, templates, and databases depending on the incoming request url. Franchising allows you to run many Django sites from a single code base. It’s ideal for those times when you want to reuse code that you have written for one site on another project. There is no need to maintain several branches of slightly different code, and as a result, improvements made on one site will simultaneously apply to all sites.

The CRM114 Discrimnator - Your own personal secretary on Mac OSX

2009-03-14T00:00:00-07:00

Imagine your boss comes in one day and says to you, "We have over 100,000 web pages on our site. Of that figure, 10,000 are from spammers. I need you to go through our list of websites and figure out which ones are spam and which are genuine." How do you accomplish this task without going crazy? Wouldn't it be great if your computer just told you whether a webpage was spam or not? Well, it can. Just give it some initial training and you'll have your own digital secretary in no time. This is all possible through the CRM114 discriminator, which is a machine-learning tool to help you classify data according to predetermined samples. We can use it in our case by first, feeding it documents that are known to be "spam"; then feeding it documents that are known to be "genuine". In these two steps, we are "training" the program to recognize the difference between spam and genuine webpages. Finally, for any unknown document, we'll run it through CRM114's "classify" function, which will guess the probability that the document belongs to either the "spam" or "genuine" group based on past training data.

Trying it out

Take a look at some sample code below. It uses Sam Dean's wrapper library, which provides an easy-to-use Python interface to the CRM114 Discriminator .

import crm
c = crm.Classifier("/Users/iamthecheese/Desktop/crm_test_data", ["genuine", "spam"])
c.learn("genuine", "did you see that jean claude van dam movie?")
c.learn("spam", "Jean claude van dam uses viagra, you should too, here's how...")
c.classify("I went to see that movie about the dam today")

If you type that into the Python interactive command prompt and all goes well, you should see the last command return to you:

('genuine', 0.65529999999999999)

Which basically means that based on the set of training data given to CRM114, the phrase "I went to see that movie about the dam today" has a 65% chance of being genuine. Pretty cool huh? Applying this to our spam problem, just find 20 pages that you know are spam and 20 pages you know are genuine; train the CRM114 with this set of data, and unleash it on the rest of the your 999,960 pages. It'll save you a lot of time and you can use your "personal classification secretary" for bunches of other problems in the future as well.

Installing the CRM114 Discriminator on Mac OSX

So now that you're hooked, lets get to installing this program on your Mac OSX. Unfortunately, there is no current macport for the CRM114 Discriminator, so you'll have to do some digging through Makefiles to get everything working. Here's how to build and install the program from the source.

First off, install a dependent regex library called Tre using macports
```
sudo port install tre
```

Get the source code for CRM114

cd ~/Desktop/some_folder
wget http://crm114.sourceforge.net/src/

Modfiy the "Makefile" under the src directory by replacing the following line:

prefix?=/usr     should become -->    prefix?=/opt/local

commenting out the following line:

LDFLAGS += -static -static-libgcc

and uncommenting the following lines:

CFLAGS += -I/opt/local/include -I${HOME}/include
LDFLAGS += -L/opt/local/lib -L${HOME}/lib
LIBS += -lintl -liconv

Now save the Makefile and run the make and make install commands in the src directory
```
make && make install
```
Congratulations, now you've got the CRM114 Discriminator installed on your computer! If it's done correctly, you should be able to run the following command in terminal to get the current version
```
crm -v
```
Finally to use the above sample code, go download Sam Dean's Python CRM114 wrapper library and put it in a place where you can import it from python. ( The site's login/password is "guest").

This piece of software uses many cool machine-learning classification techniques which are beyond my ability to explain here. If you're interested, you can read more about the algorithms below:

A Processing.js example ( Tears in Darkness )

2009-03-05T00:00:00-08:00

Click in the canvas
The processing.js visualization code

Tear[] tears = new Tears[];                   // Create new array to keep track of all the tears
int num_tears = 0;                            // Keeps track of the current number of tears

// This function sets up the canvas elment
void setup() {
  size(500, 300);                             // size( width, height ) - sets the canvas size
  background(0);                              // background ( lightness ) or background ( red, green, blue, alpha ) - sets background color
}

// This function is automatically called by processing every frame
void draw() {                                                   
  fill(0, 6);                                 // fill( lightness, alpha ) or fill( red, green, blue, alpha) -  sets fill colors of any shapes drawn hereafter 
  rect(0, 0, width, height);                  // rect ( upper_leftX, upper_leftY, lower_rightX, lower_rightY ) - draws a rectangle, 
  noStroke();                                 //  removes border on shapes drawn after this point

  for( int i=0; i < num_tears; i++){
     if ( tears[i] == null) {
        // Do nothing if current object == null
     }
     else if (tears[i].update() == false){
        // Update the current tear, if return false than set the current object to null
        tears[i] = null;
     }
     else{
        // When tear is not null, and update() wasn't false then draw() it
        tears[i].draw();
     }
  }

  if(mousePressed){
     // Add a new tear to the tears array at the current mouse location
     tears[num_tears] = new Tear(mouseX + window.scrollX,mouseY + window.scrollY);
     num_tears +=1;
  }

}

// This class object represents one tear. A global variable called "tears" stores an array of these individual tears
class Tear{
   int x,y, alpha;
   float size;
   
   // This function is called when a new tear is created from the "new Tear( x, y)" command. You can pass it the x &amp; y position of the new tear
   Tear(int xin, int yin){
      x = xin;                                    // sets the x coordinate of the tear
      y = yin;                                    // sets the y coordinate of the tear      
      alpha = 255;                                // sets the initial alpha (AKA opacity) of the tear         
      size = 1;                                   // sets original tear radius to be 1px
   }
   
   // This function is called to update the tear's attributes
   boolean update(){
      y += 1;                            // moves the tear down 1px
      alpha -= 5;                        // decrease tear's opacity by 5
      size += .5;                        // increase the tear's size by .5px
      if( y > height || alpha < 0){      // if tear's current height or opacity is out of the visible  range return false
         return false;
      }
      return true;                       // return true if everything went fine
   }
   
   // draw this tear with its current properties
   void draw(){
      fill(0,60, 100+random(0,100), alpha);        // fill ( red, green, blue, alpha )  - sets fill to these attributes, values between 0 and 255
      ellipse(x,y, size,size);                     // draws a circle
   }

}

Processing.js, makes coding graphics for the HTML <canvas> element a dream come true. Casey Reas and Ben Fry created the original library/programming environment to manipulate graphics using Java and has built up a vibrant community with their work. The 2nd revolution however, comes with John Resig's Javascript rendition of those same programming libraries. Because of his port, you can now do things with open technologies such as javascript & html that were once only possible with Flash. I've been exploring with it and this is what I've learned so far.

How was the visualization above created?

Excuse the emo title for the graphic above, but I was going for something simple so that I could quickly learn the system. Animating a tear quickly came to mind as an easy beginner's project. So to begin, let us start with an understanding of what a primitive animation is: A series of pictures rapidly displayed one after another. With Processing.js this notion is realized with the two functions setup() and draw(). You define setup() with the knowledge in mind that Processing.js will automatically call it at the beginning of your visualization to draw the first "picture" of the scene. Following this. it will repeatedly call the draw() function several times per second to generate all of the subsequent "pictures" of your scene.

The snippet below sets up your first "picture" to be 600 pixels wide and 300 pixels high.

    // This function sets up the canvas elment
    void setup() {
      size(600, 300);                              // size( width, height ) - sets the canvas size
      background(0);                               // background ( lightness ) or background ( red, green, blue, alpha ) - sets background color
    }

Creating a tear

In my world, a tear basically consists of a circle that grows as it falls. It gets to a certain size and then it stops and fades away. That is a tear. To represent that, I create an class to store its current position, size, and various other attributes:

   // This class object represents one tear. A global variable called "tears" stores an array of these individual tears
    class Tear{
       int x,y, alpha;
       float size;
       
       // This function is called when a new tear is created from the "new Tear( x, y)" command. You can pass it the x & y position of the new tear
       Tear(int xin, int yin){
          x = xin;                               // sets the x coordinate of the tear
          y = yin;                               // sets the y coordinate of the tear      
          alpha = 255;                           // sets the initial alpha (AKA opacity) of the tear         
          size = 1;                              // sets original tear radius to be 1px
       }
       
       // This function is called to update the tear's attributes
       boolean update(){
          y += 1;                                               // moves the tear down 1px
          alpha -= 5;                                           // decrease tear's opacity by 5
          size += .5;                                           // increase the tear's size by .5px
          if( y > height || alpha < 0){                         // if tear's current height or opacity is out of the visible  range return false
             return false;
          }
          return true;                                          // return true if everything went fine
       }
       
       // draw this tear with its current properties
       void draw(){
          fill(0,60, 100+random(0,100), alpha);        // fill ( red, green, blue, alpha )  - sets fill to these attributes, values between 0 and 255
          ellipse(x,y, size,size);                     // draws a circle
       }

    }

The two major things to notice are 1) the Tear.update() function which changes the tear's properties according to predefined rules and 2) the Tear.draw() function which reads in the current tear's property and does the actual drawing of the tear.

Dealing with many tears

Since our scene will have many tears at once, I'll set up an array of tears to keep track of them all

    Tear[] tears = new Tears[];                    // Create new array to keep track of all the tears

Running the draw() loop

Now that you've got the basic data structures down, we can stick them into the draw() loop to run them. The code below basically loops through all current tears, and updates and draws them.

// This function is automatically called by processing every frame
    void draw() {                                                   
      fill(0, 6);                                  // fill( lightness, alpha ) or fill( red, green, blue, alpha) -  sets fill colors of any shapes drawn hereafter 
      rect(0, 0, width, height);                   // rect ( upper_leftX, upper_leftY, lower_rightX, lower_rightY ) - draws a rectangle, 
      noStroke();                                  //  removes border on shapes drawn after this point

      for( int i=0; i < num_tears; i++){
         if ( tears[i] == null) {
            // Do nothing if current object == null
         }
         else if (tears[i].update() == false){
            // Update the current tear, if return false than set the current object to null
            tears[i] = null;
         }
         else{
            // When tear is not null, and update() wasn't false then draw() it
            tears[i].draw();
         }
      }

mousePressed interactivity

finally I create a little interactivity by listening for the mousePressed event. If someone clicks on the canvas, it will generate another tear object at that mouse location. You'll notice I've augmented the mouseX and mouseY variables to also include how much the user has scrolled the window. The mouseX and mouseY positions are relative to the original position of the canvas element and does not reflect changes to its positions when scrolling.

      if(mousePressed){
         // Add a new tear to the tears array at the current mouse location
         tears[num_tears] = new Tear(mouseX + window.scrollX ,mouseY+window.scrollY);
         num_tears +=1;
      }

    }

How did you get that nice fading effect?

The fading effect of the tears were created by painting a semi-transparent rectangle over the canvas each frame. With each frame these rectangle are painted and eventually cover up the tears.

 void draw() {                                                   
      fill(0, 6);                                  // fill( lightness, alpha ) or fill( red, green, blue, alpha) -  sets fill colors of any shapes drawn hereafter 
      rect(0, 0, width, height);                   // rect ( upper_leftX, upper_leftY, lower_rightX, lower_rightY ) - draws a rectangle,

Summary

In summary, this animation was built by setting up a Tear class which held attributes like the tear's position. The Tear class had functions to update() its position, and to draw() it. A Tear was generated every time someone clicked on the canvas screen and was stored in an array of Tears. Every time processing.js calls draw(), a loop is executed to go through the list of existing tears to update() their properties, and then to draw() them. Overall, this is fun little system that I'll be using in the future to generate some cool infographics

Stock photos in the public domain

2009-03-04T00:00:00-08:00

Objective: To find a decent sized collection of stock photos that are in the public domain or have expired copyrights. Results: EveryStockPhoto Search Engine - Nice interface, giant collection of public domain stock photos Public Domain Photos .com - well organized collection of public domain photos. Wikipedia Public Domain Photos - contains a wide collection of public domain photos. National Fish & Wildlife Service - contains lots of nature photos HubbleSite - space photos from NASA Gimp-Savvy - aggregation of photos from many government public domain photos

Writing Simple, Effective Documentation

2009-02-06T00:00:00-08:00

Documentation is not hard though there is a temptation to make it so. In this post, we’ll go over the details of generating documentation that can be exported to a beautifully typeset PDF and HTML from one source file. In case you're interested, here are the results: Markdown, PDF, XHTML. Because we'll want to produce the final documents in multiple formats, whatever we write will have to be formated in a simple “meta” syntax that can eventually be converted. For a brief moment along my personal documentation journey, I was led astray by xml, docbook, toolchains, and a dizzying myriad of other meta format options.

Salvation By MultiMarkdown

Luckily for me, salvation came and it arrived in the form of MultiMarkdown, a simple syntax to markup your text without the unnecessary bloat of xml. With this and a combination of other tools, you write your documentation once and export it to the multitude of supported formats. The diagram below provides an overview of the process for taking your text and transforming it into PDF and HTML. As you can see, I did end up using a small bit of XML technology (specifically XSLTs) to generate a table of contents from my MultiMarkdown XHTML output.

To get started, all you need is something to convert your MultiMarkdown documents into the desired format. For this task you can either download pandoc, or the conversion scripts from the creator of MultiMarkown’s site. Although pandoc looks like a promising tool, I personally had trouble installing it on my computer because of issues with macports & ghc.

Typing the Documentation

One of MultiMarkdown’s best feature is its simple-to-learn syntax. It’s so intuitive, you don’t feel like you’re learning anything new at all. Below are a few commands to help you get started. For further help in TextMate, use the Control+H key combo to pull up the built in cheatsheet.

# This is an H1 tag
## This is an H2 tag
### This is an H3 tag
... And so on
[This is a Link](http:\\www.google.com)
![This is an Image](my_image.jpg)
CSS: style.css

The last command, “CSS: style.css” tells the MultiMarkdown converter to link up style.css to the converted html file. I like to include it at the very top of any markdown document so that I can later stylize it with my own custom CSS.

Converting To Stylized HTML

Converting your MultiMarkdown to html is a simple process:

Run multimarkdown2XHTML.pl provided in the “bin” directory of the MultiMarkdown distribution to transform your text.
```
multimarkdown2XHTML.pl file.markdown > file.html
```
Add a table of contents using xsltproc and xhtml-toc-h2.xslt
```
 xsltproc xhtml-toc-h2.xslt file.html > file_with_toc.html
```
Stylize your final HTML with some nice CSS. Here’s the CSS that I’ve used for my own documentation. It’s a modification on the oh-so-beautiful FreeBSD documentation theme.

Converting to a Beautifully Typeset PDF

Converting to PDF basically follows the same path, just with some different tools.

Run multimarkdown2latex.pl provided in the “bin” directory of the MultiMarkdown distribution to transform your text into LaTex.
```
multimarkdown2latex.pl file.markdown > file.tex
```
Add some LaTex formatting options if needed. For example, I added the following options to prevent indentation at the beginnings of paragraphs.
```
\setlength{\parindent}{0em}
```
Use pdflatex to turn the output from the previous step into a PDF.
```
pdflatex file.tex
```

If you’re lucky enough to have TextMate, all of these conversion scripts mentioned above are prepackaged in the “MarkDown” bundle for you.

Final Thoughts

So there you have it. With MultiMarkdown, you write one document and transform it into PDF and HTML with some basic tools. Overall, the format is concise, easily readable in raw format, and easy to maintain since it’s just plain-ol text. It is versatile, and has a mature set of tools to work with it. I’ve come to love it just as the creator of TextMate, and many others have. For small to medium sized projects, it’s the perfect documentation tool. If you’re wondering what the final results look like, I wrote this article in markdown. Here is a sample of the Markdown, PDF, and XHTML that I generated.

The cloud is the new subprime mortgage

2009-01-23T00:00:00-08:00

Dear Developers, Please do not repeat the same mistakes that people in the housing market have made... In looking at the current financial crisis, one can't help but draw similarities between the housing market and the tech sector. The housing crisis occurred, in a simplified view, because financial institutions piled layers upon layers of complex contractual obligations on top of a faulty asset: loans to people who couldn't pay them. While there were many players in the game, each relied on the middleman before them to guarantee that the loans wouldn't go bad. They diced, repackaged, and resold these loan contracts to unsuspecting buyers and to themselves while under the illusion that these assets were safe. In doing so, everyone believed they had much more money than they actually possessed. So when one weak link broke, their ever entangling and fragile web of dependencies came tumbling down. Software in an unimaginative leap, is not so far from a contractual obligation, and data is the Internet's currency. With programming APIs to pull data from one service to another abound, developers willingly building mashable apps on cloud servers such as Google AppEngine and AWS, and with more people depending on these services than ever, its hard not to see that we are heading down the same path that finance took, a path toward the information infrastructure crash. Although only recently have the cracks begun to show with the announcement of the end of Google Notebooks and Jaiku, Pownce closing its doors, and Twitter reducing its availability to API developers, these seemingly innocuous events send a clear signal that these virtual streams of data and services can not be relied upon. Similar to the Subprime crisis, when these services close, they bring down with them a slew of other companies that relied upon them. When they go down, they carry with them the thousand of man hours of free labor that users put into updating profiles, posting pictures, and comments on each other's pages. The web can only get so social, and maybe now is the time to consider a small dose of self-reliance. If not, people will be asking one day how we ever led ourselves to believe that companies like Google or Amazon were too big to fail.

How to use chmod

2009-01-10T00:00:00-08:00

Chmod is a command used to change file permissions. Given the right permissions, you can restrict or grant access to a file for specific groups or users. Here's the basic format: "chmod 755 myfile.txt" A file can have the following permissions corresponding to these numbers: Read = 4 Write = 2 Execute = 1 In order to combine permissions, you just have to add the numbers together (ie, READ & WRITE permission is a 6). A file has to give these permissions to three groups of people. For example, each digit in the "755" of the chmod command above represents one of these groups: Left-most Digit = Owner Middle Digit = The file's group Right-most Digit = All other users so in the command above, we gave the owner read, write, and execute permissions; the group of the file got read and execute permissions; and all other users got execute permissions. Hope that helps.

Huy Nguyen

A guide to analyzing Python performance

Coarse grain timing with time

Fine grain timing with a timing context manager

Line-by-line timing and execution frequency with a profiler

How much memory does it use?

Where’s the memory leak?

Which objects are the most common?

Which objects have been added or deleted?

What is referencing this leaky object?

Effort vs precision

Refrences

Modifying Python's SimpleHTTPServer to accept directory aliases

Having great conversations, in spite of being a programmer

Being explicit about switching modes

Could testing in Python be simplified?

Textmate hack: A keyboard shortcut for opening files in a new window

A poorman's Excel using sqlite, apsw, and csv2sql

Importing data into sqlite3

Exploring the data with apsw

So that's how tracing JITs work ...

Sane color scheme for Matplotlib

Under the hood: An HTTP request with multipart/form-data

Django custom form widget for dictionary and tuple key-value pairs

Gathering Leads using MTurk

The Task

The Results

Daily Signal (May 10, 2010)

Exceptions are your Friends

Handling Exceptions

See Also

Don’t Hash Your Secrets, Here’s why in Python

Implementation

Quick Bash Tip : Directory Bookmarks

Entirely new and improved version

Do not use the stuff below

Directory Bookmark Usage

Persisting the Bookmarks

How I Located a Camera in your Back Yard

How?

Geolocating the Cameras

Plotting the Locations on a Google Map

Source Code

Python logging from multiple processes

The Log Server

Save Your Hands

Franchising: Running Multiple Sites from one Django Codebase

Why franchise your project?

Defining “Franchise”

Implementing a Franchised Django Site

Configuring the site-specific settings to override global settings

Setting The Site-Specific Database

Adding Site-Specific URLs

Testing the setup

Summary

The CRM114 Discrimnator - Your own personal secretary on Mac OSX

Trying it out

Installing the CRM114 Discriminator on Mac OSX

A Processing.js example ( Tears in Darkness )

How was the visualization above created?

Creating a tear

Dealing with many tears

Running the draw() loop

mousePressed interactivity

How did you get that nice fading effect?

Summary

Stock photos in the public domain

Writing Simple, Effective Documentation

Salvation By MultiMarkdown

Typing the Documentation

Converting To Stylized HTML

Converting to a Beautifully Typeset PDF

Final Thoughts

See Also

The cloud is the new subprime mortgage

How to use chmod