Leaving Corporate

I just finished debuging a confusing issue on socialbrowse.com and wanted to make a quick note about working out of the django trunk.  I’ve been happily working from the django-trunk for over a year now and ignorantly updating whenever I felt the fancy or I found a new feature to utilize.

Django is under a great deal of development by its very active community and awesome new features are added all the time.  There hasn’t had an official “release” since March 2007 and there have been so many improvements that it is a huge disadvantage to not work from the trunk. A few people criticize the slow release process but most are content with the the historically very stable trunk.

The issue I had, however, was with dealing with backwards incompatibility.  I had never run into an issue where my newly updated trunk would not run the code that worked on an old version.  I was so confident in the trunk in fact that it didn’t occur to me for a full hour that the issue may have been with the Django update I had done the day before and not with the large chunk of my own code I had just loaded.It was really a brain dead, but easy to make, mistake.

Too confident in Django I updated mindlessly on the live server before I tested on the development server.  Luckily it wasn’t a major issue.  For about 2 hours while I was cursing and testing, our users couldn’t change their profile pictures.  After realizing it was nothing I had done the #django IRC was quick to point out the new file upload changes.

Moral of the story:  Watch out for backward incompatibilities and make sure your development and live servers are on the same revision.

A friend is asking how to install HAproxy because I’d just done it three weeks ago, so I’m publishing my install notes, most of which I stole from another friend.

HAproxy is a pretty sweet tool but currently has little written about it so installation and configuration takes reading a large text manual. Here though is a basic setup that should work for most cases.

mkdir /opt/haproxy
cd /opt/haproxy

#Download latest version from http://haproxy.1wt.eu/download/1.3/bin/ e.g.
wget http://haproxy.1wt.eu/download/1.3/bin/haproxy-1.3.14.2-pcre-40kses-splice-linux-i586.notstripped.gz -O haproxy.gz

gunzip haproxy.gz
chmod 700 haproxy

Now you have HA proxy installed but need a config file. Make a new file called /etc/haproxy.cfg and add the following contents.

global
log 127.0.0.1 daemon debug
maxconn 4096

pidfile /var/run/haproxy.pid
daemon
defaults
log global
mode http

option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen webfarm <HAproxy host ip>:80
mode http
cookie MYSITECOOKIE insert nocache
balance roundrobin
server app1 <server1 ip>:8080 cookie app1inst1 check
server app2 <server2 ip>:8080 cookie app2inst2 check
server app3 <server3 ip>:8080 cookie app3inst3 check

This config file sets up session based balancing; meaning HAproxy will inject its own cookie into each person who visits the load balancer. Then every requests the person with that cookie makes from that point on will get directed to the same server! This is most important for apps with content heavily based around user information. Without it its really tough to make use of local caching.

Make sure you change all of the parameters inside <> brackets. You can add or remove servers by editing the “server” commands. I think you can see the trend there :). Also you’ll want to change the MYSITECOOKIE, which is the name of the cookie that HAproxy will inject and the variables app* and app*inst* are made up. You can call them whatever you want.

You should also notice that in this setup HAproxy is listening to port 80 ( <HAproxy host ip>:80 ) and that its redirecting to port 8080 on the re-direct servers (<server1 ip>:8080 ). This is because often one of the re-direct servers is usually the same as the HAproxy host. Also its not a good idea to have random IP’s that aren’t directly linked to your domain to be hosting your app on port 80.

So, next we have to change your app instance to listen to port 8080. I’ve only done this with Apache2 and will only go through those steps. Its very simple and is probably as simple with other servers. You need to edit your ports.conf (often found in /etc/apache2/ports.conf ) and add, or change the following line at the top.

#ports.conf
Listen 8080

Save it, restart Apache2, and that’s it.

Lastly here are the directions for running the proxy. You could bundle these in a script but I use them so rarely I didn’t. There is one command for kicking off the proxy, another for verifying a new configuration file if you make an update (for when you add or remove a server) and finally a third that will dynamically reload your new configuration.

First time start:
/opt/haproxy/haproxy -f /etc/haproxy.cfg -p /var/run/haproxy.pid

Verify a valid configuration:
/opt/haproxy/haproxy -f /etc/haproxy.cfg -c

Dynamically reload configuration:
/bin/sh -c “/opt/haproxy/haproxy -f /etc/haproxy.cfg -p /var/run/haproxy.pid -st $(</var/run/haproxy.pid)”

I’m no expert. Like I said I got most of this from my friend Justin (Thanks Justin!). But it was a pain sifting through the manual and I thought others might benefit from these notes. For more information visit the HAproxy docs.

Note: Interestingly this post by Justin.tv promoting Nginx over Pound just made it to the top of Hacker News as I was finishing. I want to add my recommendation of HAproxy. I also tried Nginx but ended up going with HAproxy. There is certainly more articles on using Nginx but it is more difficult to set up session based balancing.

I just got around to reading the Newsweek article on YC. In it Paul is quoted referring to the Zenters “These guys have written 40,000 lines of code in three months!”

The Zenters were just a couple of guys with some work experience similar to Zack and I. But they must have had some serious skill since they were apparently kicking Google’s butt at the Powerpoint game, or at least were offering some pretty novel aspects, because they were acquired just 6 months after they started YC.

Lines of code is far from and end-all indicator of startup success. Especially when ignoring the little things like how much code was re-written/discarded, which languages/frameworks the code is in, comments, spacing between lines, the amazing roundness of the numbers 40,000 and 3 months, etc…. But there seemed to be a high correlation with the Zenters, and its something at least so I decided to see how we stack up.

If your files are all in one directory you can use a simple “cat * | wc -l”. We have a pretty big maze of directories however so I wrote this quick python script to walk the directory structure and find out for me. I wrote it so it’d be relatively easy for others to use. There are two variables at the top of the file where you can include directory names to ignore and characters in files that will exclude them from being counted. This way it won’t count the dirs with code we didn’t write, compiled files, svn repositories, images, copied files that emacs leaves behind, etc.

The results: 20,010 lines of code in the 53 days we’ve been working. Yes, I included the code in this script to break 20,000. Assuming we keep up our pace for the next 38 days (which might be unlikely due to demo day and investors) we’ll be at 34,511 lines for the 3 months. Crap! Just short!

Also, assuming we keep up that pace, I’ll be dead. A fact that Zack is disturbingly unconcerned about.

But kudos to the Zenters. You guys rock.

You can test yourselves to see if you can keep up with the Zenters. Download the script into the root dir of your source files. Open it up and edit the excluded_dirs and exclude_files_containing variables for your needs. Then run it: python line_cnt.py

A few weeks ago we needed to set up a quick and simple server to Alpha launch 8aweek to our friends. I installed Django with Apache2 mod_python on a box in my friend’s dorm room at Stanford. We went to the Y Combinator dinner pumped to show everyone our hard work and get loads of feedback. Unfortunately, something or someone bumped the dorm room server and our plans for the big Alpha release were spoiled.

We decided that, even for our Alpha, we needed something a little more … reliable, but something not as expensive or time consuming as our planned deployment for launch. A fellow Django-using YCer recommended SliceHost. I checked them out and they’re fantastic. Its only $20 a month (3 month minimum) for a 256mb slice with your choice of several operating systems. The best part is, I just entered my credit card info, and 10 seconds later I was emailed an IP, and root password. We were back on track for Alpha!

I love Ubuntu/Debian over any other distro because of the simplicity of the apt-get install. The popularity of Ubuntu also creates a ton of very specific documentation and support. To streamline the already easy Django setup I followed Jeff Baier’s step by step “Installing Django on an Ubuntu Linux Server.”

I had to change two things, as I was installing a deployment server and not a development one. Don’t include “MaxRequestsPerChild 1″ in your httpd.conf, and instead declare your server name at the top.

Servername yoururl.com

Next log into manage.slicehost.com and select DNS. Here you set up the slicehost Dynamic Name Server to point to your registered domain. Put the url of your site in there. It will tell you to point your Domain Registrar to

ns1.slicehost.net.
ns2.slicehost.net.
ns3.slicehost.net.

This is a confusing part — you’re not done configuring the DNS after giving them your site domain. You have to log a few more “records”. To do so, follow the instructions here.

Now log into the domain registrar and follow their instructions on forwarding to those DNS servers.

The whole process, including writing this article, took me about two hours. Super simple, cheap, and a lot more stable than a computer in a dorm room.