Hugh Perkins's tech blog

Permissions on dreamhost are a pain:
- too restrictive, and apache doesn’t have access
- not restrictive enough, and apache will refuse to run them in cgi
- not restrictive enough, and other people can read one’s passwords and so on

I did some house-keeping this morning, and decided to write some scripts to make my life simpler:

• setwebperms.sh Applies restrictive permissions on a web directory
• setprivateperms.sh Applies permissions on a directory so only your own account has read or write access
• setgroupreadperms.sh Anyone in the file’s linux group can read the files, but not write, and other people can’t even read them
• setotherreadperms.sh As setgroupreadperms.sh, but for everyone. As setwebperms.sh, but allows directory listings

These scripts can be found at http://hughperkins.com/download/dreamhostperms.tar.bz2.

Use them at your own risk, your mileage may vary. The tricky thing with perms scripts is, well there are several:
- we don’t want to change the permissions on files that are already correct, otherwise we affect other scripts that detect changes to our web directory by looking at ctime.
- perms on scripts should be something like 700, but on static content, like html and gifs, should be something like 644
- perms on directories should be either 700 (user only), 711 (traversal but no listing), or 755 (read only, full listing and travers

Solutions used:

• To avoid modifying files that are already correct, we use ‘find ! -perm’.
• We run the find command separately for each file suffix, and filetype, so we can specify precisely the appropraite permissions

You can check inside each script to see how files are classified as dynamic or static, or simply left alone (if not classified).

Config files such as LocalSettings.php or wp-config.php are both counted as dynamic scirpt files, so get 700 permissions when using setwebperms.sh, which keeps them secure from other linux users on the shared hosting.

I ran setwebperms.sh on my entire website this morning, and everything seems to work ok still, so that is cool!

One last point, setwebperms.sh prevents directory listings, which means that directory listings by apache when it doesnt find an apropriate index file will be forbidden. You can get round that by downloading:

http://hughperkins.com/download/directoryindexer.tar.bz2 and adding the index.cgi file inside into any directory you want to be indexed by apache. It is simply a python script that prints an ‘a’ link for each file in the directory which doesnt start with ‘.’ and isn’t index.cgi itself.

Edit: looks like if you go to the dreamhost panel, select ‘users’ and click on a user, then you can select ‘increased security’, which will prevent anyone, even another of your accounts, from being able to read anything from that account, even if they know the exact path, and independent of the permissions on that file. It changes the group and permissions on the home directory itself, right at the top, and that prevents traversal to any other files and directories. Apache is not affected and runs fine.

Installed openid to a mediawiki wiki.

Turned out to be *fairly* painless, but not, exactly, painless, but maybe that’s because I was using an uncommon functionality? Which is: $wgOpenIDConsumerDenyByDefault = true, which means that only a predefined whitelist of people can log in.

The problem for me was that the whitelist was a list of regexps, rather than the, more intuitive to me, what I was expecting, a simple list of strings. I feel it would be more user-friendly if it would handle a list of strings anyway, converting them into the appropriate regexp automatically, ie adding the /s at the end.

Anyway, it turns out that the correct syntax for the whitelist is something like:

$wgOpenIDConsumerAllow = array(“/^someuser\.myopenid\.com$/”);

Erk…

I felt that on the whole the mediawiki experience felt rather less slick than the wordpress admin experience. WordPress admin experience I feel is *awesome*, very slick. I feel an average user without much time can handle wordpress. Mediawiki… I did it… but it took an amount of perseverance that I would prefer not to have to do just to turn on openid…

Downloaded php-openid and copied the examples/consumer directory to http://hughperkins.com/openid_testconsumer. This domain has been converted to use the custom php executable, with gmp enabled, as documented in building php on dreamhost. To get this working, I simply copied in addition the ‘Auth’ directory from php-openid into the openid_testconsumer directory.

Using virtualenv on dreamhost works rather well. Then the question is: how to point one’s python pages at the virtualenv?

One could search and replace #!/usr/bin/python by one’s virtualenv path.

A potentially easier way is to:
- create a directory in the webdirectory called cgi-bin
- create an executable bash script in cgi-bin called ‘python2.4.cgi’
- ‘python2.4.cgi’ is a bash script something like:

#!/bin/bash
/home/yourusernamehere/local/bin/python2.4 $PATH_TRANSLATED

- in the root of your web directory, or anywhere where you want to switch to your virtual environment python, add a file .htaccess with the following contents:

AddHandler python24 py
Action python24 /cgi-bin/python2.4.cgi

Summary:

• Download buildphp.tar.bz2
• Untar, by running: ‘tar -xjf buildphp.tar.bz2′
• Run ‘bash buildphp.sh’ to build php
• Run ‘bash installphptodomain.sh’ to install the customized php to a specific domain/website

Note that this will change the entire website to use the newly built php executable, so you may want to test on a test/staging/dev domain first.

Detail:

Using certain php openid libraries on dreamhost implies using a custom php build, in order to include support for gmp, which is a much faster maths library than that currently included in dreamhost by default, I think.

Excellent post on building php on dreamhost here.

I’ve updated the script somewhat with:
- newer versions for some components. Some of the old versions don’t exist.
- added in gmp library, needed by some php openid libraries
- I removed the cleanup on sourcedir and distdir, since we have pots of disk space
- added in code so that you can rerun the script multiple times, and it wont rerun the download and the extract (but it will rerun the compilation…)

Edit: build finished, and ran, BUT big gotcha, held me up for a while. There are actually two php executables built: php, and php-cgi. php is for CLI usage, whilst php-cgi is for cgi usage. php-cgi adds in the appropriate content-type headers and so on for cgi usage.

The script referenced above seems to copy php rather than php-cgi, which doesn’t seem to work for me. Copying php-cgi into cgi-bin/php.cgi worked ok for me.

Edit2: right, that seems to work, so here are scripts that can be used to install a custom php, including gmp, on dreamhost. Use at your own risk, your mileage may vary.

First, to build php, save this file as buildphp.sh, then run it, ie ‘bash buildphp.sh’.

Once that is built, hopefully to completion, then download this script as installphptodomain.sh. Run it by doing ‘bash installphptodomain.sh’, and give the path of your top level website directory.

Note that this will change the entire website to use the newly built php executable, so you may want to test on a test/staging/dev domain first.

Created a project to run ai bot games on headless grid servers, which is here:

SpringGrid

There is a sister project to get ai bots to fight against each other on different maps, mods, settings, and create automatic ladders, which is here:

AILadder

Also, we need a way of running Spring RTS headlessly, which we do using:

headless spring

I took some time to write a library for Python to make it behave somewhat like Erlang for processes/threading, and communications.

Python has a global interpreter lock (GIL), so each python process basically runs single-threaded, in the presence of multiple processor cores.

Three possible solutions:

• don’t use python
• modify python
• use heavyweight processes as threads

I chose the latter, since it is an easy option, and has a bunch of other advantages.

The scenario I’m targeting is where the number of processes is comparable with the number of processors, so the overhead of running each thread as a heavyweight process seems somewhat reasonable.

Erlang message-passing is done by sending some data, then calling receive.

In the library I wrote, which is here , message-passing is done by calling a method on a proxy object representing the other process.  The called process registers function calls, or an object instance with methods, with the threading library, and those will get called when two things happen:

• the called process calls ‘receive’, as in erlang
• another process calls a method on the called process’s proxy object

In a simple benchmark against classic python threading, it ran nearly twice as fast, on a dual-proc machine (an EEEPC ).

I haven’t tried against stackless, but stackless uses green threads that run in a single os thread, hence I anticipate on a single processor, and I believe stackless will suffer from the same single-threading issue as classic python threads.

C# is very cool.  Just wanted to say that   It’s not really politically acceptable in opensource I feel, and it’s objectively not particularly portable (sure there is mono, but its implementation is far from 100% identical to the Microsoft.Net implementation, it’s more of a ‘we have a c# program we want to run on linux, let’s hack it to run in mono’ than ‘oh, c#, it runs everywhere!’).

But it is very cool: memory utilization seems reasonable, it’s very fast to develop in I feel (maybe three times faster than C++?), it runs very quickly (to within 50% of C++ I feel, in rough benchmarks, including things like opengl, where there are lots of native calls to the hardware going on).

Also, it compiles very fast, but… Parkinson’s Law   I worked on a C# project in 2008.  One might think that C# compiles so fast that even a relatively large project would compile in a minute or two, but that doesn’t account for the issue that we let  ours get so large that it still took twenty minutes to compile!

I couldn’t help thinking that if it was in C++, we’d have refactorized it into smaller compilation units, and each unit would have taken 20 minutes to compile, same as for C#

So, actually I thought this post would be over-the-top enthusiastic about C#, but in fact it became a highly qualified backhanded enthusiasm   C# should be really cool I feel, but in fact, in reality, ultimately the developer experience is basically identical to that in C++

While not quite as earth-shattering as my ephipany in discovering sqlalchemy for python, Jinja2 is pretty cool too.  No longer do I have to have partial envy of php’s jsp/asp-type syntax.  It’s fairly easy to use.  I like it.  It’s not the only templating library around for Python – there is also Mako – but it seems pretty cool.

SQLAlchemy is an Object Relational Manager (ORM) for Python.  It’s like Hibernate, but for Python.  Before finding out about it, I was rather unsure how one could go about creating hibernate in python, since there are no annotations or static types, but sqlalchemy works great!  Really easy.  Excellent.  Professional-looking website too, very polished.  Awesome.