scot hacker's scripts and utils

Recently I was invited to participate in the California Data Camp and DataSF App Contest hosted by California Watch and spot.us. The unconference would feature lots of discussion about making use of publicly available data sets to improve quality of life. The App Contest challenged developers to choose one of the many data sets available at DataSF.org and build something cool with it in a relatively short period of time. Here’s a showcase of existing apps built on those data sets.

The Knight Digital Media Center (where I work) invited me to take part, and I chose a database of 64,000 San Francisco trees and plants. The goal of the project was to:

• Make it easy for citizens to explore and discover the huge number of plant species and individual trees maintained by the city
• Make it easy for citizens to “flag” a tree as needing maintenance, water, food, etc.
• Make it easy for citizens to request a tree at a particular location
• Provide data visualization tools to let citizens explore and understand the plant variety visually
• Make it easy to see what a given species will look like in 5,10,15,20 years when requesting a tree
• Ideally, a future version of the app would include ecology data on all species, listing the water consumption and carbon offset of each

I decided to build the project on Django, of course. Put a total of around 15 hours into the project, about half of which was spent massaging and cleaning the provided data, which had multiple pieces of information stuffed into single fields, non-standard date formats, and was completely non-relational. Cities implementing django-treedata “fresh,” without having to be compatible with an existing data entry system, won’t have to worry about data conversion/format issues.

Once the data was clean, the rest was pretty straightforward Django stuff. The one non-standard aspect is the external “lastcount” script, which counts the number of instances of each species and stores the result on a field in the Species model. Doing this in real time for such a large number of trees turned out to be very computationally expensive, so the script needs to be run from a crontab periodically.

Because dev time was so limited, all of it went into data cleaning and building out the models and views. We’ve put ZERO work into design considerations, so please don’t crucify us for that. The CSS is built on top of the excellent 960 Grid framework, so layout will be easy. Some of the data visualization is done via the excellent Google Charts API.

Much to our surprise, the django-treedata app won the competition!

Please note that the project has only been run in a development environment and has never been publicly deployed – the project as it stands should be considred a starting point for cities to built on. The readme explains more. The project is completely open source and is released under the very liberal BSD license – do with it as you will.

Thanks also to J-School webmaster for Chuck Harris for his contributions to the project

Django projects can end up with complex sets of Users, Groups, and multiple profile types representing different types of people. For example a school site might have Students, Parents, Teachers, Staff, and Alumni. The mailing lists for that school will live completely outside of the Django project, but there’s a good chance you’d like to be able to populate list membership from your membership database rather than maintaining lists by hand. And you’d like to be able to use any combination of criteria to populate your lists (Group membership,  profile types, join date, privileges, etc.)

Since Mailman is installed on so many web hosts by default, there’s a good chance you’re using it, and have lots of overlapping and non-overlapping groups subscribed to various lists. I recently went through the process of integrating a bunch of Mailman lists with the membership representing a school intranet built in Django and thought I’d document it for anyone going through a similar process.

Once everything is set up, you’ll never need to use the Mailman interface to manage lists again – everything will be automated and self-maintaining.

django-todo is a multi-user, multi-group task management and assignment system for the Django web application framework.

Designed as a “pluggable” application ready to import into existing Django projects.

You must already have a  login/authorization system installed and working in your project before installing django-todo.

This version supports drag and drop task prioritization, email notification to task assignees, and lots more. Feedback welcome.

The code is hosted on Google Code and can be checked out via svn.

This is a very crude mass category deletion script for WordPress. I recently inherited a WordPress site with more than 6000 categories. After deleting a ton of old posts, we were left with more than 4000 unused categories, and I needed a way to remove them quickly. Couldn’t find a plugin up to the task, so wrote this. Very crude, but effective. Not a plugin, but a small WP companion script. See instructions in the comment at top of script.

Download Cleancats

Super fast way to install WordPress for clients, via subversion. Yes, users can often self-install via Fantastico or similar programs, but what guarantee do you have that they’ll upgrade as soon as new releases become available? Letting users run old versions of web software is a great way to get hacked. Take control of users’ installations by checking them out via svn (with this script) and managing them with wp-mass-upgrade.

This script performs the following tasks:

• Gather installation info
• Create install dir and check out a copy of WordPress
• Create database, db user, set db privs via external .sql file
• Create WP config file
• Create upload dir and set filesystem permissions
• Generate array line for wp-mass-upgrade.sh

Final setup is done via browser.

Download wp-create 1.0

A dirt-simple WordPress plugin that lets you provide workflow docs to your authors and editors from within the WP admin interface.

A WordPress plugin aimed primarily at events sites, where you want to be able to timestamp posts in the future but have them appear immediately (by default, WordPress will not display a future timestamped post until its go-live date rolls around). This plugin sets the post_status field to “publish” rather than “future” when publishing a post, even if its timestamp is in the future. Written by Ryan Boren – I’m just hosting it.

Designed for Mac users with an iPod but no PDA who want to do paperless geocaching complete with descriptions, log entries and hints. Uses the iPod’s “Notes” feature to display complete cache description pages. gpx2ipod is built on top of gpsbabel.

Note: I’m not likely to continue development of gpx2ipod, for two reasons: 1) MacCaching finally grew the ability to store cache metadata in the iPod’s “Notes” feature rather than in the address book (which was the whole impetus behind my developing gpx2ipod to begin with), and 2) I switched to a Garmin Colorado a few months ago, which displays description, logs, and hints directly in the GPS. I no longer need to offload that data to the iPod, so my itch has been scratched. Do with this software what you will!

Designed for hosts with lots of individual WordPress blogs to manage. If all blogs are managed via subversion, this script can iterate through all of them and upgrade each to the latest version in seconds.

Here at Birdhouse Hosting, we use this system every time WordPress is updated, and are able to bump 100+ WordPress installations to the next version in three minutes flat.

Designed to be used in conjunction with WP-Create, which is the fastest way to set up fresh WP installations via svn.

Download wp-mass-upgrade

Sends periodic email notifications of new WordPress blog entries (in either plain text or HTML mode) to a separate subscribers mailing list. Maintains cache file recording ID of last-sent post. On subsequent runs, sends all posts created since the previous run.

Currently driving weekly email updates from birdhouse.org and other sites.

Sends email announcements to your MT Notifications list or to an external mailing list on recent Movable Type weblog entries.

Harvests bad IP addresses from multiple sources and adds them to the ConfigServer firewall (CSF). This version works with WordPress and Movable Type weblogs, and optionally the exim ACL deny list. Future versions will scan other sources for bad IPs as well.

Download reducer

This is a script for WHM/cPanel administrators to help them keep tabs on client mailboxes approaching or exceeding their quotas. Sends warning messages to owners of near-quota mailboxes and a separate summary message to postmaster or other address. Must be run as root, or via root crontab.

Download checkmailquota

vim has a bad habit of chewing up a ton of CPU if user backgrounds it, closes their terminal window, or gets disconnected from the net with a vim session open. The vim reaper checks whether owner of running vim processes are logged in. If not, the process gets reaped.

Download vimreaper

For users of Apple’s QuickTime Streaming Server who need to perform batch-processing of QT movie files. Adds metadata, adds hint tracks, and generates .qtl reference movies, then moves everything to a destination directory. Requires the qtmedia and qtref command-line binaries only present in OS X Server.

Download makestreams

A quick-n-dirty shell script to rename comment and trackback scripts across a Movable Type site. For emergency use when a server is undergoing a spam blitzkrieg and server performance is suffering. Written for syadmins dealing with high loads on MT-heavy servers.

Download comstop

WPBlogMail is now deprecated. Please use WP-Digest instead.

This is a very simple contacts database written in PHP/MySQL as a companion to a November 2002 article I wrote for MacWorld: Serve It Up. OS X and its Unix underpinnings were new, and MacWorld wanted to run articles showcasing the Mac’s entry into Unix-side server capabilities at the desktop level.

MacWorld no longer links to this sample application, but it should still be useful to anyone wanting to learn PHP/MySQL with a simple (but useful), well-commented starter application.

Download contacts.tgz

In the BeOS glory days, the RipEnc shell script was the premiere MP3 ripping/encoding solution, and made full use of the database-like BeOS filesystem to store MP3 metadata. After I left the BeOS world, others took over the script. The current version is available at BeBits. The version here is the last one created and maintained solely by me.

Two downloads are available – a .pkg installer for people still running BeOS/Haiku, and a text version for people without access to BeOS but curious about the command line tools BeOS provided to manipulate the filesystem database.

Download ripenc_26a.pkg

Download ripenc_2.6a.txt