The Knight Digital Media Center (where I work) invited me to take part, and I chose a database of 64,000 San Francisco trees and plants. The goal of the project was to:

• Make it easy for citizens to explore and discover the huge number of plant species and individual trees maintained by the city
• Make it easy for citizens to “flag” a tree as needing maintenance, water, food, etc.
• Make it easy for citizens to request a tree at a particular location
• Provide data visualization tools to let citizens explore and understand the plant variety visually
• Make it easy to see what a given species will look like in 5,10,15,20 years when requesting a tree
• Ideally, a future version of the app would include ecology data on all species, listing the water consumption and carbon offset of each

I decided to build the project on Django, of course. Put a total of around 15 hours into the project, about half of which was spent massaging and cleaning the provided data, which had multiple pieces of information stuffed into single fields, non-standard date formats, and was completely non-relational. Cities implementing django-treedata “fresh,” without having to be compatible with an existing data entry system, won’t have to worry about data conversion/format issues.

Once the data was clean, the rest was pretty straightforward Django stuff. The one non-standard aspect is the external “lastcount” script, which counts the number of instances of each species and stores the result on a field in the Species model. Doing this in real time for such a large number of trees turned out to be very computationally expensive, so the script needs to be run from a crontab periodically.

Because dev time was so limited, all of it went into data cleaning and building out the models and views. We’ve put ZERO work into design considerations, so please don’t crucify us for that. The CSS is built on top of the excellent 960 Grid framework, so layout will be easy. Some of the data visualization is done via the excellent Google Charts API.

Much to our surprise, the django-treedata app won the competition!

Please note that the project has only been run in a development environment and has never been publicly deployed – the project as it stands should be considred a starting point for cities to built on. The readme explains more. The project is completely open source and is released under the very liberal BSD license – do with it as you will.

Thanks also to J-School webmaster for Chuck Harris for his contributions to the project

Screenshots

Download

django-treedata is available on Google Code, via svn.

Since Mailman is installed on so many web hosts by default, there’s a good chance you’re using it, and have lots of overlapping and non-overlapping groups subscribed to various lists. I recently went through the process of integrating a bunch of Mailman lists with the membership representing a school intranet built in Django and thought I’d document it for anyone going through a similar process.

Once everything is set up, you’ll never need to use the Mailman interface to manage lists again – everything will be automated and self-maintaining.

Manual Overrides / Prerequisites

• The simple queries are easy. But in the real world you end up in situations where, say, the Maintenance person needs to be able to write to the Teachers list but not receive mail from it. Or maybe teachers need to be able to write to the Board, but not be privy to the Board’s list traffic.
• Some people may fit a particular group query but not want to receive mail from lists for those groups (because maybe their spouse handles that stuff).
• Some people don’t match a particular query group but need to be made into regular list members anyway.  For example you might want to have the principal be on all mailing lists, even though he’s not a parent of any child enrolled in a class.

The key to making some of this happen is a feature of Mailman called “nomail” mode, which allows certain subscriber addresses to send to a list but not receive mail from it. This recipe interfaces with Mailman from the command line, using the add_members command. But here’s the rub – the official Mailman distribution’s add_members command is missing the option to add a person to a list in nomail mode (grrrr). However, the version of add_members that comes with OS X Server does include this option. I have no idea why Mailman hasn’t seen fit to integrate Apple’s changes into the official distro, since it’s insanely useful.  But since add_members is just a Python script, you can replace the version of add_members that comes with the official distro with Apple’s version and it’ll work fine on any server platform.

Because the Mailman command-line scripts must be run as root, you must be the administrator of the server this recipe runs on.

To handle all the exceptions and special cases your organization probably has, you’re going to need two tweaks to your apps/models. First, on your Profile model(s), create a new field for people who don’t want to be on the lists at all:

no_lists = models.BooleanField(default=False,
     help_text="When checked, this parent will NOT be subscribed
     to the mailing lists they normally would be.")

You can now check this box in any user profile for people who show up in group queries but don’t wish to receive mail from any of the organization’s lists.

You’re also going to need an additional model to store the exceptions – additional people to add to lists, either in yesmail or nomail mode:

class ListExtra(models.Model):
    """
    Extra email addresses to be added to mailing lists.
    """
    list = models.SlugField()
    addresses = models.TextField(blank=True,
         help_text='Add addresses here, one per line.')

    def __unicode__(self):
        return u'%s' % (self.list)

Once the model has been created and you’ve run syncdb, create a record for each list you want to track, e.g. “teachers” for the list “teachers@ourschool.org” .  Do not enter real teachers here – they’ll come from ORM queries on the regular User table.  Instead, this is where you’ll add non-teachers  who should be regular members of the teachers@ mailing list anyway. Add such people’s addresses to the field one-per-line.

If you have the need to let certain people write to the teachers list but NOT receive mail from it, create an additional record called “teachers-nomail”.

How It Works

This recipe is really two scripts – one in Python and the other bash – that I end up running from a single shell alias as needed. The two scripts I use can be downloaded at the end of this post and are well commented.

The Python script (listgen.py) interacts with the Django ORM, extracting list memberships out into text files. The bash script (listgen.sh) iterates through that set of text files and, for each, quickly unsubscribes the entire membership and then re-subscribes them from the text files generated by the python script. On my server, the whole process unsubs and resubs 14 lists covering 125 people in  5 seconds flat, so there’s no real danger of anyone freaking out because they were temporarily unsubscribed.  And  listgen.sh makes sure that subs and unsubs occur without notifying users of list membership changes, so they don’t get scary messages.

listgen.py sets up the Django environment for the given project, defines a function for processing query results into text files, then iterates through a list of mailing list names (without the domain name). For each, it runs a Django ORM query. Then it excludes from that queryset anyone who has opted out of all lists.

Next, it looks for a corresponding ListExtras record and extracts the “extra” addresses it contains. Finally, it looks for a corresponding “nomail” ListExtra record (e.g. teachers-nomail). It then removes any duplicate members and passes the final query data to the file processing function and writes text files to a directory on the server.

listgen.sh iterates through an array of all mailing lists and, for each, unsbuscribes all members (without sending unsub messages). It then locates the text file corresponding to the current list and subscribes all regular members in yesmail mode. Finally, it looks for a corresponding listname-nomail.txt file and subscribes the addresses it contains in nomail mode.

Deployment

The scripts I’m using are linked below.

First, check to see whether the version of add_members on your server supports the -e flag (for nomail mode). If it does not, rename the add_members script on your server to -old and put the replacement add_members script in its place. Run “locate add_members” if not sure where to put it. If it complains when you try to run it, make sure its first line matches the python path shown on other scripts in the same dir.

Put listgen.py in a “scripts” directory in your Django project.

Put listgen.sh somewhere where root can get to it.

You’ll need to modify the list of mailing lists in each script, as well as the most important bit – the Django ORM queries in listgen.py. This will probably take quite a bit of testing and experimentation – make sure you’re working with test data until everything is humming – users will get righteously pissed if you break their lists.

You’ll also need to make sure the two scripts agree on the filesystem location for the output  (for the generated files that listgen.py creates and listgen.sh reads).  By default this will be a dir at myproject/scripts/listgen. You’ll need to edit the python path vars at the top of listgen.py.

During testing you’ll probably want to just run listgen.py and not listgen.sh — then you can study the text file output without actually altering the lists. When you’re confident listgen.py is doing the right thing, add the listgen.sh step.

Once it’s all working, create an alias for root that runs both scripts, something like:

alias schoollist='python /path/to/scripts/listgen.py; \
/bin/sh /root/scripts/for_customers/school/listgen.sh'

You could set this up to run as a cron job, but there’s a good reason not to: Mailman keeps track of bad addresses with its bounce processing feature, which counts the number of times a message has bounced from a bad address and then unsubscribes that user and alerts the list administrator. If you run listgen.sh every night, every member will always be “new” as far as Mailman is concerned and bounce thresholds will never be reached.  You’ll never be informed when/if you have bad addresses in your Django system. To address this, I wrote a Django signal handler that sends me email whenever a User object has been updated. I then run the schoollist command whenever I know something has changed (after the system stabilizes this shouldn’t happen very often).

The Scripts

add_members (copied from OS X Server’s Mailman distro, but works on any platform)

listgen.py (put in a scripts dir inside your Django project)

listgen.sh (put somewhere root can access easily)

Designed as a “pluggable” application ready to import into existing Django projects.

You must already have a  login/authorization system installed and working in your project before installing django-todo.

This version supports drag and drop task prioritization, email notification to task assignees, and lots more. Feedback welcome.

The code is hosted on Google Code and can be checked out via svn.

Screenshots

Download Cleancats

This script performs the following tasks:

• Gather installation info
• Create install dir and check out a copy of WordPress
• Create database, db user, set db privs via external .sql file
• Create WP config file
• Create upload dir and set filesystem permissions
• Generate array line for wp-mass-upgrade.sh

Final setup is done via browser.

Download wp-create 1.0

Instructions: Unpack this folder to your plugins directory and activate it. Click on the Dashboard and you’ll see a new tab called “NGNO Docs.” Within that, you’ll find documentation I wrote for my authors for one particular site. In order for this to be useful, you’ll need to replace the HTML it contains with your own by editing the file docs.html.

You’ll want to remove the image files I’ve included and, if desired, add your own. Note: I found it tricky to get image paths working – you’ll need to embed images like this:

img src="../wp-content/plugins/wp-workflow-docs/more-result.gif"

You can change the plugin name and tab name by editing the plugin file itself – it’s a tiny plugin and it will be pretty obvious where to do this in the file. Then de-activate and re-activate it to see the new tab name.

Download wp-workflow-docs

Contributions / improvements welcome!

Instructions: Place future-post.php in your plugins directory and activate it. Write a post with a future timestamp and hit publish. Notice that it goes live on your site immediately.

Note: This seemingly simple plugin was graciously written by the magical Ryan Boren when I was facing a deadline. He doesn’t have time to maintain/host it, so I agreed to.

Download version 1.0

Contributions / improvements welcome!

Note: I’m not likely to continue development of gpx2ipod, for two reasons: 1) MacCaching finally grew the ability to store cache metadata in the iPod’s “Notes” feature rather than in the address book (which was the whole impetus behind my developing gpx2ipod to begin with), and 2) I switched to a Garmin Colorado a few months ago, which displays description, logs, and hints directly in the GPS. I no longer need to offload that data to the iPod, so my itch has been scratched. Do with this software what you will!

Overview

Mac-based paperless caching for people who own an iPod but not a PDA.
Batch-converts a pile of .gpx files to plain text for use with the iPod’s “Notes” feature.
Super-fast — cut your geocaching prep time to a few minutes.

gpx2ipod handles both individual and Pocket Query (multiple-cache).gpx files.
Cache files will display alphabetically on the iPod for easy access in the field.

gpx2ipod can inject generated text files directly into your iPod (most users) or into a local “output” folder (you might not have an iPod but might still want the text files for other purposes).

gpx2ipod is a Terminal application (shell script), but can be run painlessly with a double-click — no shell experience required.

gpx2ipod is available either with or without the gpsbabel command-line binary bundled. gpsbabel is licenced under the GPL, and thus so is gpx2ipod. See LICENSE for info.

Who this is for

• You’re a Mac user who has an iPod but not a PDA.
• You want to do paperless geocaching.
• You want complete descriptions, log entries and hints while out on the trail.
• Your iPod has this great “Notes” feature, but you haven’t found any OS X software to get GPX files into a format you can read in the iPod’s Notes reader. MacCaching is wonderful, but it outputs to the iPod’s “Contacts” system, which means no logs and no choice of whether to encrypt hints. It also intermingles hundreds of geocache data files with your existing pile of Contacts – very messy. GPSBabel+ has a similar “Export to v-card feature” with similar problems. The iPod’s “Notes” feature seems a much better choice to me.

Requirements

Mac OS X. This script will not run on Windows or Linux (with modifications, gpx2ipod could be made to run under Cygwin or Linux).

A copy of the free and open source gpsbabel (version 1.3.4 or higher). Thanks to all contributors to gpsbabel for your hard work on this amazing workhorse! Huge thanks also to Robert Lipe, who added gpsbabel’s ability to split multi-location files into individual files, so gpx2ipod wouldn’t be stuck with the 99-split limitation in the Unix “cut” command.

An iPod with Storage Mode enabled. Storage Mode lets you use your iPod as a hard drive. If your iPod does not show up as disk volume on the Desktop when plugged in, go to iTunes and select “Enable disk use,” then click Apply.

Installation

Unzip this folder and drag it into your Applications folder (or wherever you like). Then drag the gpx2ipod folder icon from your Applications folder into your Dock for quick access.

• If you downloaded gpx2ipod with gpsbabel:You’re good to go! Skip to Usage.
• If you downloaded gpx2ipod without gpsbabel:gpx2ipod requires the gpsbabel command line application, (version 1.3.4 or higher). When you install GPSBabel (probably into your Applications folder), you’ll see two files: GPSBabel+, which is a graphical application, and gpsbabel itself, which is a command-line application. There are two ways to make gpsbabel available to gpx2ipod:The easy way: Select the file “gpsbabel” in that folder and copy it (don’t move it!) to the bin folder inside the gpx2ipod folder.The hard (but better) way: If you’re comfortable at the command line and prefer not to duplicate the binary, you can create a symlink instead, for example:

  ln -s "/Applications/GPS Babel/gpsbabel" /Users/[yourlogin]/Desktop/gpx2ipod/bin/gpsbabel

  Modify that command to match your setup. Note: Mac aliases won’t work – you’ll need a proper symlink.

Usage

1. Download .gpx files from geocaching.com and move or copy them into the “input” folder. You can mix and match individual .gpx files and Pocket Query (multi-cache) files in the same input folder. gpx2ipod includes a “samples” folder for testing – copy some or all of these into the “input” folder and let ‘er rip.
2. Double-click the gpx2ipod “squid” and let it do its thing.
3. • If this is the first time you’ve run gpx2ipod, you’ll be asked to set a few preferences, such as whether you want to include log files, whether to encrypt hints, whether to copy the original .gpx files into an “archive” folder, and (most importantly) the system path to your iPod (the script will let you choose your iPod from a list). For the yes/no questions, use 0 for “no” and 1 for “yes.”
   • If the gpsbabel binary can’t be found, or is an old version, the script will tell you and then halt.
   • If the iPod you selected during Setup can’t be found, you’ll be given the option to mount it now, or, alternatively, to have generated files sent to an “output” folder instead.
4. When everything’s in order, gpx2ipod will crunch through all those .gpx files, spit out text files optimally formatted for iPod use, and copy them into the Notes folder on your iPod. On future runs of gpx2ipod, your last-used preferences will be displayed as the script launches. Hit [Return] to accept the settings, or C to change them.
5. Go geocaching! You’ll find the cache entries on the iPod in Extras > Notes.

Limitations

This version does not load .gpx files into your GPSr – it just generates text files for use on the iPod. A future version may load the same files into the GPSr.

History

v1.3 – 9/15/2007
Now retains character formatting for international character sets. Default is UTF-8, but any character set supported by gpsbabel can be selected in preferences. Tested against Swedish .gpx samples (see GCY7XZ-Swedish_chars.gpx in Samples folder). Many thanks to Bengt Bäverman for excellent contributions on this feature.

v1.2 – 9/15/2007
Bug fix for other cache name characters that would generate illegal filenames.

v1.1 – 9/08/2007
No longer generates errors when encountering caches with slashes in their names. Now works properly when installed in a path containing a space (such as “/Applications/GPS Apps”).

v1.0 – 8/03/2007
Renamed project to “gpx2ipod.” Now requires gpsbabel 1.3.4 or higher, taking advantage of new gpsbabel “splitoutput” feature in order to overcome the cut command’s 99-split limit (thanks Robert Lipe!). Now double-clickable – much better user experience. Now copies generated files directly to iPod. Now stores user preferences in a config file. Now optionally copies original .gpx files into an “archive” folder for future use.

v0.2 – 7/17/2007
Now handles both individual .gpx files and Pocket Query (multiple-cache) .gpx files.

v0.1 – 7/14/2007
Original version

Download

Download without GPSBabel

Download with GPSBabel

Here at Birdhouse Hosting, we use this system every time WordPress is updated, and are able to bump 100+ WordPress installations to the next version in three minutes flat.

Designed to be used in conjunction with WP-Create, which is the fastest way to set up fresh WP installations via svn.

Download wp-mass-upgrade

Currently driving weekly email updates from birdhouse.org and other sites.

About

Sends periodic email notifications of new WordPress blog entries, either in plain text or as formatted HTML, to a separate subscribers mailing list. Records ID of last-sent post. On subsequent runs, sends all posts created since the previous run. Mail is sent to a single email address, which is most likely a separate subscribers mailing list.

The first time WP-Digest is run, it operates on the last n entries made in your blog. After the first run, it will save the ID of your last blog entry in a cache file. On subsequent runs, it will operate on however many entries have been made since the previous run. You can manually control the value of n by changing $MinPosts in conf.php.

Requirements

This script is desgined to be run on Unix/Linux/OS X servers. It may work on Windows servers as well, but has not been tested in that environment. Of course you’ll need to use some alternative to cron if working under Windows.

This script requires access to the lynx web browser installed on your server. To determine whether you have it, type:

which lynx

If the system returns “Command not found” then you don’t have it, and should ask your sysadmin to install it. This script is designed to be run at regular intervals via crontab.

See included readme.txt for setup instructions.

History

2.0: Now sends your choice of plain text or HTML-formatted (multipart) email.

1.2: Completely new architecture, now based on lynx for improved formatting options and an end to problems with incorrectly formatted characters and error messages in output generated by 3rd party plugins when run via cron.

1.1: Bug fixes

1.0: Started as a port of mtblogmail, but quickly turned into a complete rewrite in order to take advantage of the WordPress APIs.

Download

Download wp-digest