Yes, users can often self-install via Fantastico or similar programs, but what guarantee do you have that they’ll upgrade as soon as new releases become available? Letting users run old versions of web software is a great way to get hacked. This kit lets you take control of users’ installations by checking them out via svn and upgrading them en masse.

At Birdhouse Hosting we use WordPress Mass Management Tools to create all new installations. When WP updates are released, we’re able to upgrade more than 100 WP installations belonging to 80+ users in a few minutes flat.

The kit is designed to be run by root – these are not tools for WordPress end users. The kit assumes basic knowledge of the bash shell.

This kit replaces the old wp-create and wp-mass-upgrade scripts which were distributed separately and are no longer supported. Includes:

wp-create.sh: Super fast way to install WordPress for clients, via subversion. Performs the following tasks:

• Gather installation info
• Create install dir and check out a copy of WordPress
• Create database, db user, set db privs via external .sql file
• Create WP config file
• Create upload dir and set filesystem permissions
• Generate array line for wp-sites.sh

Final setup is done via browser.

wp-mass-upgrade.sh: Iterates through all sites listed in wp-sites.sh, backs up the site’s database, upgrades to a specified version of WordPress, and sends the site owner email announcing the upgrade.

wp-mass-plugins.sh: Generates a list of all active plugins used by all sites on the server. The list can be compared to lists of known-incompatible plugins to help you make decisions about whether to skip any sites on the list.

Download wp-mass-tools 1.1

See the included readme.txt for documentation.

Unfortunately, over the past couple of years I’ve favorited way more things than I’ll ever have time to read. As of now, I’ve got 1600 favorites waiting to be read. Ain’t never gonna happen. I declare Twitter Favorite bankruptcy! Needed a way to batch-unfavorite the whole collection, and twitter.com doesn’t provide a tool for that. Time to dive into the Twitter API.

I heart Python, and Tweepy is the best Twitter API tool for Python (documentation here). So start by getting Tweepy installed and on your Python path. If it’s installed correctly you should be able to type “import tweepy” in a python shell and get no errors.

Once installed, you’ll need to authenticate to Twitter. Note that we’re using basic authentication here. The “right” way is to use OAuth, but that requires registering your application with twitter.com – something users with this simple need won’t want to get into. Let’s explore.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

import tweepy
auth = tweepy.BasicAuthHandler('your_username', 'your_password')
api = tweepy.API(auth)
 
# Get recent items from your timeline:
posts = api.user_timeline('shacker')
[p.text for p in posts]
 
# Cool, it works. Now get the list of  favorites for the 
# authenticating user (you).
favs = api.favorites()
[p.text for p in favs]
 
# How many did you get?
>>> len(favs)
20
 
# Oops, that's only 20. But we need all 1600! Unfortunately, 
# you can only get a "page" of favorites at a time - which 
# is usually a batch of 20. We're going to have to cycle 
# through them. The docs say to use:
 
API.favorites([id][, page])
 
# So since page is the 2nd arg, we're going to need 
# your user ID in order to access pages. So find your user ID:
 
>>> api.get_user('your_username').id
14098497
 
# So 14098497 is your user ID. Now you can go back and 
# use the favorites API to get things in pages. 
# IOTW for page 2 of the results, use:
 
>>> favs = api.favorites('14098497', '2')
>>> [t.text for t in favs]
 
# So how do we un-favorite a tweet? The docs say:
# API.destroy_favorite(id)
# So if we instead list the tweet IDs of the result set:
 
>>> [t.id for t in results]
[4601622480, 4587326808, 4527173532, 4514186302, 4497778507, 4472272715, 4431075519, 4425332211, 4048507627, 4043010657, 4023672253, 4023524092, 4022466872, 4022334337, 4016169277, 4015641948]
 
# We can grab the first ID in the list and test with that. 
# Go to twitter.com, click on Favorites, find the first 
# tweet listed, and click the timestamp. The URL should 
# include the ID of the first tweet in the list. 
# Back in the shell, try and unfavorite it:
 
api.destroy_favorite(4601622480)
 
# Now refresh your Favorites page and notice that the 
# first tweet is gone from the list. Groovy. All we 
# have to do now is script the process.

Some of the tweepy methods provide a ‘cursor’ argument to help you iterate through pages, but the favorites() method does not. But we don’t really need one – we can simply iterate through pages until the results list comes back empty.

But there’s a hitch: Twitter’s basic authentication is limited to 150 API requests per hour, and each un-favorite counts as one request. If you have a lot of favorites like I did, you’ll end up hitting your API limit pretty quickly. The solution is to “sleep” the script between each page of results. The math says we should be able to sleep for 8-10 minutes per page and come in under the 150/hour limit. But in practice, I found myself hitting the limit anyway (browser and desktop client running at the same time skew the results, and we want to be safe). In other words, this is not a script that will complete in a few seconds – with a 15 minute pause between each cycle, you’ll want to run it in the background while you do other things. The 15-minute pause is designed to allow for enough spare cycles every hour that you can continue using Twitter normally. No rush, right? Chillax!

So save the following as a text file (in the same directory as tweepy.py if you don’t grok python paths), change the username and password at the top, and run it with python filename.py.

Warning: This script commits total bankruptcy on your favorites – it will remove ALL of them. Run it only if you’re sure you want to do that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import tweepy
from time import sleep
 
username = 'your_username'
password = 'your_password'
 
auth = tweepy.BasicAuthHandler(username, password)
api = tweepy.API(auth)
userid = api.get_user(username).id
 
# Counter is used for reporting purposes only
i = 1
 
'''
Tweepy will return pages of favorites as lists.
In Python, an empty list is considered false, 
so we we don't need to know the number of result pages -
just keep going until the results list is empty. 
'''
 
# Initiate an infinite loop, to be broken when we run out of favorites.
while True: 
 
    results = api.favorites(userid)
    if results:
        print "Page %d exists. Unfavoriting %d tweets." % (i, len(results))
        for r in results:
            api.destroy_favorite(r.id) # Here's the money shot.
 
        # Rest for a while.
        print "Script is sleeping for 15 minutes to accommodate Twitter rate limits."
        print
        i += 1        
        sleep(900)       # 900 seconds = 15 minutes
    else:
        print "No more favorites. Buh-bye."
        break

By morning, you should have no favorites left. Enjoy the clean slate while it lasts!

Oh, by the way, I’m shacker on Twitter.

Update: If you have a lot of favorites, you may encounter some weirdness after running this script for a while. I found that after the first thousand un-favorite requests, Twitter started rate-limiting me down from 20 tweets per page to 14, then to 8, then 2, then 1. When it got to one, I tweaked the script to request one per minute instead of waiting 15 minutes between. At that rate, I was making only 60 requests per hour. But eventually, twitter started returning zero. And lo and behold, when I accessed twitter.com/favorites in a browser, I got a “Something is technically wrong” message from Twitter.

I logged out and then in again with another account and /favorites was working fine – they had basically made it impossible to access my favorites in any way from the account that had been making all the un-favorite requests. This was puzzling and troubling. I had taken great pains to play with the API by the stated rules, to not exceed the rate limits, and to be gentle on their system… but still they treated this script like it was a bad actor. If anyone from Twitter reads this and can explain what’s going on, I’d love to hear what I can do to make it better.

A post at TechCrunch explains that Twitter has been slashing its rate limits lately, which is probably related to the behavior I’m experiencing.

If you have fewer than a thousand favorites, this script should still work fine, with no ill effects.

The Knight Digital Media Center (where I work) invited me to take part, and I chose a database of 64,000 San Francisco trees and plants. The goal of the project was to:

• Make it easy for citizens to explore and discover the huge number of plant species and individual trees maintained by the city
• Make it easy for citizens to “flag” a tree as needing maintenance, water, food, etc.
• Make it easy for citizens to request a tree at a particular location
• Provide data visualization tools to let citizens explore and understand the plant variety visually
• Make it easy to see what a given species will look like in 5,10,15,20 years when requesting a tree
• Ideally, a future version of the app would include ecology data on all species, listing the water consumption and carbon offset of each

I decided to build the project on Django, of course. Put a total of around 15 hours into the project, about half of which was spent massaging and cleaning the provided data, which had multiple pieces of information stuffed into single fields, non-standard date formats, and was completely non-relational. Cities implementing django-treedata “fresh,” without having to be compatible with an existing data entry system, won’t have to worry about data conversion/format issues.

Once the data was clean, the rest was pretty straightforward Django stuff. The one non-standard aspect is the external “lastcount” script, which counts the number of instances of each species and stores the result on a field in the Species model. Doing this in real time for such a large number of trees turned out to be very computationally expensive, so the script needs to be run from a crontab periodically.

Because dev time was so limited, all of it went into data cleaning and building out the models and views. We’ve put ZERO work into design considerations, so please don’t crucify us for that. The CSS is built on top of the excellent 960 Grid framework, so layout will be easy. Some of the data visualization is done via the excellent Google Charts API.

Much to our surprise, the django-treedata app won the competition!

Please note that the project has only been run in a development environment and has never been publicly deployed – the project as it stands should be considred a starting point for cities to built on. The readme explains more. The project is completely open source and is released under the very liberal BSD license – do with it as you will.

Thanks also to J-School webmaster for Chuck Harris for his contributions to the project

Screenshots

Download

django-treedata is available on Google Code, via svn.

Since Mailman is installed on so many web hosts by default, there’s a good chance you’re using it, and have lots of overlapping and non-overlapping groups subscribed to various lists. I recently went through the process of integrating a bunch of Mailman lists with the membership representing a school intranet built in Django and thought I’d document it for anyone going through a similar process.

Once everything is set up, you’ll never need to use the Mailman interface to manage lists again – everything will be automated and self-maintaining.

Manual Overrides / Prerequisites

• The simple queries are easy. But in the real world you end up in situations where, say, the Maintenance person needs to be able to write to the Teachers list but not receive mail from it. Or maybe teachers need to be able to write to the Board, but not be privy to the Board’s list traffic.
• Some people may fit a particular group query but not want to receive mail from lists for those groups (because maybe their spouse handles that stuff).
• Some people don’t match a particular query group but need to be made into regular list members anyway.  For example you might want to have the principal be on all mailing lists, even though he’s not a parent of any child enrolled in a class.

The key to making some of this happen is a feature of Mailman called “nomail” mode, which allows certain subscriber addresses to send to a list but not receive mail from it. This recipe interfaces with Mailman from the command line, using the add_members command. But here’s the rub – the official Mailman distribution’s add_members command is missing the option to add a person to a list in nomail mode (grrrr). However, the version of add_members that comes with OS X Server does include this option. I have no idea why Mailman hasn’t seen fit to integrate Apple’s changes into the official distro, since it’s insanely useful.  But since add_members is just a Python script, you can replace the version of add_members that comes with the official distro with Apple’s version and it’ll work fine on any server platform.

Because the Mailman command-line scripts must be run as root, you must be the administrator of the server this recipe runs on.

To handle all the exceptions and special cases your organization probably has, you’re going to need two tweaks to your apps/models. First, on your Profile model(s), create a new field for people who don’t want to be on the lists at all:

no_lists = models.BooleanField(default=False,
     help_text="When checked, this parent will NOT be subscribed
     to the mailing lists they normally would be.")

You can now check this box in any user profile for people who show up in group queries but don’t wish to receive mail from any of the organization’s lists.

You’re also going to need an additional model to store the exceptions – additional people to add to lists, either in yesmail or nomail mode:

class ListExtra(models.Model):
    """
    Extra email addresses to be added to mailing lists.
    """
    list = models.SlugField()
    addresses = models.TextField(blank=True,
         help_text='Add addresses here, one per line.')

    def __unicode__(self):
        return u'%s' % (self.list)

Once the model has been created and you’ve run syncdb, create a record for each list you want to track, e.g. “teachers” for the list “teachers@ourschool.org” .  Do not enter real teachers here – they’ll come from ORM queries on the regular User table.  Instead, this is where you’ll add non-teachers  who should be regular members of the teachers@ mailing list anyway. Add such people’s addresses to the field one-per-line.

If you have the need to let certain people write to the teachers list but NOT receive mail from it, create an additional record called “teachers-nomail”.

How It Works

This recipe is really two scripts – one in Python and the other bash – that I end up running from a single shell alias as needed. The two scripts I use can be downloaded at the end of this post and are well commented.

The Python script (listgen.py) interacts with the Django ORM, extracting list memberships out into text files. The bash script (listgen.sh) iterates through that set of text files and, for each, quickly unsubscribes the entire membership and then re-subscribes them from the text files generated by the python script. On my server, the whole process unsubs and resubs 14 lists covering 125 people in  5 seconds flat, so there’s no real danger of anyone freaking out because they were temporarily unsubscribed.  And  listgen.sh makes sure that subs and unsubs occur without notifying users of list membership changes, so they don’t get scary messages.

listgen.py sets up the Django environment for the given project, defines a function for processing query results into text files, then iterates through a list of mailing list names (without the domain name). For each, it runs a Django ORM query. Then it excludes from that queryset anyone who has opted out of all lists.

Next, it looks for a corresponding ListExtras record and extracts the “extra” addresses it contains. Finally, it looks for a corresponding “nomail” ListExtra record (e.g. teachers-nomail). It then removes any duplicate members and passes the final query data to the file processing function and writes text files to a directory on the server.

listgen.sh iterates through an array of all mailing lists and, for each, unsbuscribes all members (without sending unsub messages). It then locates the text file corresponding to the current list and subscribes all regular members in yesmail mode. Finally, it looks for a corresponding listname-nomail.txt file and subscribes the addresses it contains in nomail mode.

Deployment

The scripts I’m using are linked below.

First, check to see whether the version of add_members on your server supports the -e flag (for nomail mode). If it does not, rename the add_members script on your server to -old and put the replacement add_members script in its place. Run “locate add_members” if not sure where to put it. If it complains when you try to run it, make sure its first line matches the python path shown on other scripts in the same dir.

Put listgen.py in a “scripts” directory in your Django project.

Put listgen.sh somewhere where root can get to it.

You’ll need to modify the list of mailing lists in each script, as well as the most important bit – the Django ORM queries in listgen.py. This will probably take quite a bit of testing and experimentation – make sure you’re working with test data until everything is humming – users will get righteously pissed if you break their lists.

You’ll also need to make sure the two scripts agree on the filesystem location for the output  (for the generated files that listgen.py creates and listgen.sh reads).  By default this will be a dir at myproject/scripts/listgen. You’ll need to edit the python path vars at the top of listgen.py.

During testing you’ll probably want to just run listgen.py and not listgen.sh — then you can study the text file output without actually altering the lists. When you’re confident listgen.py is doing the right thing, add the listgen.sh step.

Once it’s all working, create an alias for root that runs both scripts, something like:

alias schoollist='python /path/to/scripts/listgen.py; \
/bin/sh /root/scripts/for_customers/school/listgen.sh'

You could set this up to run as a cron job, but there’s a good reason not to: Mailman keeps track of bad addresses with its bounce processing feature, which counts the number of times a message has bounced from a bad address and then unsubscribes that user and alerts the list administrator. If you run listgen.sh every night, every member will always be “new” as far as Mailman is concerned and bounce thresholds will never be reached.  You’ll never be informed when/if you have bad addresses in your Django system. To address this, I wrote a Django signal handler that sends me email whenever a User object has been updated. I then run the schoollist command whenever I know something has changed (after the system stabilizes this shouldn’t happen very often).

The Scripts

add_members (copied from OS X Server’s Mailman distro, but works on any platform)

listgen.py (put in a scripts dir inside your Django project)

listgen.sh (put somewhere root can access easily)

Designed as a “pluggable” application ready to import into existing Django projects.

You must already have a  login/authorization system installed and working in your project before installing django-todo.

This version supports drag and drop task prioritization, email notification to task assignees, and lots more. Feedback welcome.

The code is hosted on Google Code and can be checked out via svn.

Screenshots

Download Cleancats

Instructions: Unpack this folder to your plugins directory and activate it. Click on the Dashboard and you’ll see a new tab called “NGNO Docs.” Within that, you’ll find documentation I wrote for my authors for one particular site. In order for this to be useful, you’ll need to replace the HTML it contains with your own by editing the file docs.html.

You’ll want to remove the image files I’ve included and, if desired, add your own. Note: I found it tricky to get image paths working – you’ll need to embed images like this:

img src="../wp-content/plugins/wp-workflow-docs/more-result.gif"

You can change the plugin name and tab name by editing the plugin file itself – it’s a tiny plugin and it will be pretty obvious where to do this in the file. Then de-activate and re-activate it to see the new tab name.

Download wp-workflow-docs

Contributions / improvements welcome!

Instructions: Place future-post.php in your plugins directory and activate it. Write a post with a future timestamp and hit publish. Notice that it goes live on your site immediately.

Note: This seemingly simple plugin was graciously written by the magical Ryan Boren when I was facing a deadline. He doesn’t have time to maintain/host it, so I agreed to.

Download version 1.0

Contributions / improvements welcome!

Note: I’m not likely to continue development of gpx2ipod, for two reasons: 1) MacCaching finally grew the ability to store cache metadata in the iPod’s “Notes” feature rather than in the address book (which was the whole impetus behind my developing gpx2ipod to begin with), and 2) I switched to a Garmin Colorado a few months ago, which displays description, logs, and hints directly in the GPS. I no longer need to offload that data to the iPod, so my itch has been scratched. Do with this software what you will!

Overview

Mac-based paperless caching for people who own an iPod but not a PDA.
Batch-converts a pile of .gpx files to plain text for use with the iPod’s “Notes” feature.
Super-fast — cut your geocaching prep time to a few minutes.

gpx2ipod handles both individual and Pocket Query (multiple-cache).gpx files.
Cache files will display alphabetically on the iPod for easy access in the field.

gpx2ipod can inject generated text files directly into your iPod (most users) or into a local “output” folder (you might not have an iPod but might still want the text files for other purposes).

gpx2ipod is a Terminal application (shell script), but can be run painlessly with a double-click — no shell experience required.

gpx2ipod is available either with or without the gpsbabel command-line binary bundled. gpsbabel is licenced under the GPL, and thus so is gpx2ipod. See LICENSE for info.

Who this is for

• You’re a Mac user who has an iPod but not a PDA.
• You want to do paperless geocaching.
• You want complete descriptions, log entries and hints while out on the trail.
• Your iPod has this great “Notes” feature, but you haven’t found any OS X software to get GPX files into a format you can read in the iPod’s Notes reader. MacCaching is wonderful, but it outputs to the iPod’s “Contacts” system, which means no logs and no choice of whether to encrypt hints. It also intermingles hundreds of geocache data files with your existing pile of Contacts – very messy. GPSBabel+ has a similar “Export to v-card feature” with similar problems. The iPod’s “Notes” feature seems a much better choice to me.

Requirements

Mac OS X. This script will not run on Windows or Linux (with modifications, gpx2ipod could be made to run under Cygwin or Linux).

A copy of the free and open source gpsbabel (version 1.3.4 or higher). Thanks to all contributors to gpsbabel for your hard work on this amazing workhorse! Huge thanks also to Robert Lipe, who added gpsbabel’s ability to split multi-location files into individual files, so gpx2ipod wouldn’t be stuck with the 99-split limitation in the Unix “cut” command.

An iPod with Storage Mode enabled. Storage Mode lets you use your iPod as a hard drive. If your iPod does not show up as disk volume on the Desktop when plugged in, go to iTunes and select “Enable disk use,” then click Apply.

Installation

Unzip this folder and drag it into your Applications folder (or wherever you like). Then drag the gpx2ipod folder icon from your Applications folder into your Dock for quick access.

• If you downloaded gpx2ipod with gpsbabel:You’re good to go! Skip to Usage.
• If you downloaded gpx2ipod without gpsbabel:gpx2ipod requires the gpsbabel command line application, (version 1.3.4 or higher). When you install GPSBabel (probably into your Applications folder), you’ll see two files: GPSBabel+, which is a graphical application, and gpsbabel itself, which is a command-line application. There are two ways to make gpsbabel available to gpx2ipod:The easy way: Select the file “gpsbabel” in that folder and copy it (don’t move it!) to the bin folder inside the gpx2ipod folder.The hard (but better) way: If you’re comfortable at the command line and prefer not to duplicate the binary, you can create a symlink instead, for example:

  ln -s "/Applications/GPS Babel/gpsbabel" /Users/[yourlogin]/Desktop/gpx2ipod/bin/gpsbabel

  Modify that command to match your setup. Note: Mac aliases won’t work – you’ll need a proper symlink.

Usage

1. Download .gpx files from geocaching.com and move or copy them into the “input” folder. You can mix and match individual .gpx files and Pocket Query (multi-cache) files in the same input folder. gpx2ipod includes a “samples” folder for testing – copy some or all of these into the “input” folder and let ‘er rip.
2. Double-click the gpx2ipod “squid” and let it do its thing.
3. • If this is the first time you’ve run gpx2ipod, you’ll be asked to set a few preferences, such as whether you want to include log files, whether to encrypt hints, whether to copy the original .gpx files into an “archive” folder, and (most importantly) the system path to your iPod (the script will let you choose your iPod from a list). For the yes/no questions, use 0 for “no” and 1 for “yes.”
   • If the gpsbabel binary can’t be found, or is an old version, the script will tell you and then halt.
   • If the iPod you selected during Setup can’t be found, you’ll be given the option to mount it now, or, alternatively, to have generated files sent to an “output” folder instead.
4. When everything’s in order, gpx2ipod will crunch through all those .gpx files, spit out text files optimally formatted for iPod use, and copy them into the Notes folder on your iPod. On future runs of gpx2ipod, your last-used preferences will be displayed as the script launches. Hit [Return] to accept the settings, or C to change them.
5. Go geocaching! You’ll find the cache entries on the iPod in Extras > Notes.

Limitations

This version does not load .gpx files into your GPSr – it just generates text files for use on the iPod. A future version may load the same files into the GPSr.

History

v1.3 – 9/15/2007
Now retains character formatting for international character sets. Default is UTF-8, but any character set supported by gpsbabel can be selected in preferences. Tested against Swedish .gpx samples (see GCY7XZ-Swedish_chars.gpx in Samples folder). Many thanks to Bengt Bäverman for excellent contributions on this feature.

v1.2 – 9/15/2007
Bug fix for other cache name characters that would generate illegal filenames.

v1.1 – 9/08/2007
No longer generates errors when encountering caches with slashes in their names. Now works properly when installed in a path containing a space (such as “/Applications/GPS Apps”).

v1.0 – 8/03/2007
Renamed project to “gpx2ipod.” Now requires gpsbabel 1.3.4 or higher, taking advantage of new gpsbabel “splitoutput” feature in order to overcome the cut command’s 99-split limit (thanks Robert Lipe!). Now double-clickable – much better user experience. Now copies generated files directly to iPod. Now stores user preferences in a config file. Now optionally copies original .gpx files into an “archive” folder for future use.

v0.2 – 7/17/2007
Now handles both individual .gpx files and Pocket Query (multiple-cache) .gpx files.

v0.1 – 7/14/2007
Original version

Download

Download without GPSBabel

Download with GPSBabel