scot hacker's scripts and utils

Users of webmail systems like GMail, Yahoo, etc. are accustomed to having a “Mark as Spam” button in the interface. Clicking the button tells the server that the selected message is spam, to prevent similar messages from showing up in the inbox again. So how can administrators of standard cPanel-based hosting systems provide similar functionality?

I’ve written up some techniques you can use to let desktop mail client users train server-side mail systems about what’s spam and what’s ham.

Update: I’ve since stopped trying to train Bayes databases through this system, as I’ve discovered the incredibly effective Barracuda RBL. I’ve written that up here.

Alex Gaynor’s Django-Taggit is a great piece of software, but was missing two features essential for Bucketlist:

1) Users could create tags “Music” and “music” which would create two logically different taxonomies on the site, which was confusing and messy. I’ve added a settings option to force lowercase. When this is enabled, all tags will be lowercased as they’re submitted, preventing duplication.

2) Many users don’t read instructions, and enter multi-word tags without quotes, which results in the system being polluted with lots of tags “the”, “of”, “an” etc. You can now define a set of “stopwords” in settings. Any detected stop words will be removed from the submitted tag set.

I forked Gaynor’s taggit to add these features. See shacker/django-taggit.

I also wrote a script to fix all of the duplicate tags that had gone into the system after installing the original django-taggit but before doing my forked version. You’ll find that script on the repo’s Wiki page.

A quick Python scraper script which takes a page URL as an argument, then locates all MP3 links on that page and downloads them to the current directory. Requires BeautifulSoup and Mechanize.

To run:

cd somefolder
python /path/to/scrapemp3s.py http://example.com/music.php

Download scrapemp3s.py

SoloMail is a WordPress plugin designed to email an HTML-formatted copy of single posts (not batch digests) to site subscribers. There are quite a few solutions out there designed to send regular digests of recent posts to subscribers. SoloMail solves a different problem – you may want to cherry pick just certain posts to go out via email, and you may want the ability to “send now.”

The subscriber list can EITHER be the group of all registered users on a site OR a single address, such as a mailing list. Though you can use SoloMail either way, I recommend sending through a proper mailing list, which provides advantages such as handling and unsubscribing bounced/dead addresses, and metered sending for large lists.

Please see the included readme.txt for configuration and usage instructions.

Download SoloMail

For sites with multiple authors and editors, there is often a need to provide a style guide and/or technical instructions for site contributors. This very simple plugin will create a “Site Notes” menu item in the WP Dashboard. Clicking it will take the user to a page or set of pages. The administrator simple creates the site documentation in HTML format and stores it in the plugin body itself. I know, not very elegant – this is meant to be quick-n-dirty – but gets the job done.

Create a file called site-docs.php or similar with the following contents, upload it to your plugins directory, and activate. That’s all there is to it.

In my example, we just use in-page anchors to separate the different documentation sections, but you can do whatever you like. Season to taste.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

<?php
/*
Plugin Name: Site Documentation
Plugin URI: http://birdhouse.org/software
Description: Documentation for site authors and editors
Version: 1.0
Author: Scot Hacker
Author URI: http://birdhouse.org/
License: GPL
Last modified: 2007-05-05 9:54pm EDT
*/
 
// mt_add_pages() is the sink function for the 'admin_menu' hook
function mt_add_pages() {
    // Add a submenu to the Dashboard:
    add_submenu_page('index.php', 'Stuck Notes', 'Stuck Notes', 2, 'stuck-notes', 'stuck_notes');
}
 
 
 
// stuck_notes() displays the page content for the Stuck Notes submenu
function stuck_notes() {
echo <<< EOF
<p><a name="top" id="top"></a></p>
 
<div class="wrap">
	<h2>Stuck Notes - Docs and tips for writers/editors</h2>
 
	<div class="wrap">
		<ul>
			<li><a href="#process">Review process</a></li>
			<li><a href="#style">Editorial style notes</a></li>
			<li><a href="#contrib">...</a></li>					
		</ul>
	</div>
 
	<p><a name="process" id="process"></a></p>
 
	<div class="wrap">
		<h3>Review process</h3>
 
		<p>...</p>
 
		<p><a href="#top">Back to top</a></p>
	</div>
 
	<p><a name="style" id="style"></a></p>
 
	<div class="wrap">
		<h3>Style notes</h3>
 
		<p>...</p>
		<p><a href="#top">Back to top</a></p>
	</div>
 
</div>
 
EOF;
}
 
// Insert the mt_add_pages() sink into the plugin hook list for 'admin_menu'
add_action('admin_menu', 'mt_add_pages');
?>

WordPress Mass Management Tools is a small collection of tools for managing multiple WordPress installations on a Unix/Linux/Mac server (or perhaps a Windows server with Cygwin (untested)).

Yes, users can often self-install via Fantastico or similar programs, but what guarantee do you have that they’ll upgrade as soon as new releases become available? Letting users run old versions of web software is a great way to get hacked. This kit lets you take control of users’ installations by checking them out via svn and upgrading them en masse.

At Birdhouse Hosting we use WordPress Mass Management Tools to create all new installations. When WP updates are released, we’re able to upgrade more than 100 WP installations belonging to 80+ users in a few minutes flat.

The kit is designed to be run by root – these are not tools for WordPress end users. The kit assumes basic knowledge of the bash shell.

This kit replaces the old wp-create and wp-mass-upgrade scripts which were distributed separately and are no longer supported. Includes:

wp-create.sh: Super fast way to install WordPress for clients, via subversion. Performs the following tasks:

• Gather installation info
• Create install dir and check out a copy of WordPress
• Create database, db user, set db privs via external .sql file
• Create WP config file
• Create upload dir and set filesystem permissions
• Generate array line for wp-sites.sh

Final setup is done via browser.

wp-mass-upgrade.sh: Iterates through all sites listed in wp-sites.sh, backs up the site’s database, upgrades to a specified version of WordPress, and sends the site owner email announcing the upgrade.

wp-mass-plugins.sh: Generates a list of all active plugins used by all sites on the server. The list can be compared to lists of known-incompatible plugins to help you make decisions about whether to skip any sites on the list.

Download wp-mass-tools 1.1

See the included readme.txt for documentation.

Want to contribute/make this project better? It’s also available at github.

I read Twitter primarily on the iPhone, and find tons of great links I want to read in a proper browser later on (I personally find reading most web sites on an iPhone to be more hassle than it’s worth). Perfect solution: Side-swipe an item in Tweetie and tap the star icon to mark it as a favorite. Later, visit the Favorites section at twitter.com to follow up.

Unfortunately, over the past couple of years I’ve favorited way more things than I’ll ever have time to read. As of now, I’ve got 1600 favorites waiting to be read. Ain’t never gonna happen. I declare Twitter Favorite bankruptcy! Needed a way to batch-unfavorite the whole collection, and twitter.com doesn’t provide a tool for that. Time to dive into the Twitter API.

Recently I was invited to participate in the California Data Camp and DataSF App Contest hosted by California Watch and spot.us. The unconference would feature lots of discussion about making use of publicly available data sets to improve quality of life. The App Contest challenged developers to choose one of the many data sets available at DataSF.org and build something cool with it in a relatively short period of time. Here’s a showcase of existing apps built on those data sets.

The Knight Digital Media Center (where I work) invited me to take part, and I chose a database of 64,000 San Francisco trees and plants. The goal of the project was to:

• Make it easy for citizens to explore and discover the huge number of plant species and individual trees maintained by the city
• Make it easy for citizens to “flag” a tree as needing maintenance, water, food, etc.
• Make it easy for citizens to request a tree at a particular location
• Provide data visualization tools to let citizens explore and understand the plant variety visually
• Make it easy to see what a given species will look like in 5,10,15,20 years when requesting a tree
• Ideally, a future version of the app would include ecology data on all species, listing the water consumption and carbon offset of each

I decided to build the project on Django, of course. Put a total of around 15 hours into the project, about half of which was spent massaging and cleaning the provided data, which had multiple pieces of information stuffed into single fields, non-standard date formats, and was completely non-relational. Cities implementing django-treedata “fresh,” without having to be compatible with an existing data entry system, won’t have to worry about data conversion/format issues.

Once the data was clean, the rest was pretty straightforward Django stuff. The one non-standard aspect is the external “lastcount” script, which counts the number of instances of each species and stores the result on a field in the Species model. Doing this in real time for such a large number of trees turned out to be very computationally expensive, so the script needs to be run from a crontab periodically.

Because dev time was so limited, all of it went into data cleaning and building out the models and views. We’ve put ZERO work into design considerations, so please don’t crucify us for that. The CSS is built on top of the excellent 960 Grid framework, so layout will be easy. Some of the data visualization is done via the excellent Google Charts API.

Much to our surprise, the django-treedata app won the competition!

Please note that the project has only been run in a development environment and has never been publicly deployed – the project as it stands should be considred a starting point for cities to built on. The readme explains more. The project is completely open source and is released under the very liberal BSD license – do with it as you will.

Thanks also to J-School webmaster for Chuck Harris for his contributions to the project

Django projects can end up with complex sets of Users, Groups, and multiple profile types representing different types of people. For example a school site might have Students, Parents, Teachers, Staff, and Alumni. The mailing lists for that school will live completely outside of the Django project, but there’s a good chance you’d like to be able to populate list membership from your membership database rather than maintaining lists by hand. And you’d like to be able to use any combination of criteria to populate your lists (Group membership,  profile types, join date, privileges, etc.)

Since Mailman is installed on so many web hosts by default, there’s a good chance you’re using it, and have lots of overlapping and non-overlapping groups subscribed to various lists. I recently went through the process of integrating a bunch of Mailman lists with the membership representing a school intranet built in Django and thought I’d document it for anyone going through a similar process.

Once everything is set up, you’ll never need to use the Mailman interface to manage lists again – everything will be automated and self-maintaining.

django-todo is a multi-user, multi-group task management and assignment system for the Django web application framework.

Designed as a “pluggable” application ready to import into existing Django projects.

You must already have a  login/authorization system installed and working in your project before installing django-todo.

django-todo supports drag and drop task prioritization, email notification to task assignees, search, and lots more. Feedback welcome.

The code is hosted on GitHub and can be downloaded or checked out via git. The package also includes a setup.py, and can be installed with pip.

This is a very crude mass category deletion script for WordPress. I recently inherited a WordPress site with more than 6000 categories. After deleting a ton of old posts, we were left with more than 4000 unused categories, and I needed a way to remove them quickly. Couldn’t find a plugin up to the task, so wrote this. Very crude, but effective. Not a plugin, but a small WP companion script. See instructions in the comment at top of script.

Download Cleancats

WP-Create has been folded into the WordPress Mass Management Tools kit.

A dirt-simple WordPress plugin that lets you provide workflow docs to your authors and editors from within the WP admin interface.

A WordPress plugin aimed primarily at events sites, where you want to be able to timestamp posts in the future but have them appear immediately (by default, WordPress will not display a future timestamped post until its go-live date rolls around). This plugin sets the post_status field to “publish” rather than “future” when publishing a post, even if its timestamp is in the future. Written by Ryan Boren and modified by Andrew Nacin – I’m just hosting it.

Designed for Mac users with an iPod but no PDA who want to do paperless geocaching complete with descriptions, log entries and hints. Uses the iPod’s “Notes” feature to display complete cache description pages. gpx2ipod is built on top of gpsbabel.

Note: I’m not likely to continue development of gpx2ipod, for two reasons: 1) MacCaching finally grew the ability to store cache metadata in the iPod’s “Notes” feature rather than in the address book (which was the whole impetus behind my developing gpx2ipod to begin with), and 2) I switched to a Garmin Colorado a few months ago, which displays description, logs, and hints directly in the GPS. I no longer need to offload that data to the iPod, so my itch has been scratched. Do with this software what you will!

Designed for hosts with lots of individual WordPress blogs to manage. If all blogs are managed via subversion, this script can iterate through all of them and upgrade each to the latest version in seconds.

WP-Mass-Upgrade has been folded into the WordPress Mass Management Tools kit.

Sends periodic email notifications of new WordPress blog entries (in either plain text or HTML mode) to a separate subscribers mailing list. Maintains cache file recording ID of last-sent post. On subsequent runs, sends all posts created since the previous run.

Currently driving weekly email updates from birdhouse.org and other sites.

Sends email announcements to your MT Notifications list or to an external mailing list on recent Movable Type weblog entries.

Harvests bad IP addresses from multiple sources and adds them to the ConfigServer firewall (CSF). This version works with WordPress and Movable Type weblogs, and optionally the exim ACL deny list. Future versions will scan other sources for bad IPs as well.

Download reducer

This is a script for WHM/cPanel administrators to help them keep tabs on client mailboxes approaching or exceeding their quotas. Sends warning messages to owners of near-quota mailboxes and a separate summary message to postmaster or other address. Must be run as root, or via root crontab.

Download checkmailquota

vim has a bad habit of chewing up a ton of CPU if user backgrounds it, closes their terminal window, or gets disconnected from the net with a vim session open. The vim reaper checks whether owner of running vim processes are logged in. If not, the process gets reaped.

Download vimreaper

For users of Apple’s QuickTime Streaming Server who need to perform batch-processing of QT movie files. Adds metadata, adds hint tracks, and generates .qtl reference movies, then moves everything to a destination directory. Requires the qtmedia and qtref command-line binaries only present in OS X Server.

Download makestreams

A quick-n-dirty shell script to rename comment and trackback scripts across a Movable Type site. For emergency use when a server is undergoing a spam blitzkrieg and server performance is suffering. Written for syadmins dealing with high loads on MT-heavy servers.

Download comstop

WPBlogMail is now deprecated. Please use WP-Digest instead.

This is a very simple contacts database written in PHP/MySQL as a companion to a November 2002 article I wrote for MacWorld: Serve It Up. OS X and its Unix underpinnings were new, and MacWorld wanted to run articles showcasing the Mac’s entry into Unix-side server capabilities at the desktop level.

MacWorld no longer links to this sample application, but it should still be useful to anyone wanting to learn PHP/MySQL with a simple (but useful), well-commented starter application.

Download contacts.tgz

In the BeOS glory days, the RipEnc shell script was the premiere MP3 ripping/encoding solution, and made full use of the database-like BeOS filesystem to store MP3 metadata. After I left the BeOS world, others took over the script. The current version is available at BeBits. The version here is the last one created and maintained solely by me.

Two downloads are available – a .pkg installer for people still running BeOS/Haiku, and a text version for people without access to BeOS but curious about the command line tools BeOS provided to manipulate the filesystem database.

Download ripenc_26a.pkg

Download ripenc_2.6a.txt