Django Unit Tests Against Unmanaged Databases

A Django project I’m working on defines two databases in its config: The standard/default internal db as well as a remote legacy read-only database belonging to my organization. Models for the read-only db were generated by inspectdb, and naturally have managed = False in their Meta class, which prevents Django from attempting any form of migration on them.

Unfortunately, that also prevents the Django test runner from trying to create a schema mirror of it during test runs. But what if you want to stub out some sample data from the read-only database into a fixture that can be loaded and accessed during unit tests? You’ll need to do the following:

  • Tell Django to create the second test database locally rather than on the remote host
  • Disable any routers you have that route queries for certain models through the remote db
  • Tell Django to override the Managed = False attribute in the Meta class during the test run

Putting that all together turned out to be a bit tricky, but it’s not bad once you understand how and why you need to take these steps. Because you’ll need to override a few settings during test runs only, it makes sense to create a separate test_settings.py to keep everything together:

from project.local_settings import *
from django.test.runner import DiscoverRunner


class UnManagedModelTestRunner(DiscoverRunner):
    '''
    Test runner that automatically makes all unmanaged models in your Django
    project managed for the duration of the test run.
    Many thanks to the Caktus Group: http://bit.ly/1N8TcHW
    '''

    def setup_test_environment(self, *args, **kwargs):
        from django.db.models.loading import get_models
        self.unmanaged_models = [m for m in get_models() if not m._meta.managed]
        for m in self.unmanaged_models:
            m._meta.managed = True
        super(UnManagedModelTestRunner, self).setup_test_environment(*args, **kwargs)

    def teardown_test_environment(self, *args, **kwargs):
        super(UnManagedModelTestRunner, self).teardown_test_environment(*args, **kwargs)
        # reset unmanaged models
        for m in self.unmanaged_models:
            m._meta.managed = False

# Since we can't create a test db on the read-only host, and we
# want our test dbs created with postgres rather than the default, override
# some of the global db settings, only to be in effect when "test" is present
# in the command line arguments:

if 'test' in sys.argv or 'test_coverage' in sys.argv:  # Covers regular testing and django-coverage

    DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
    DATABASES['default']['HOST'] = '127.0.0.1'
    DATABASES['default']['USER'] = 'username'
    DATABASES['default']['PASSWORD'] = 'secret'

    DATABASES['tmi']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
    DATABASES['tmi']['HOST'] = '127.0.0.1'
    DATABASES['tmi']['USER'] = 'username'
    DATABASES['tmi']['PASSWORD'] = 'secret'


# The custom routers we're using to route certain ORM queries
# to the remote host conflict with our overridden db settings.
# Set DATABASE_ROUTERS to an empty list to return to the defaults
# during the test run.

DATABASE_ROUTERS = []

# Set Django's test runner to the custom class defined above
TEST_RUNNER = 'project.test_settings.UnManagedModelTestRunner'

With that in place, you can now run your tests with:

./manage.py test --settings=project.test_settings

… leaving settings untouched during normal site operations. You can now serialize some data from your read-only host and load it as a fixture in your tests:

class DirappTests(TestCase):

    # Load test data into both dbs:
    fixtures = ['auth_group.json', 'sample_people.json']

    ...

    def test_stub_data(self):
        # Guarantees that our sample data is being loaded in the test suite
        person = Foo.objects.get(id=7000533)
        self.assertEqual(person.first_name, "Quillen")

Stranded Sea Lion, Angry Poison Oak, Ukulele Maiden

Huge day out with friends yesterday, along Coastal Trail to Alamere Falls (Pt. Reyes). Poison oak in full bloom. Recent stories about sea lion pups washing ashore, disconnected from their mothers who are out foraging for food made scarce by warming waters turned all too real when we encountered one, grumbling for a meal. Added another couple-three miles to route (for a total of 11?) as we walked up the beach looking for another access point to the trail above. Perfect weather, huge vistas, maltey barley wine enjoyed on a driftwood log after lunch. Great company, gorgeous day. Life is good.

Images in the Flickr set:

Alomere Falls - Steve & Andrew

Please Don’t Text Me

In the olden days, a typical worker’s desk had an “inbox” and an “outbox.” The “inbox” represented things that needed to be dealt with, and the “outbox” represented things that were done. When email came along, its designers wisely emulated this metaphor.

onedoesnotsimply

Your email inbox represents everything you haven’t dealt with yet, but that needs to be. While managing your email, you’re engaged in an ongoing process of deleting things you don’t need to ever see again, or archiving things that have been dealt with but need to be kept for reference. If it doesn’t need to be dealt with, it has no excuse to exist in your inbox. At the end of every day, what’s left in your inbox is the (hopefully very small) list of things you haven’t gotten around to. But you know they’ll still be there tomorrow – they won’t be lost. Your inbox is, in essence, the most important on-going to-do list you’ve got.

Text messaging apps have no such concept. When a text is new, you get an alert. But the moment you glance at it, there is no mechanism for separating it out from all of the thousands of other texts piled up in your app – it becomes part of the noise. There is no way to know what in your text app needs responding to and what does not.

Therefore, when you send me a text, I have two choices:

  1. Drop everything and respond right now so your message doesn’t get forgotten
  2. Add your message to the “mental stack” of things that need to be dealt with later.

Most of the time, when a new text rolls in, I’m not able to deal with it right now. Ipso facto, most of the time, when a new text rolls in, it’s bound to get forgotten – I’ll never see it again. Unless I add it to my mental stack, i.e. unless I incur a cognitive burden.

Case in point: A few days ago a text rolled in while I was on a long bike ride, asking for information I didn’t have access to at the time. When I arrived home six hours later, that text was the absolute last thing on my mind. It was gone, virtually forgotten. There was nothing to remind me that it ever existed. If it had been an email, the fact of it existing in my inbox would have ensured that it got the response it deserved. The sender had simply chosen the wrong tool for the job.

Because of this reality, when you send me a text, you are putting a burden on me. You are saying, “Drop what you’re doing and respond to me right now, regardless whether it’s convenient for you, lest this communication be forgotten.”

When you email me, you’re saying “Please respond to this when the timing is convenient for you.” With email, I have the luxury of being able to delay my response a day or two if needed. There is no cognitive burden – I don’t have to remember to respond. I’ll know to respond later, because your message is there in my inbox.

So in what occasions is a text more appropriate than email?

  • We’re arranging details about something that’s happening now or in a few hours
  • You just want say hello or share something simple that doesn’t demand an immediate response
  • Computer is on fire.

If you’re planning something that is not happening today, please don’t text. If you’re communicating important information, that needs real typing to work out, please don’t text. If you’re communicating information I might want to be able to refer to later, please don’t text.

I know there’s a lot of talk about how “email is dead” and “email belongs to the old,” and about how some young people actually prefer text over email. I say it’s not about youth – it’s about respecting people’s time, regardless of age (and everyone is busy!). Also: Claims about the death of email are grossly exaggerated – for me and millions of others, email is still the centerpiece of online communication.

I’m not asking you never to text me. I’m asking to ask yourself whether what you have to say rises to the level of deserving a time-stealing text.

Photo365 for 2014

At the start of 2014, I made it my New Year’s resolution to take at least one photo per day for the year. I had done the project once before, in 2011, after a suggestion by the amazing Richard Koci-Hernandez. The goal is to keep your photographic “eye” always open. That’s easy when you’re traveling or out having adventures, much more difficult through an ordinary workday, treading the same old offices and streets. But it’s amazing how things just seem to “turn up” when you have an eye out for possibilities.

It’s also a fantastic way to end up with your own “year in review” – really fun to walk back through some of the year’s best memories.

Watch the embedded slideshow here (full-screen please!) or check out the Flickr set

Taming a Mammoth Music Collection

Whether you’re talking about LPs or MP3s, people have really different ideas about what constitutes “the ultimate music collection.” For some, it means a process of endless refinement, boiling down a set of music to the purest essentials: All signal, no noise. For others, it’s an archival process (“Why have one Bix Biederbeck CD when you could have 23?”)

record-collection

It’s possible to have the best of both worlds: Maintain a large collection so you have access to everything, but create a playback system so you only end up hearing what you truly love.

I’ve been an eMusic subscriber for nearly a decade. I’ve spent a good deal of my spare time over the past four years digitizing my entire record collection, followed by my entire CD collection, followed by the large CD collections of six record-collecting friends (one of which alone was basically the Musical Library of Alexandria). All told, I’ve managed to amass a collection of ~120,000 tracks spanning ~9,100 albums, mostly in lossless format, and all with high-quality album art.

collection2

The accreted set now weighs around  2.25 terabytes  – large enough to have “special needs.” Over the past four years of building the collection, I’ve  picked up a few tips. Thought I’d share some of the most useful bits here,  in case anyone finds them helpful.

Love it or hate it, iTunes has enough traction to be considered the “default” music player for almost everyone, so I’m going on the assumption that it’s your player too. If you use something else, power to you! Everything below assumes you use iTunes 11 or 12.

This guide is split up into four major sections:

  • Remote Control (Playback techniques)
  • Miscellaneous iTunes Tips (Rare B-sides)
  • Digitization Notes
  • Building a Server

Continue reading

How To Keep Your Volkswagen Alive

Adjusting your own valves will not only change your relationship with your car, it will change your  relationship with yourself.

Muir Volkswagen - 1 - CoverThis book kind of changed my life. In the early 90s, I spent all my money in the world on a 1974 air-cooled VW Squareback, similar to the one I grew up in. A few weeks later, cylinder #3 seized up (cyl. 3 was famous for that). While I was kind of freaking out, my Harley-riding/building housemate calmly urged me to “Quit complaining. Drop the engine and fix the damn thing.”

That sounded impossible at the time, but what choice did I have? Went and got a copy of John Muir’s classic How to Keep Your Volkswagen Alive: A Manual of Step-by-Step Procedures for the Compleat Idiot and got lost in its pages for a week. With its Robert Crumb-style drawings and groovy prose style, it sucked me into a world of mechanical competence I previously couldn’t have imagined for myself.

Muir Volkswagen - 2

What astounded me about the process was the power of incremental demystification. It’s not until you remove the valve cover and see for yourself how the throw rods connect to the rocker arms connect to the valve springs that you have that “Ah ha!” moment, and all of the mystery of internal combustion suddenly starts to make sense. Every part you remove strips away one more layer of skin from the onion, revealing previous mysteries as simple mechanical truths. It’s an amazing experience.

Muir Volkswagen - 3

The process of fixing that car ushered in a few years of wrenching around for fun – first on the Squareback, and later on a 1964 bus that became the love of my life. I don’t do that anymore – I never took it beyond the simple mechanics of the air-cooled engine and into modern computer-controlled stuff. But the sense of empowerment that came from having gone through it lasted my entire life.

Muir Volkswagen - 5

For my 50th birthday, bought myself a used copy of the original book, and have been leafing through it at random, reliving great memories from that period of my life. So grateful.

Muir Volkswagen - 6

Muir Volkswagen - 4

 

Blocking Malicious Bots

Over the past few months, we’ve watched as customer sites at Birdhouse Hosting seemed to hit their monthly bandwidth allotments sooner and sooner. At a certain point, it became obvious that this could not be explained by upticks in popularity – upon closer study of awstats logs, it became apparent that a great deal of that traffic was coming from malicious bots.

And the traffic was not just attempts to post spam into weblog comment forms either – this was traffic on images, random pages, RSS feeds, PDFs, everything.

A few days ago, a new suite of ModSecurity rule management tools landed in cPanel (cPanel is the hosting platform I use to run Birdhouse). I went looking for mod_sec rules intended to curb bad bot traffic, and seem to have hit the jackpot with a rule that consults the spamhaus Malicious Bot RBL. And because it’s installed globally, it protects all of my customer sites simultaneously. Here’s the rule I used (all on one line of course):

SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "phase:1,id:'981138',t:none,pass,nolog,auditlog,msg:'RBL Match for SPAM Source',tag:'AUTOMATION/MALICIOUS',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.automation_score=+%{tx.warning_anomaly_score},setvar:tx.anomaly_score=+%{tx.warning_anomaly_score},setvar:tx.%{rule.id}-AUTOMATION/MALICIOUS-%{matched_var_name}=%{matched_var},setvar:ip.spammer=1,expirevar:ip.spammer=86400,setvar:ip.previous_rbl_check=1,expirevar:ip.previous_rbl_check=86400,skipAfter:END_RBL_CHECK"

Over the past 24 hours it’s blocked  over 150,000 requests by bad bots to all of my customer sites. Absolutely incredible.

I’d  like to thank the fine folks  at spamhaus for doing what they do, and for helping to make the internet a better place – for free!

The Spamhaus Project is an international nonprofit organization whose mission is to track the Internet’s spam operations and sources, to provide dependable realtime anti-spam protection for Internet networks, to work with Law Enforcement Agencies to identify and pursue spam and malware gangs worldwide, and to lobby governments for effective anti-spam legislation.

 

Hatari

This version of tUnE-yArDs “Hatari” doesn’t get nearly enough love. Better than the version recorded on BiRd-BraInS, IMO. Just the right mix of low-tech avante and sophisticated. Merril Garbus, as always, a force of nature.