Scripting with the Be File System
Automating the native BeOS database
Scot Hacker, March 2001I've written before about Be's unique filesystem and its database-like properties. This month I'd like to talk about ways you can put Be's native database to work for you on a deeper level -- by working with BFS queries and attributes from the command-line -- or from within scripts -- to create custom solutions.
For those of you just joining us, files on a BFS volume can store arbitrary amounts and kinds of associated meta-data, in the form of "attributes." Typically, attributes are associated with specific filetypes. For example, all BeOS People files can take Address, Phone Number, State, Country, Email, etc. as attributes of those files. Attributes can be viewed and sorted on in the Tracker, and can be queried for with the system's Find tool, just like you would query any database. Attributes can be added to existing filetypes, or modified from the FileTypes preferences panel.
The BeOS contact database consists of 0-byte files called People files. All People data is stored in attributes of those files. We're going to turn this collection of files into an HTML Rolodex by using Be's command-line BFS tools. Click for larger version.
Because many attributes are indexed, most queries are lightning-fast. So not only is it possible to run queries on BeOS that just aren't possible with other operating systems, but attribute queries do not force the operating system to iterate over every file in the search path.
Pulling Oysters from the Shell
There are plenty of ways in which the BFS database makes life easier for the average end user, but it gets even better for those comfortable working with the bash shell and basic Unix-style shell scripting.Be has provided a collection of tools for extracting attributes and running queries from the command-line. Let's take a look at those tools one at a time, culminating in the creation of a custom solution -- we're going to publish an arbitrary collection of BeOS People files as an HTML "Rolodex" which you can place on any web server, so you can retrieve important contact info without having direct access to your BeOS machine.
Mixed right in with all the other POSIX command-line tools in /boot/beos/bin are the following tools provided by Be, designed to interact with the special properties of the Be File System:
listattr catattr addattr rmattr query lsindex mkindex rmindex settype
listattr poots forth a list of all the attributes attached to a given filename -- even attributes not visible from the Tracker. For example, here's what I get when running listattr on the People file representing Walter Mitty:
listattr "Walter Mitty"
file Walter Mitty
Type Size Name
---------- --------- -------------------------------
0x52415754 20 _trk/pinfo_le
MIME str 21 BEOS:TYPE
Text 1 META:email2
Text 15 META:nickname
Text 15 META:name
Text 17 META:email
Text 19 META:address
Text 15 META:city
Text 8 META:state
Text 8 META:zip
Text 7 META:country
Text 13 META:hphone
0x42595445 1 META:allow_HTML
0x42595445 1 META:favorite
Text 3 META:group
The datatype of the attribute is in the left column, the size of the attribute in the middle column, and the attribute name on the right. For our purposes, we're really only interested in attributes with a datatype of "Text," although other applications can happily read and manipulate other datatypes as well.
What this list doesn't tell us is the stuff we really care about -- the actual contents of the attributes. For that, we use the catattr command, which dumps the contents of an attribute to stdout. Its usage is:
catattr attribute_name filenameIf I want to find out what city Walter lives in without using the Tracker, I would type:
catattr META:city "Walter Mitty"the shell returns:
Walter Mitty : string : Munster HamletThat's fine for a quick visual, but for the purposes of a shell script, I'm probably going to want just the last bit -- the name of the city. For this, we can use standard UNIX sed, cut, or awk tools:
catattr META:city "Walter Mitty" | cut -f3 -d:(This cuts all but the 3rd field from the string, where fields are delimited by a colon). We'll probably want to save the result of this into a variable for further use:
ThisCity=$(catattr META:city "$This Person" | cut -f3 -d:)The value of $ThisCity is now "Munster Hamlet" whenever the script is passing over the file "Walter Mitty." With small and fairly obvious variations, you can use the same process to extract any text attribute from any file on your system.
Other BFS Tools
It's just as easy to add or remove attributes from files. addattr and rmattr work pretty much as you would expect:addattr datatype attribute_name attribute_value filename rmattr attribute_name filenameThese can be useful if you find yourself wanting to make changes to the attributes of dozens or even thousands of files at once. For example, let's say you created a custom filetype with the type text/x-foobar, which has a custom attribute called Author (TEXT:Author). Now you've got a hundred FooBar files without an author attribute, and you want to have each one default to your own name. You'd use something like this:
addattr -t string TEXT:Author "Scot Hacker" *If you mess up or want to set them to something else in the future, use:
rmattr TEXT:Author *
Command-Line Queries and Index Manipulation
Command-line queries don't just duplicate the functionality of the BeOS Find panel -- they actually enhance it. For example, the BeOS Find panel will let you assemble fairly complex search criteria, but it only lets you search for those criteria in one filetype at a time. The command-line query circumvents that limitation.Unfortunately, the complete command-line query syntax is too involved to explain thoroughly here. Fortunately, I don't have to.
To generate a formula query, use the BeOS Find panel and construct a query to taste. When you're happy with the results, change the query type in the panel from Attribute to Formula and you'll see the query formula appear.
Copy this to the clipboard, open a Terminal, and type query, followed by a double quote. Paste from the clipboard, enter another double quote, and hit Enter. This formula query returns all of my contacts living in CA:
query "((META:state=="*[cC][aA]*")&&(BEOS:TYPE=="application/x-person"))"The list of results from this query can then be passed into other commands for further parsing or output to special formats (like the Rolodex we're going to build in a minute).
Earlier, I mentioned that many attributes are indexed, which is what makes BFS queries so dag fast. For performance reasons, not all attributes are indexed by default. To see the indexes on your system, type lsindex, or lsindex -l to get more details. To tell BFS to start indexing a particular attribute, use
mkindex datatype index_nameFor the example above, you might use:
mkindex -t string TEXT:AuthorIf you don't specify the datatype, it will default to type "string." If you're going to want to run numerical queries, be sure to set the datatype to an integer. For example, if you have an attribute for year and you're going to want to query on files written after 1967 but before 1983, you'll need to use something like:
mkindex -t int TEXT:yearNote that new indexes are not retroactive -- old files with the named attribute are not automatically added to the new index. If you want them to be, simply copy (don't move) the folder containing the files to a new location and then back again. The act of copying will cause the files' attributes to be written to the new index. To remove unneeded indexes, use rmindex.
Finally, you can set the MIME type of a given filename with the settype command. This is useful if your script is creating files which aren't immediately being given filetypes by the Tracker, or in case you want to override BeOS' default filetype identification routines. Usage is pretty much what you would expect:
settype mime_type filenameFor example, to set a newly created file to type text/html, use:
settype text/html newfile.html
About That Rolodex
While all of these tools are useful in their own ways, we only need a couple of them, combined with some standard shell commands and a few blocks of static HTML, to run a query against the BFS database, extract attributes for each line of data in the results, and interpolate that data with chunks of static HTML.For the purposes of this article, we'll keep the output as simple as possible, but you could easily extend these techniques to make a fairly sophisticated online contact database, or output the same data to comma-separated data for import into other 3rd-party utilities. In fact, I recently updated PeoplePorter to output data from the collection of BeOS People files into the Eudora for Windows address book format.
Save the section below as /boot/home/config/bin/rolodex and run rolodex from any Terminal. You'll end up with rolodex.html in your home directory. Note that this script is quite slow - a much faster version would extract all attributes at once and store them in an array. The simple version here is used for purposes of illustration.
Portions of the script using Be's BFS command-line tools are in red.
--------------------------------
#!/bin/sh
# Simple Rolodex in HTML format, generated from
# a collection of BeOS People files and using the native
# BeOS Filesystem database. We'll just query for Californians
# in this example.
# Location of people files:
PeopleDir=~/people
# Output the HTML header and top of document
echo '
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML LANG="EN">
<HEAD>
<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="TEXT/HTML; CHARSET=UTF-8">
<TITLE> California Contacts</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#46528A" LINK="#46528A" VLINK="#46528A" >
<CENTER>
<H2> California Contacts</H2>
<TABLE WIDTH=95% BORDER=0 CELLPADDING=5 CELLSPACING=0>
<TD BGCOLOR=#46528A><FONT COLOR=#D0D7BB SIZE=+1><B> Name </b></FONT></TD>
<TD BGCOLOR=#46528A><FONT COLOR=#D0D7BB SIZE=+1><B> E-Mail </b></FONT></TD>
<TD BGCOLOR=#46528A><FONT COLOR=#D0D7BB SIZE=+1><B> Home Phone </b></FONT></TD>
' > /boot/home/rolodex.html
# Start by running a query for all Person files which have
# a META:State attribute including the string CA (Californians).
# Output results to temp location.
query "((META:state=="*[cC][aA]*")&&(BEOS:TYPE=="application/x-person"))" > /tmp/rolo
# Loop through each person listed in rolo, extract
# some of their attributes, and output to an HTML file
# We're also going to link the email addresses, for convenience.
# But we'll have to remove the stray space
# character from the start of the email first.
{
while read ThisPerson; do
ThisName=$(catattr META:name "$ThisPerson" | cut -f3 -d:)
ThisEmail=$(catattr META:email "$ThisPerson" | cut -f3 -d: | sed s@\ @@g)
ThisPhone=$(catattr META:hphone "$ThisPerson" | cut -f3 -d:)
echo "
<TR>
<TD>"$ThisName"</TD>
<TD><a href="mailto:"$ThisEmail"">"$ThisEmail"</a></TD>
<TD>"$ThisPhone"</TD>
</TR>"
done
# Use the temp file as input to the steps above, and
# append to the end of the rolodex file.
} < /tmp/rolo >> ~/rolodex.html
# Output some more static HTML - the document footer
echo '
</TABLE>
</BODY>
</HTML>
' >> ~/rolodex.html
# Set the new file's type properly so it doesn't just get a generic icon
settype text/html ~/rolodex.html
# Clean up
rm /tmp/rolo
# Launch the new rolodex in user's preferred web browser
/system/Tracker ~/rolodex.html
--------------------------------
The script's output will be an HTML document similar to this. Dressing it up and making it more functional is merely a matter of inserting more HTML, extracting more attributes, and customizing to taste. Click for larger version.
For a much more complete example of this technique check out the "bepeople" script in the TrackerBase package. RipEnc also uses many of the techniques described in this article, but applies the concepts to MP3 databasing.