A carnival of technology

Today America celebrates 230 years of Independence, but, instead of blowing up a little piece of it using the fireworks I bought from Alabama a few years ago that may or not still be good, I updated nanocomment. I’ve added trackback support, for those of you into that sort of thing, and added the option for anonymous named comments, like I’ve been using in the blog for a while, to the released files. nanocomment-2.0.tar.gz. Enjoy.

I’ve been playing around with the MARTA map again trying to add in some of the bus routes. I’ve been working my way North to South and have a few added, but one thing I noticed pretty quickly is that adding a lot of polylines to the google map slows it way down. Ajax is pretty damn slow to begin with, so this just won’t do. One workaround I found was to use an undocumented tile overlay feature. Google maps allows for custom tile overlays to be created, and since these are just a set of PNGs like the rest of the map data, it doesn’t tax the browser any more than displaying all the streets and stuff. There’s also a trick you can use to get Google to make the tiles for you based on a KML file, similar to how you can put a KML URL into the Google Maps search box and see it displayed on the web. I’ve switched the rail and bus routes to KML, but I’m not sure how long this will work.

One of the bigger backend changes I made when I copied David’s map was to put the data in an XML file. I’m not a huge fan of XML and all it stands for, but it makes more sense to have the data in XML than to encode it in Javascript, and I expanded on that to create a big ridiculous schema so I can pursue hopeless dreams of creating route searches, kind of like Google Transit. This decision allowed me to generate KML from this source data file, but I still use the original XML for marker placement since the KML overlay doesn’t allow as much flexibility as far as information formatting, and I need to make my own marker objects in order to provide that station search form. That soure file is pretty big and unweildy, though, so I’m thinking of creating more compact XML files from this source file containing just the data needed for the map.

One of the limitations of the KML overlay, besides it being undocumented, unsupported and possibly a violation of the terms of use, is that KML files are limited 1MB. That’s a pretty good number of bytes, but I’m already at 84k with only eight routes. I may end up having to generate my own tiles yet.

I put together a release of my comments system. It’s not pretty, and it’s probably not complete, but I think it’s easier to manage than NBCom, and it’s certainly a lot nicer as far as keeping with the Nanoblogger appearance. I hope this helps, world.

You probably know about that firefox extension I made a while back to modify the browser features that annoy me. Last week I decided to submit it to addons.mozilla.org mostly just to see what would happen. I figure that sites like mozilla’s and freshmeat are places I go myself when I can’t find what I want in a simple google search, and maybe more than three people out there are interested in disabling autocomplete popups and such.

What ended up happening was I filled out the funky little web form, placing all the useful details in the comments to reviewers and forgetting to include anything explaining the features in the comments to users. After a few days of sitting in the queue, my extension was approved, I got a page (NB: NSFW language in comments), and I soon after received three comments pointing out that I didn’t really explain myself. The first guy, by far the angriest, mentioned this pretty directly in a short screed against myself and the mozilla.org moderators, but the second guy found the README and posted its entire contents, so the third guy really had no excuse for not knowing what the extension does. I’ve since fixed the description, but the comments are a helpful reminder to me concerning placement of documentation as well as being a source of amusement. I’ve had 278 downloads so far, so maybe one or two people amongst the various humans and bots who blindly download everything are getting some use out of it.

One of the bots is apparently a site called Softpedia. I just received an email from them informing me that they’ve certified rmannoy as being “completely clean of adware/spyware components,” and they are “impressed with the quality of [my] product and encourage [me] to keep this high standards [sic] in the future.” So fear not, fair user: I’m not keeping track of how often you refresh Fark while saving you from browser popups and unwanted META refresh directives. Web browser people are weird.

On the nanowrimo front, I had a few bad days last week, but I’m starting to catch up. I’m about half a day behind in words, having just now crossed the halfway mark. With tonight’s work in progress of 750 words, I have a total of 25594, putting me at 656 behind if I hit today’s 1700 word goal. I figure I still have a shot at this, but Thanksgiving next week is probably going to be a little hard.

I guess that Mozilla released another version of Firebird recently. This one bumps the major version number, so it must be all kinds of better. For example, “Extensions” are now lumped in with themes and called “Add-ons,” and none of the ones that you had before work because extension developers are tired of trying to figure out all the dumb crap that Mozilla pointlessly changed. Well, one of those developers took a half-hearted shot at an upgrade: rmannoy 1.1.2 is now available. This one has actual code changes, so you can’t just modify your local copy of the install.rdf and expect things to work. I’ve updated update.rdf, so using the auto-update thing might work.

And now for a handy Firefox tip. You know how when you go to install an extension, it won’t let you do anything for five seconds, and if you lose the window’s focus the timer starts over again? Man, that’s annoying. It’s like Schilly wrote the extension installer or something. Anyhow, I was going to try making that window less annoying and adding that as a feature to rmannoy, but I found that there’s already a way to disable the timer and just let you freaking install whatever you clicked on. In about:config, set security.dialog_enable_delay to 0. Apparently annoyance is an essential element to a secure system. Enjoy.

You know that XML thing? Ever notice that one of the frequent XML examples is how to encode a mailing address? I guess mailing addresses are something people use pretty frequently in their data. Well, I’ve noticed this, and I’ve also noticed that no one has ever (that I can find) made a standard for specifying mailing addresses. ADIS is the only thing I’ve seen, and that format contains a ridiculous granularity for addresses and requires that you include data that only a postal database could love. This XML Web 2.0 open source Internet is all about reusable modules, right? I can’t find anything to reuse, so I made my own thing. http://gophernet.org/schemata/address.xsd. It gives you enough data, according to the Fedex international waybill, to encode addresses for about any country. It won’t let you create databases searchable by county or delivery zone or whatever weird thing that foreign people use, but I don’t care about that, and I’m betting that you don’t either. For a quick example, here’s the president’s address using my new standard:

<?xml version="1.0" encoding="utf-8"?>
<address xmlns="http://www.gophernet.org/schemata/address.xsd">
  <recipient>George W. Bush</recipient>
  <address1>1600 Pennsylvania Avenue NW</address1>
  <city>Washington</city>
  <state>DC</state>
  <postalCode>20500</postalCode>
  <country>USA</country>
</address>

The address type or element in the schema can easily be used in other schemata that need an address. You’re welcome, world.

You may be familiar with the Google maps overlay that David Cantrell created a while back using the MARTA rail lines. Since David doesn’t live in Georgia anymore, I took over the role of MARTA fanatic and copied his data for my own purposes. The only improvements I’ve made so far are using the different types of circle-P icons for the different varieties of parking at the stations, adding icons for the park and ride lots and adding a handful of points of interest for CCT. I also bent the track lines a little for a few sections north of Lindbergh, but I stopped that while I come to terms with the fact that the line data files are going to be really freaking huge. I also switched the data files to XML, since I hear that makes everything more fun.

Of perhaps more interest than the map itself, though, are a couple little tools I made to simplify data input. The first is a tool for finding points. Anywhere you click on the map will add a marker, and clicking on the marker will give you the coordinates. The second is a tool for making lines. Every point you click will be added to a GPolyline object, and the coordinates will be displayed in the textbox at the bottom of the page. Comes with an undo button, because I have poor mouse control. I’ve found these two pages to be quite useful, so I thought that I would share.

I found a bug in svnacl caused by some weird behavior in svn. It turns out that if if you modify the properies on a subdirectoy and merge it in from a branch, the action for the subdirectory will be “replace” even if its parent directory had an action of “add.” I think that’s kind of unintuitive, so the new version treats these directories (and possibly files, I guess) as adds both for the sake of making my property comparisons work and for the access check.

One of the issues I’ve run into with writing Firefox extensions is that now, since I hold the keys to their arcana, I want to fix the problems in other people’s extensions. Adblock leaks memory and sullies my javascript console with its strange warnings and errors, and I doubt that the guy who writes Toolbar Enhancements is ever going to make a new release for new firefoxes. I can create my forked version of extensions easily enough, but what do I do about the updateURL?

Firefox extensions contain a link to a source of update information. Extensions can be upgraded automatically, but even if automatic upgrades are turned off, the information associated with an extension concerning what versions of Firefox it supports create another reason to check the update information: the compatibility upgrade. Your profile and installed extensions outlive any particular installation of Firefox, so when you upgrade the browser, manually or automatically, it can create a version incompatibility. Since developers aren’t expected to know the future, it’s possible to change the minVersion and maxVersion properties in the update source without making a new release, so any users looking for a compatibility upgrade may learn that their copy already works and they can continue to happily plug along.

The problem this creates for me is whether to provide information only on my forked versions or whether to also inform upgrade-seekers about new upstream releases. Though there might be times when I don’t want my changes undone by a careless upgrade, it seems easier for me to just leave everything in the hands of the user, and for other cases, such as simple compatibility changes, I would want any newer upstream release to supersede my changes. That leaves me to try to figure out how to make two sources of information into one.

Like many lazy people out there, when faced with a task needing dynamic content on the Web, I turn to PHP. It’s a terrible language, but it’s easy to use and does ok at Web things, so half-assed is good enough. Firefox update sources are in RDF, a data-description model that’s been around since, I think, Netscape 4. RSS was born of attempts to summarize a site’s content in terms of RDF, and RDF had another brief surge of interest with Tim Berners-Lee’s Semantic Web idea before people figured out that Web 2.0 was a better name to use to trick investors, but none of that really matters. What’s important to me and my stupid problems is that the XML syntax for RDF allows too many ways to say the same thing for me to just drop in my augmentations to an update file through string processing or even an XML parser, so I’m going to need to actually parse the RDF.

RDF is one of those standards that’s really simple at its base—describe data using a set of triples: subject, predicate and object—but still manages to fill 227 pages of W3C recommendation and countless related drafts and member submissions. After briefly flirting with the idea of reinventing the wheel, I decided that this would be really dumb and started to look for other people’s code to misappropriate. rdfapi is the first google hit for “rdf php,” but it blows. I neither want to try to fix its code nor to fill up my hard drive with its logged warnings and errors. The first hit for “rdf library” is the Redland suite, a much more mature library written by a guy who works at Yahoo! and also happens to be the editor for the RDF/XML syntax recommendation. Redland seems like a better idea, and it comes with PHP bindings.

As I’ve mentioned before, PHP is a crappy language. I do not blame Mr. Beckett one bit for not wanting to code any PHP, instead taking the route of the rest of the PHP library by providing direct bindings to the C functions, but, since Redland has a pretty nice API to begin with, written in that bizarre style of halway object-oriented programming that only C can pull off, it becomes rather obvious on the PHP side that I’m using bindings into a language that expects me to manage my own memory. To alleviate this, I created a PEAR package (are they called packages? modules? extensions?) to provide things like garbage collection, exceptions and all that fancy stuff. As long as you have both the Redland PHP bindings and SPL (comes with PHP), you can use pear install http://gophernet.org/projects/librdf-php/LibRDF-1.0.0.tgz and it’ll dump it wherever PHP extmodackages are stored. On this journey through the fancier aspects of a bad language, I found an attempt to do what javadoc does, so I wrote big, redundant comments for everything and created some files. I also have a demonstration (source) of adding a fictional new version, 1.1.1.0ds1, of rmannoy to the existing update.rdf. You may notice that I replaced all of the list items in the main sequence with “li” predicates; librdf serializes sequences back out using the actual value of the predicate, _1, _2 and so forth, and the firefox RDF parser doesn’t dig this. The Netscape RDF parser has been around longer than the standard, so I don’t care enough to try to point fingers and find out who’s more right. Anyhow, if you ever need to manipulate RDF in PHP, I hope this helps.

I don’t think I’d ever make it working for the IETF. Although I can usually see the rationale behind some of the weirder aspects of their standards, they quite often do things in a fashion entirely different from how I think I would have done it. For an example, let’s take a look at secure FTP. Were I given the task of securing FTP communications, I would have used a new TCP port, connecting to which would require an SSL connection to be established on top of the TCP connection, thus inserting something into one of those OSI layers that don’t really mean anything, leaving the transport and application layers unchanged. The IETF instead decided to implement it as an extension to the existing FTP protocol, creating a sort of encryption switch that can be turned on and off at will for both data and control connections and, though creating something extensible to other means of encryption, producing a handful of caveats for the casual implementor.

RFC 4217 isn’t a real standard, not currently listed in the STD0001 document, but it’s what everyone uses, and, in the great tradition of the Internet, everyone does a half-assed job of actually implementing it. The most major issue on the server side seems to be that every server, when closing the connection on binary data transfers, doesn’t bother to shut down the TLS session. I don’t care about the server, though; I just want a working client.

I had a need for a means in python of doing FTP over TLS transfers, and I quickly found that everyone who’s tried to create such a class implements approximately half of the necessary extensions. Since I need to send files as well as receive them, I figured I’d take a shot at doing it right, and so I put together ftpslib. If you’re looking for a Python class that can connect to secure FTP servers, give it a shot. You have to pick your favorite SSL module, but I’ve provided the means of tying in the builtin socket.ssl and M2Crypto. I use M2Crypto myself, since it doesn’t lock up on close() like socket.ssl. There was a fun little issue with M2Crypto not closing the TCP connection when you call close(), an issue quickly discovered when trying to upload anything in either binary or ASCII mode, but I added a workaround for that in the glue function provided in the ftpslib module, so if you use that you’ll be fine.