heyblog

« Berlin sticker streetart | Main | Katamari Damacy is real »

December 29, 2004

A solution to MT-Amazon build problems

I read about a new and interesting Typepad feature that lets you display your Amazon wishlist in your weblog. I then read a post by Sixapart honcho Anil Dash in which he implies (in a comment) that one could do something similar in MoveableType using the MT-Amazon and BookQueueToo plugins. So, I spent a couple of days off and on installing and configuring those two plugins, including getting my hosting provider to upgrade to Perl 5.8.1, and then found the setup doesn’t work. Actually, BookQueueToo works, and even has a nice interface that works inside MT, just like a good plugin should. But MT-Amazon would fail almost every time I asked it to help build a page with Amazon content on it.

If I’d looked before I leaped, I probably wouldn’t have bothered at all. Kottke described his problems using MT-Amazon back in July. The problem is that Amazon at some time around November 2003 stopped allowing calls from outside applications to its web service more than once per second, as MT-Amazon developer Adam Kalsey wrote to the list here. When MT uses MT-Amazon during page rebuilds, it definitely tries to call Amazon’s web services faster than that, and fails when Amazon sends back an error telling it to slow down. I can’t exactly blame Amazon for restricting people like me from pointlessly hitting their servers all the time—I’m not exactly paying for the privilege—but this restriction is pretty lame. Obviously, if you need to create a page that includes, say, 40 Amazon items, it will take no less than 40 seconds to do it. (Incidentally, how does Typepad get around this restriction? Do they have a special deal with Amazon allowing more frequent calls?)

If you can live with that restriction, there’s an easy fix. Myself I don’t mind: the page I’ll use MT-Amazon on won’t get rebuilt often (only when I manually add a book to it using BookQueueToo, maybe once every two weeks), and isn’t incorporated into any other page on my site. In other words, I don’t care if it takes a long time to rebuild because I’m the only one who has to sit and wait for it. Anyone else browsing the page won’t have to wait any longer than normal. If you can’t live with that restriction, and don’t mind that your Amazon content might not appear, Kalsey suggested a fix. Adam’s proposed solution is basically to suppress the error and stop building the page without the Amazon content, which is no good to me. (As an aside, it looks like Adam’s pretty much stopped working on the plugin. It would be nice if someone could add some error-checking to it to deal with this now-common problem. MT-Amazon should stop if it gets an error, but finish building the page with the data it’s gotten to that point. It should also be easy to turn this behavior on and off with a debug flag or something.)

Anyway, prepare yourself the mad Perl skillz I’m about to display which let me solve the problem. I reasoned that since MT-Amazon was calling Amazon’s web service too often, why not force it to slow down? Turns out there’s a simple way in Perl to “pause” a script briefly: the sleep command. So, I opened up MTAmazon.pm (which is in my /mt/extlib/MTPlugins folder, as per MT-Amazon’s installation instructions), and found these lines:

for (my $page = 1; $page <= $pages; $page++) {
    $content = AmazonGetXML($search, $associateid, $method, $line, $devtoken, $page, $format, $sort);

This section of the code is saying “for each book in the list, go get some information from Amazon.com”, and it’s what’s happening too frequently. I changed the code to:

for (my $page = 1; $page <= $pages; $page++) {
    sleep 1;
    $content = AmazonGetXML($search, $associateid, $method, $line, $devtoken, $page, $format, $sort);

That sleep 1; means “pause for one second before going to the next line”, and it effectively throttles this otherwise-fast process….to…take…one…second…in…between….calls…to…Amazon, which is just what we want.

Again, this will force rebuilds of any page using MT-Amazon to go more slowly. My page of about 60 books takes, surprise, about a minute to rebuild, which is painfully long. But it does finish correctly, and I am sort of pleased that “sleep 1;” is the only line Perl I’ve ever written, and it solves my problem perfectly.

Posted by Andrew at December 29, 2004 08:16 PM

Comments

Posted by: Becky Blitch at March 4, 2005 03:08 PM