Month: May 2008

For an upcoming project, I need to pull in metadata about movies and TV shows — genres, plot summaries, actors, etc. The de-facto source is, of course, IMDB. Unfortunately, they’re behind the times and don’t offer an API to access their data. (At least not one that I’ve ever found.)

So, here’s a quick PHP class that takes a movie title (doesn’t have to be exact) or a filename (!) and scrapes IMDB for the relevant info.

Using the scraper is simple.

<?PHP
    $m = new MediaInfo();
    $info = $m->getMovieInfo('American Beauty');
    print_r($info);

will output:

Array
(
    [kind] => movie
    [id] => tt0169547
    [title] => American Beauty
    [rating] => 8.6
    [director] => Sam Mendes
    [release_date] => 1 October 1999
    [plot] => Lester Burnham, a depressed suburban father in a mid-life crisis...
    [genres] => Array
        (
            [0] => Drama
        )
    [cast] => Array
        (
            [Kevin Spacey] => Lester Burnham
            [Annette Bening] => Carolyn Burnham
            [Thora Birch] => Jane Burnham
            [Wes Bentley] => Ricky Fitts
            [Mena Suvari] => Angela Hayes
            [Chris Cooper] => Col. Frank Fitts, USMC
            [Peter Gallagher] => Buddy Kane
            [Allison Janney] => Barbara Fitts
            [Scott Bakula] => Jim Olmeyer
        )
)

At the moment, the class only returns data for movies. For TV shows I’m planning on pulling data directly from the database I’ve created for Schmooze.TV (which, in turn, scrapes its info from TVRage).

You can download the source from my Google Code project. As always, this code is released under the MIT License. Comments and suggestions are always welcome.

So it’s been two weeks since my Drobo arrived from Amazon. I’m sure they’re selling like hotcakes, but I don’t personally know anyone else with one — I figured I might as well gather my thoughts and post a review. Hopefully this will give a good overview if you’re not already familiar with the device. If you are, I’ll be including a couple features that surprised me — so keep reading.

What’s a Drobo?

In a nutshell, Drobo is an external device that takes four hard drives and makes them appear as a combined, larger one to your Mac or PC. A better way to describe Drobo might simply be magic. Arthur C. Clarke’s third law of prediction states “any sufficiently advanced technology is indistinguishable from magic.” That very much applies here because, as far as I’m concerned, what Drobo does is magical. Take four hard drives of any size, from any manufacturer, slide them into the Drobo and not only do you get a larger pool of storage, but all of the data is redundantly backed up.

Think about that.

Without any technical know-how, special tools, or backup schedules, all of your data is safely stored away without any intervention on your part. If one of the drives fails, you just pull it out and put a new one in it’s place — all while Drobo is up and running.

I could go on and on about how cool all this is, but I think it’s best if you just take ten minutes and watch the Drobo introductory video yourself. My jaw was on the floor the first time I watched it. It’s totally worth your time if you haven’t seen it.

The Drobo Mindset

I’m pretty fanatical about backing up my data. My backup process is always evolving, but lately it has been a combination of multiple, external hard drives, Time Machine, .Mac, and Amazon S3. Even with all that, it’s still a struggle. Hard drives fill up, and large files (over 1GB) are problematic to store in multiple locations. It’s easy to get neurotic about keeping your data safe. But after using the Drobo for a few days, after getting into the Drobo mindset, a sense of calm came over me. I know it sounds crazy to wax philosophical about a glorified hard drive, but it’s true. I realized that I could stop worrying about the two biggest problems I faced backing up my data:

• Storage capacity — if get low on space, I just replace the smallest drive in Drobo with a new, larger one. I can never run out of room.
• Data redundancy — as long as two drives inside Drobo don’t fail at the same time (a slim possibility) and barring acts of god or theft, any data I store is safe.

My Setup

My wife and I have four computers in our house: two Mac laptops, an iMac, and a Mac Mini. Everything is wireless except for the Mini which is connected to our living room TV and hardwired into an Airport Extreme. The Drobo is connected to the Mini and shared throughout the network via AFP. (I’ve found the Airport Extreme’s USB sharing feature to be flaky at best with any hard drive — so I never attempted to connect Drobo to it.)

The two laptops and my wife’s iMac do regular Time Machine backups wirelessly to Drobo. (My laptop does an additional, full SuperDuper! clone to a drive at work every afternoon.) Also stored on the Drobo is a mirrored copy of my iTunes library (so Front Row on the Mini can access my music), disk images of important (read: large) software like Office, Creative Suite, and even my OS X install DVD. Plus a couple hundred gigs of photos, home videos, and movies and TV shows we’ve bought and ripped from DVD. All in all, Drobo is holding nearly a terabyte of data.

This data used to be spread across two external drives attached to the Mini and the Mini itself. Not only was none of it backed up, but it was confusing trying to remember which drive held which data. Now everything is organized and available in one location. If we want to watch a movie or TV show, there’s only one place to look.

How Does Drobo Perform in the Real World?

All the convenience Drobo affords wouldn’t be worth anything if it didn’t perform well. Fortunately, it’s very good. Not perfect — but definitely up to snuff.

I say it’s not perfect because in an ideal world Drobo would connect via Firewire 800 rather than the comparatively slow USB 2.0 standard. So you’ll definitely notice a speed hit if you’re a Mac user with an ultrafast Firewire drive. Typically, though, USB 2.0 is the norm. And, for the type of usage I throw at Drobo, it’s more than acceptable in most situations. Most of my access is over the network where I’m limited by the 300Mbps bandwidth of my wireless N router versus the 320Mbps (480Mbps theoretical) speed of USB 2.0.

But what about when Drobo is connected directly to a computer instead of over the network? When I first got it, I initially had to transfer over all the data on my existing drives. From an Iomega Firewire drive plugged into my MacBook Pro, I was able to copy files to Drobo at about 1GB per minute. Considering I had over 500GB to transfer, that sucked. Fortunately, that’s really just a one-time penalty. I usually don’t move more than a few gigs at any one time, so waiting a minute or two (especially over the network) isn’t a deal breaker for me.

The bottom line is that I would never use Drobo for intensive operations like a Photoshop swap disk, but for most normal situations it’s great.

What about noise? Is Drobo loud? With four hard drives spinning inside, it’s definitely not going to be silent. I can hear its fans and the “click” of the hard drives in a quiet room. But with any sort of background noise like music or a TV, I never notice that it’s there. The volume is comparable to the noise a Wii makes when its disc is spinning up to full speed.

Little Known Drobo Facts

The faceplate that you remove to access the drives is magnetic. It pulls off and snaps into place with a satisfying thwpp. Also, the inside of the faceplate has a quick reference chart of what Drobo’s various status lights mean.

Drobo has an internal, rechargeable battery that provides enough time during a power loss to finish the data it was writing and leave itself in a “safe state” as the manual puts it.

When running, the drives are hot to the touch. Not the Drobo — I mean if you remove the faceplate and touch the four drives inside. The Drobo itself stays cool. Even the air the fans blow out the back is relatively cool.

There is a small reset button on the back you can press with a paperclip to reset the data on the Drobo. I don’t know exactly what this does technically speaking. I imagine it just marks all the data as “erased” — I can’t imagine it actually zeroes out all the bits.

It’s bigger than I expected. In hindsight, I should have anticipated the size considering it has to be big enough to hold four hard drives plus room for a fan and circuitry. Still, it’s definitely not pint sized.

Drobo comes with software you can install on your Mac or PC to get detailed status info, but it’s not necessary to actually use the device. For example: each drive has a status light which shows the current health of the drive. There’s a line of ten blue LEDs which give you a percentage of the total capacity used. More clever even, what happens if you’re running dangerously low on space but the Drobo is out of sight so you can’t see the capacity gauge? Drobo artificially slows down write operations to alert you of the problem.

For technical reasons, Drobo always reports that its total capacity is 2TB, however the reported amount of disk usage is correct. If you use large enough drives to push the total capacity over 2TB, Drobo makes the extra space appear to be on a separate volume. The manual says this is necessary because of a limitation in the USB 2.0 protocol.

The Final Verdict

Is the Drobo awesome? Most definitely. If you’re looking to buy one and you liked this review, consider buying it via this Amazon link. Doing so will earn me a few referral dollars from Amazon 🙂

Clever solution from Daniel Jalkut of Red Sweater Software to speed up reading referral links pointing to your website. His script highlights the relevant sections so you can quickly scan down to what was said about you.

[Update: You might also be interested in s3up for storing static content in Amazon S3.]

Earlier this week I posted about my experience redesigning this site, focusing on optimizing my page load times using YSlow. A large part of that process involved storing static content (images, stylesheets, JavaScript) on Amazon S3 and using it like a poor man’s content delivery network (CDN). I made some hand-waving references to a deploy script I wrote which handles syncing content to S3 and also adding expiry headers and gzipping that data. A couple users emailed asking for more info, so, here goes.

Why Amazon S3?

Since its launch, nearly every technical blogger on the net has weighed in on why Amazon S3 is (for lack of a better word) awesome. No need for me to repeat them. I’ll just say quickly that it’s cheap (as in price, not in quality), fast, and easy to use. If you’ve got the deep pockets of a corporation backing you, you could probably find a better deal with another CDN provider, but for bloggers, startups, and small businesses, it’s the best game in town.

Amazon S3 is platform and language agnostic. It’s a massive harddrive in the sky with an open API sitting on top. You can connect to it from any system using just about any programming language. For this tutorial, I’ll be using a slightly modified version of the S3 library I wrote in PHP. I say “slightly modified” because I had to make a few changes to enable setting the expires and gzip headers. These changes will eventually make their way into the official project — I just haven’t done it yet.

The Deploy Process

YSlow recommends hosting static content such as images, stylesheets, and JavaScript files on a CDN to speed up page load times. It’s also best to give each file a far future expiration header (so the browser doesn’t try to reload the asset on each page view) and to gzip it. On a typical webserver like Apache, these are simple changes that you can do programatically through a config file. But Amazon S3 isn’t really a web server. It’s just a “dumb” storage device that happens to be accessible over the web. We’ll need to add the headers ourselves, manually, upfront when we upload.

The deploy script will also need to be smart and not re-upload files that are already on S3 and haven’t changed. To accomplish this we’ll be comparing the file on disk with the ETag value (md5 hash) on S3. Let’s get started.

Images

Deploying images is straight forward.

• Loop over every image in our /images/ directory.
• Calculate the file’s md5 hash and compare to the one in S3.
• If the file doesn’t exist or has changed, upload it using a far futures header.
• Repeat for the next image.

Source:

<?PHP
    $files = scandir(DOC_ROOT . IMG_PATH);
    foreach($files as $fn)
    {
        if(!in_array(substr($fn, -3), array('jpg', 'png', 'gif'))) continue;
        $object   = IMG_PATH . $fn;
        $the_file = DOC_ROOT . IMG_PATH . $fn;
        // Only upload if the file is different
        if(!$s3->objectIsSame($bucket, $object, md5_file($the_file)))
        {
            echo "Putting $the_file . . . ";
            if($s3->putObject($bucket, $object, $the_file, true, null, array('Expires' => $expires)))
                echo "OK\n";
            else
                echo "ERROR!\n";
        }
        else
            echo "Skipping $the_file\n";
    }

JavaScript and Stylesheets

The same process applies to JavaScript and stylesheets. The only difference is we need to serve gzip encoded versions to browsers that support it. As I said above, S3 won’t do this natively so we need to fake it by uploading a plaintext and a gzipped version of each file and then use PHP to serve the appropriate one to the user.

In the master config file on my website, I set a variable called $gz like so:

<?PHP
    $gz  = strpos($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip') !== false ? 'gz.' : '';

That snippet detects if the user’s browser supports gzip encoding and sets the variable appropriately. Then, throughout the site, I link to all of my JavaScript and CSS files like this:

<link rel="stylesheet" href="https://clickontyler-clickonideas.netdna-ssl.com/css/main.<?PHP echo $gz;?>css" type="text/css">

That way, if the $gz variable is set, it adds a “gz.” to the filename. Otherwise, the filename doesn’t change. It’s a quick way to transparently give the right file to the browser.

With that out of the way, here’s how I deploy the gzipped content:

<?PHP
    // List your stylesheets here for concatenation...
    $css  = file_get_contents(DOC_ROOT . CSS_PATH . 'reset-fonts-grids.css') . "\n\n";
    $css .= file_get_contents(DOC_ROOT . CSS_PATH . 'screen.css') . "\n\n";
    $css .= file_get_contents(DOC_ROOT . CSS_PATH . 'jquery.lightbox.css') . "\n\n";
    $css .= file_get_contents(DOC_ROOT . CSS_PATH . 'syntax.css') . "\n\n";
    file_put_contents(DOC_ROOT . CSS_PATH . 'combined.css', $css);
    shell_exec('gzip -c ' . DOC_ROOT . CSS_PATH . 'combined.css > ' . DOC_ROOT . CSS_PATH . 'combined.gz.css');
    if(!$s3->objectIsSame($bucket, CSS_PATH . 'combined.css', md5_file(DOC_ROOT . CSS_PATH . 'combined.css')))
    {
        echo "Putting combined.css...";
        if($s3->putObject($bucket, CSS_PATH . 'combined.css', DOC_ROOT . CSS_PATH . 'combined.css', true, null, array('Expires' => $expires)))
            echo "OK\n";
        else
            echo "ERROR!\n";

        echo "Putting combined.gz.css...";
        if($s3->putObject($bucket, CSS_PATH . 'combined.gz.css', DOC_ROOT . CSS_PATH . '/combined.gz.css', true, null, array('Expires' => $expires, 'Content-Encoding' => 'gzip')))
            echo "OK\n";
        else
            echo "ERROR!\n";
    }
    else
        echo "Skipping combined.css\n";

You’ll notice that the first thing I do is concatenate all of my files into a single file — that’s another YSlow recommendation to speed things up. From there, we compress using gzip and then up the two versions. Looking at this code, there’s probably a native PHP extension to handle the gzipping instead of exec’ing a shell command, but I haven’t looked into it (yet).

Also, make sure and notice that I’m adding a Content-Encoding: gzip header to each file. If you don’t do this, the browser will crap out on you when it tries to read the file as plaintext.

And We’re Done

So those are the main bits of the script. You can download the full script (and the S3 library) from my Google Code project.

It’s been a long time coming, but I finally pushed out a new design for this website last month. I rebuilt it from the ground up using two key tools from the Yahoo! Developer Network:

• YUI Grids and
• YSlow

The new design is really a refresh of the previous look with a focus on readability and speed. I want to take a few minutes and touch on what I learned during this go-round so (hopefully) others might benefit.

Color Scheme

Although I really liked the darker color scheme from before, it was too hard to read. There simply wasn’t enough contrast between the body text and the black background. I tried my best to make it work — I searched around for various articles about text legibility on dark backgrounds. I increased the letter spacing, the leading, narrowed the body columns, and everything else I learned in the intro graphic design class I took in college. The results were better, but my gut agreed with all the articles I read online which basically said “don’t do it.” So I compromised and switched to a white body background, while leaving the header mostly untouched. I find the new look much more readable — hopefully this will encourage me to begin writing longer posts.

CSS and Semantic Structure

The old site was built piecemeal over a couple months and, quite frankly, turned into a mess font-wise. I had inconsistent headers, font-weights, and anchor styles depending on which section you were viewing. With the new design, I sat down (as I should have before) and decided explicitly on which font family, size, and color to use for each header. I specced out the font sizes using YUI’s percent-based scheme which helps ensure a consistent look when users adjust the size. (Go ahead, scale the font size up and down.) An added bonus was that it forced me to think more about the semantic structure of my markup. (If you have Firefox’s Web Developer toolbar installed, try viewing the site with stylesheets turned off.) If there’s one thing I learned working for Sitening, it’s that semantic structure plays a huge part in your SERPs.

Optimizing With YSlow

At OSCON last summer, I attended one of the first talks Steve Souders gave on YSlow — a Firefox plugin that measures website performance. That, plus working for Yahoo!, has kept the techniques suggested by YSlow in the back of my head with every site I build. But this redesign was the first time I committed to scoring as high as I could.

As usual, I coded everything by hand, paying attention to all the typical SEO rules that I learned at Sitening. Once the initial design was complete and I had a working home page, I ran YSlow.

Ouch. A failing 56 out of 100. What did YSlow suggest I improve? And how did I fix it?

• Make fewer HTTP requests – My site was including too many files. Three CSS stylesheets, four JavaScript files, plus any images on the page. I can’t cut down on the amount of images (without resorting to using sprites – which are usually more trouble than they’re worth), so I concatenated my CSS and JS into single files. That removed five requests and brought me up to an “A” ranking for that category. (I’m further toying with adding the YUI Compressor into the mix.)
• Use a content delivery network – At Yahoo! we put all static files on Akamai. Other large websites like Facebook, Google, and MySpace push to their own CDNs, too. But what’s a single developer to do? Use Amazon S3 of course! I put together a quick PHP script which syncs all of my static content (images, css, js) and stores them on S3. Throughout the site, I prepend each link with a PHP variable that lets me switch the CDN on or off depending on if I’m running locally or on my production server. (And, in the event S3 ever goes down or away, I can quick switch back to serving files off my own domain.)
• Add expiry headers – Expiry headers tell the browser to cache static content and not attempt to reload it on each page view. I didn’t want to put a far future header on my PHP files (since they change often), but I did add them to all of the content stored on S3. This is fine for images that should never change, but for my JavaScript and CSS files it means I need to change their filename whenever I push out a new update so the browser knows to re-download the content. It’s extra work on my part, but it pays off later on.
• Gzip files – This fix comes in two parts. First, I modified Apache to serve gzipped content if the browser supports it (most do) — not only does this cut down on transfer time, but it also decreases the amount of bandwidth I’m serving. But what about content coming from S3? Amazon doesn’t support gzipping content natively. Instead, in addition to the static files stored there, I also uploaded their gzipped counterparts. Then, using PHP, I change the HTML links to reference the gzip versions if I detect the user’s browser can handle it.
• Configure ETags – ETags are a hash provided by the webserver that the browser can use to determine if a file has been modified before downloading it. Amazon S3 automatically generates ETags for every file — it’s just a free benefit of using S3 as my CDN.

So, all of the changes above took about three hours to implement. Most of that time was spent writing my S3 deploy script and figuring out how to make Amazon serve gzipped content. Was it worth it? See for yourself.

Wow. Three short hours of work and I jumped to a near perfect 96 out of 100. The only remaining penalty is from not serving an expires header on my Mint JavaScript.

Do these optimization techniques make a difference? I think so. Visually, I can tell there’s a huge increase in page rendering time on both Firefox and Safari. (IE accounts for 6% of my traffic, so I don’t bother testing there any longer.) More amazing, perhaps, is the site’s performance on iPhone. The page doesn’t just load — it appears.

I’ve made a bunch of vague references to the S3 deploy script I’m using and how to setup gzip on Amazon. In the interest of space, I’ve left out the specifics. If you’re interested, email me with any questions and I’ll be happy to help.