Tyler

This afternoon I needed an easy way to upload files to Amazon S3 and set specific headers on them. I’ve built one-off scripts like this in the past, but this time I wanted to generalize the problem into a reusable shell command.

I pulled in some code from PHP-AWS and got it working fairly quickly. However, I got swamped with an ugly nest of if and switch statements when I began adding support for multiple arguments and switches. A quick Google search didn’t find any ready-made PHP solutions, so I rolled my own.

It’s a small class — about 60 lines. It could easily be turned into a single function instead, but I like using classes as ad-hoc namespaces. Anyway, it’s fairly robust for a twenty minute solution. Here are a few examples of the syntax it supports.

cmd -a -b -c // Single letter flags

cmd -abc // Same as above

Beyond setting a few flags, you can also assign values to those flags.

cmd -a foobar.jpg -bc

In this case, b and c are set just like the previous example. And a is set to the value foobar.jpg as you’d expect.

Double-dash, long names are also supported.

cmd --some-flag
cmd --another-flag=charlie // You can use an equal sign
cmd --another-flag charlie // Or not. It's up to you.

And those can be mixed with the single dash variants.

cmd -ab --bigflag=foo -c bar apricot orange

So, a and b are set. bigflag is set to “foo” and c is set to “bar”. “apricot” and “orange” will appear as arguments not associated with any specific flag.

My point is that you can use nearly any standard Unix convention.

So, given that last example, here’s how you’d parse it using the PHP class.

<?PHP
    $args = new Args();

    if($args->flag('a'))
        // -a was set, so do something

    if($args->flag('bigflag') == 'foo')
        // --bigflag was set to 'foo'

    foreach($args->args as $arg)
        // Do something with each argument.
        // Args will be bar, apricot, and orange.

So that’s it. The class doesn’t assume much. It just picks out what you give it. It’s up to you, the programmer, to interpret those results.

As always, the code is released under the MIT License and available here.

Update: Want to stream your iTunes music over the internet? Try Highwire! It streams your iTunes library and a whole lot more 🙂

I keep all of my iTunes music stored on a Drobo attached to an Airport at home. This frees up valuable space on my laptop and lets me listen to it via Front Row on my TV as well. It’s a much more convenient solution all around.

The only problem is that I lose access to my music when I’m on the road or at work. I’ve seen other services or 3rd party apps that let you stream your music online. They work well enough, but I’ve always thought it silly to pay for a feature that iTunes used to have. In honor of iTunes’ new 8.0 release this morning, here’s my simple workaround.

First off, you’ll need a Mac at home with iTunes open that is always connected to the net. In my case, that’s a Mac Mini in our living room. It also needs to have “Remote Login” (ssh) enabled in System Preferences → Sharing.

Then, poke a hole in your router’s firewall to that machine on TCP port 3689. Here’s a screenshot of my router’s settings:

(Note: your computer’s IP address might be different than mine.)

Then, with that done, anytime you want to listen to your music elsewhere, run these two commands in Terminal on the Mac you’re listening with — not your Mac at home.

ssh your_username@your-home-ip-address -N -f -L 3689:your-home-ip-address:3689

That creates a tunnel from port 3689 on your local machine to port 3689 on your Mac at home. (That’s why you needed to open the hole in your firewall.)

mDNSProxyResponderPosix 127.0.0.1 squeal "My Music" _daap._tcp. 3689 &

That creates an iTunes Bonjour broadcast notification locally which points back to your Mac at home. In other words, it tricks your copy of iTunes into thinking there’s a Mac nearby sharing its iTunes library. When iTunes tries to connect, the traffic is automatically rerouted to your other Mac.

Sidenote: if you don’t have the mDNSProxyResponderPosix command installed on your Mac, I have download links at the bottom of this post.

So that’s it. No need for third party software. When you open iTunes, you should see your music library appear in the “Shared” section of the sidebar. (Much to my excitement, the newly announced Genius playlists appear as well!) You can close the Terminal window once you’ve connected to your music library. (The mDNSProxyResponder command needs to stay active until then.)

Making it Better

To speed things up a bit, I’d put those two command into a shell script and place it in your ~/Library/Scripts folder. I call mine music-tunnel. That way, you can run that one command and have everything up and running automatically.

Or, if you’re feeling adventurous, you could wrap the whole thing inside a simple Cocoa app with a nice On / Off button. But that’s a project for another day…

Downloading mDNSProxyResponderPosix

Here’s a binary download of the mDNSProxyResponderPosix command for Intel. Place it somewhere in your path. And here’s the source if you’d like to compile it yourself for PowerPC.

I think everyone has a core group of artists they grew up listening to — that made an indelible mark on their lives. LeRoi Moore was one of mine.

If you’re lucky enough to have a Grand Central account, here’s a quick PHP class that will login to your account and dial a phone number. This is probably one of the more random bits of code I’ve ever written, but I think it’s useful.

<?PHP
    $gc = new GrandCentral('gc_username', 'gc_password');
    $gc->call($your_number, $their_number);

And that’s it. Grand Central will call $yournumber and connect you to $theirnumber.

So, you can dial phone numbers from PHP. Now what?

Well, for starters, here’s an AppleScript Address Book plugin that will let you dial a contact from your Mac. Or, if you’re like me and have lots of conference calls each week, you can attach a script to your iCal events to dial you into the conference number a minute or two prior to the call start time.

As usual, the code is MIT Licensed and available in GitHub.

Contact me if you have any questions.

Astonishing.

One of the major challenges in web scraping is figuring out which page to scrape in the first place. Here’s a scenario: Say you need to pull some information for the film 30 Days of Night off IMDB. It would be great if you knew in advance what the URL was — something you could construct programatically — unfortunately, it’s actually http://www.imdb.com/title/tt0389722/. How can you possibly figure that out?

One solution would be to scrape IMDB’s built-in search feature and from there extract the correct URL. For IMDB, that works, but what about a site that doesn’t have a search feature? Or one that does, but it doesn’t work very well?

I’ve been web scraping for years and the hands-down, best solution I’ve come up with is to simply let Google do the work for you.

The trick is to take advantage of their “I’m Feeling Lucky” feature. Clicking that button, instead of the standard “Google Search” button, skips the results page and takes you directly to the first result. If you construct your query properly, it will almost always be the page you’re looking for.

Going back to the IMDB example, if you run an I’m Feeling Lucky search for “site:imdb.com $movie_title”, Google will send you a 302 Redirect to the appropriate page within IMDB. Voila! Not only does this get us where we need to be, but (since we’re relying on Google’s ever improving search index) it will also adjust for spelling mistakes or even partial movie titles. It’s a great technique for scraping any sort of online “encyclopedia” like IMDB, TVRage, Wikipedia, etc.

Here’s the code. Pass it a search query and it’ll extract the redirect Google sends back.

<?PHP
    function feelingLucky($q)
    {
        ob_start();
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, "http://www.google.com/search?hl=en&q=" . urlencode($q) . "&btnI=I%27m+Feeling+Lucky");
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
        curl_setopt($ch, CURLOPT_HEADER, 1);
        curl_setopt($ch, CURLOPT_NOBODY, 1);
        curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1) Gecko/20061024 BonEcho/2.0");
        curl_setopt($ch, CURLOPT_REFERER, "http://www.google.com");
        curl_exec($ch);
        curl_close($ch);
        $head = ob_get_contents();
        ob_end_clean();
        return (preg_match('/Location:(.*?)$/ms', $head, $matches) == 0) ? false : trim($matches[1]);
    }

I don’t know how I ever missed this Apple Mail plugin, but you absolutely have to give MsgFiler a try if you’re a heavy keyboard user. It lets you move messages into any folder in your mailbox using only the keyboard.

Press ⌘9 to pull up a TextMate-like list of your mailbox folders. Then, select a folder by typing the first few letters in its name and move the currently selected message(s) into it with ↩.

You can also jump to a folder with ⌘O instead of ↩. Awesome.

Today I’m proud to announce the release of two new open source projects: Appcaster and OpenFeedback. I’ve been working on them off and on for over nine months, so I’m very excited to finally see them out the door.

Appcaster, which I’ve written about before, is a web-based dashboard for indie Mac developers. It’s designed to manage payment and order processing and generate license files for your users. It even handles your product’s revision history in Amazon S3 and can produce reports from your users’ demographic info. It also serves as a central location to collect user feedback, bug reports, and support questions.

OpenFeedback is a Cocoa framework written in Objective-C that collects feedback from your users directly within your application. Instead of sending your users to a website or asking them to write an email, OpenFeedback gives them a simple window where they can ask support questions, file bug reports, or suggest new features. Their data is automatically sent to Appcaster for you to review. They never have to leave your application.

Collectively, I’m calling the two projects Appcaster since they’re designed to work closely with one another (and since I wanted them to be part of the same Google Code project). However, OpenFeedback can send data to any server-side script that accepts HTTP POST requests — you can easily integrate it into your existing bug tracker or reporting system.

Appcaster

When I first built Appcaster last year, I wrote a detailed overview of the application here. Aside from cleaning up a few bugs and upgrading it to use the latest version of the Simple PHP Framework, the only major additions have been adding support for OpenFeedback and graphing user demographic data using the Google Charts API.

Google’s Chart API is such a slick, clever way of doing things that I couldn’t pass by the opportunity to use it in a project. After aggregating your data, you simply craft it into a special URL and use that as the source of an <img> tag. Google parses your data out of the URL and returns a PNG formatted chart.

It took all of half an hour to create some basic stats from the Sparkle update data Appcaster collects. It would be trivial for other developers to add their own custom reports in the future.

OpenFeedback

The idea for OpenFeedback came from Cultured Code‘s task management application Things. Clicking on the Support menu item in their Help menu brings up a dialog where you can submit questions and feedback from inside the app — no need to visit a website or open an email.

I thought their implementation was a great idea and emailed them to ask if I could recreate that functionality as a Cocoa framework for other developers to use. They were nice enough to say yes 🙂

Adding OpenFeedback to your application is trivial. Like Sparkle, there’s no code required. You simply link your app against the framework and hook-up the appropriate actions in Interface Builder. In under five minutes you can have an elegant way to encourage users to provide feedback.

My long term goal is to see the Mac developer community standardize around OpenFeedback much like they have around Sparkle. Not only would it save time for developers, but it would provide users with a consistent interface for submitting feedback. That should help improve the dialogue between developers and our users — improving Mac software all around.

Speaking of Feedback

Your feedback is always welcome. This is my first open source Cocoa project, so I’m very much flying by the seat of my pants. Suggestions, improvements, bug reports — they’re all welcome. You can send them directly to me or file an issue in our bug tracker.

For an upcoming project, I need to pull in metadata about movies and TV shows — genres, plot summaries, actors, etc. The de-facto source is, of course, IMDB. Unfortunately, they’re behind the times and don’t offer an API to access their data. (At least not one that I’ve ever found.)

So, here’s a quick PHP class that takes a movie title (doesn’t have to be exact) or a filename (!) and scrapes IMDB for the relevant info.

Using the scraper is simple.

<?PHP
    $m = new MediaInfo();
    $info = $m->getMovieInfo('American Beauty');
    print_r($info);

will output:

Array
(
    [kind] => movie
    [id] => tt0169547
    [title] => American Beauty
    [rating] => 8.6
    [director] => Sam Mendes
    [release_date] => 1 October 1999
    [plot] => Lester Burnham, a depressed suburban father in a mid-life crisis...
    [genres] => Array
        (
            [0] => Drama
        )
    [cast] => Array
        (
            [Kevin Spacey] => Lester Burnham
            [Annette Bening] => Carolyn Burnham
            [Thora Birch] => Jane Burnham
            [Wes Bentley] => Ricky Fitts
            [Mena Suvari] => Angela Hayes
            [Chris Cooper] => Col. Frank Fitts, USMC
            [Peter Gallagher] => Buddy Kane
            [Allison Janney] => Barbara Fitts
            [Scott Bakula] => Jim Olmeyer
        )
)

At the moment, the class only returns data for movies. For TV shows I’m planning on pulling data directly from the database I’ve created for Schmooze.TV (which, in turn, scrapes its info from TVRage).

You can download the source from my Google Code project. As always, this code is released under the MIT License. Comments and suggestions are always welcome.

So it’s been two weeks since my Drobo arrived from Amazon. I’m sure they’re selling like hotcakes, but I don’t personally know anyone else with one — I figured I might as well gather my thoughts and post a review. Hopefully this will give a good overview if you’re not already familiar with the device. If you are, I’ll be including a couple features that surprised me — so keep reading.

What’s a Drobo?

In a nutshell, Drobo is an external device that takes four hard drives and makes them appear as a combined, larger one to your Mac or PC. A better way to describe Drobo might simply be magic. Arthur C. Clarke’s third law of prediction states “any sufficiently advanced technology is indistinguishable from magic.” That very much applies here because, as far as I’m concerned, what Drobo does is magical. Take four hard drives of any size, from any manufacturer, slide them into the Drobo and not only do you get a larger pool of storage, but all of the data is redundantly backed up.

Think about that.

Without any technical know-how, special tools, or backup schedules, all of your data is safely stored away without any intervention on your part. If one of the drives fails, you just pull it out and put a new one in it’s place — all while Drobo is up and running.

I could go on and on about how cool all this is, but I think it’s best if you just take ten minutes and watch the Drobo introductory video yourself. My jaw was on the floor the first time I watched it. It’s totally worth your time if you haven’t seen it.

The Drobo Mindset

I’m pretty fanatical about backing up my data. My backup process is always evolving, but lately it has been a combination of multiple, external hard drives, Time Machine, .Mac, and Amazon S3. Even with all that, it’s still a struggle. Hard drives fill up, and large files (over 1GB) are problematic to store in multiple locations. It’s easy to get neurotic about keeping your data safe. But after using the Drobo for a few days, after getting into the Drobo mindset, a sense of calm came over me. I know it sounds crazy to wax philosophical about a glorified hard drive, but it’s true. I realized that I could stop worrying about the two biggest problems I faced backing up my data:

• Storage capacity — if get low on space, I just replace the smallest drive in Drobo with a new, larger one. I can never run out of room.
• Data redundancy — as long as two drives inside Drobo don’t fail at the same time (a slim possibility) and barring acts of god or theft, any data I store is safe.

My Setup

My wife and I have four computers in our house: two Mac laptops, an iMac, and a Mac Mini. Everything is wireless except for the Mini which is connected to our living room TV and hardwired into an Airport Extreme. The Drobo is connected to the Mini and shared throughout the network via AFP. (I’ve found the Airport Extreme’s USB sharing feature to be flaky at best with any hard drive — so I never attempted to connect Drobo to it.)

The two laptops and my wife’s iMac do regular Time Machine backups wirelessly to Drobo. (My laptop does an additional, full SuperDuper! clone to a drive at work every afternoon.) Also stored on the Drobo is a mirrored copy of my iTunes library (so Front Row on the Mini can access my music), disk images of important (read: large) software like Office, Creative Suite, and even my OS X install DVD. Plus a couple hundred gigs of photos, home videos, and movies and TV shows we’ve bought and ripped from DVD. All in all, Drobo is holding nearly a terabyte of data.

This data used to be spread across two external drives attached to the Mini and the Mini itself. Not only was none of it backed up, but it was confusing trying to remember which drive held which data. Now everything is organized and available in one location. If we want to watch a movie or TV show, there’s only one place to look.

How Does Drobo Perform in the Real World?

All the convenience Drobo affords wouldn’t be worth anything if it didn’t perform well. Fortunately, it’s very good. Not perfect — but definitely up to snuff.

I say it’s not perfect because in an ideal world Drobo would connect via Firewire 800 rather than the comparatively slow USB 2.0 standard. So you’ll definitely notice a speed hit if you’re a Mac user with an ultrafast Firewire drive. Typically, though, USB 2.0 is the norm. And, for the type of usage I throw at Drobo, it’s more than acceptable in most situations. Most of my access is over the network where I’m limited by the 300Mbps bandwidth of my wireless N router versus the 320Mbps (480Mbps theoretical) speed of USB 2.0.

But what about when Drobo is connected directly to a computer instead of over the network? When I first got it, I initially had to transfer over all the data on my existing drives. From an Iomega Firewire drive plugged into my MacBook Pro, I was able to copy files to Drobo at about 1GB per minute. Considering I had over 500GB to transfer, that sucked. Fortunately, that’s really just a one-time penalty. I usually don’t move more than a few gigs at any one time, so waiting a minute or two (especially over the network) isn’t a deal breaker for me.

The bottom line is that I would never use Drobo for intensive operations like a Photoshop swap disk, but for most normal situations it’s great.

What about noise? Is Drobo loud? With four hard drives spinning inside, it’s definitely not going to be silent. I can hear its fans and the “click” of the hard drives in a quiet room. But with any sort of background noise like music or a TV, I never notice that it’s there. The volume is comparable to the noise a Wii makes when its disc is spinning up to full speed.

Little Known Drobo Facts

The faceplate that you remove to access the drives is magnetic. It pulls off and snaps into place with a satisfying thwpp. Also, the inside of the faceplate has a quick reference chart of what Drobo’s various status lights mean.

Drobo has an internal, rechargeable battery that provides enough time during a power loss to finish the data it was writing and leave itself in a “safe state” as the manual puts it.

When running, the drives are hot to the touch. Not the Drobo — I mean if you remove the faceplate and touch the four drives inside. The Drobo itself stays cool. Even the air the fans blow out the back is relatively cool.

There is a small reset button on the back you can press with a paperclip to reset the data on the Drobo. I don’t know exactly what this does technically speaking. I imagine it just marks all the data as “erased” — I can’t imagine it actually zeroes out all the bits.

It’s bigger than I expected. In hindsight, I should have anticipated the size considering it has to be big enough to hold four hard drives plus room for a fan and circuitry. Still, it’s definitely not pint sized.

Drobo comes with software you can install on your Mac or PC to get detailed status info, but it’s not necessary to actually use the device. For example: each drive has a status light which shows the current health of the drive. There’s a line of ten blue LEDs which give you a percentage of the total capacity used. More clever even, what happens if you’re running dangerously low on space but the Drobo is out of sight so you can’t see the capacity gauge? Drobo artificially slows down write operations to alert you of the problem.

For technical reasons, Drobo always reports that its total capacity is 2TB, however the reported amount of disk usage is correct. If you use large enough drives to push the total capacity over 2TB, Drobo makes the extra space appear to be on a separate volume. The manual says this is necessary because of a limitation in the USB 2.0 protocol.

The Final Verdict

Is the Drobo awesome? Most definitely. If you’re looking to buy one and you liked this review, consider buying it via this Amazon link. Doing so will earn me a few referral dollars from Amazon 🙂