tyler.io

Two and a half years ago I began working with Amazon Web Services — first with S3 and then SQS and EC2. The code was eventually cleaned up and released as an open source project called PHP-AWS. Since then, it has remained relatively unchanged. Just bug fixes and the occasional support for new AWS features when users contribute patches. It’s not particularly pretty, but it has stood up well against time. The S3 class was recommended by Raymond Yee in his book Pro Web 2.0 Mashups, and a number of people have emailed examples of production sites where the library is being used.

Still, the biggest drawback (particularly with the S3 class) was that the curl commands were piped to the shell — they didn’t use PHP’s native curl extension. This was done to bypass PHP’s memory limits to allow large file uploads (and also because we didn’t have time to find a workaround). It was a dirty hack, plain and simple.

But, today, the S3 class redeems itself with full support for PHP’s curl extension. Not only do large file uploads work (I just tested a 1.5GB disk image), but there is support for new S3 features like query string authentication and copying objects.

As this is a complete rewrite, I consider it a beta release — user beware. I’ve converted over all of my own code to use the new class, which has given it a good bit of testing. It’s stable enough for my purposes, but I don’t want anyone to be surprised if there’s a bug or three lurking about.

Download the code, kick the tires, and please report any bugs you find.

This afternoon I needed an easy way to upload files to Amazon S3 and set specific headers on them. I’ve built one-off scripts like this in the past, but this time I wanted to generalize the problem into a reusable shell command.

I pulled in some code from PHP-AWS and got it working fairly quickly. However, I got swamped with an ugly nest of if and switch statements when I began adding support for multiple arguments and switches. A quick Google search didn’t find any ready-made PHP solutions, so I rolled my own.

It’s a small class — about 60 lines. It could easily be turned into a single function instead, but I like using classes as ad-hoc namespaces. Anyway, it’s fairly robust for a twenty minute solution. Here are a few examples of the syntax it supports.

cmd -a -b -c // Single letter flags

cmd -abc // Same as above

Beyond setting a few flags, you can also assign values to those flags.

cmd -a foobar.jpg -bc

In this case, b and c are set just like the previous example. And a is set to the value foobar.jpg as you’d expect.

Double-dash, long names are also supported.

cmd --some-flag
cmd --another-flag=charlie // You can use an equal sign
cmd --another-flag charlie // Or not. It's up to you.

And those can be mixed with the single dash variants.

cmd -ab --bigflag=foo -c bar apricot orange

So, a and b are set. bigflag is set to “foo” and c is set to “bar”. “apricot” and “orange” will appear as arguments not associated with any specific flag.

My point is that you can use nearly any standard Unix convention.

So, given that last example, here’s how you’d parse it using the PHP class.

<?PHP
    $args = new Args();

    if($args->flag('a'))
        // -a was set, so do something

    if($args->flag('bigflag') == 'foo')
        // --bigflag was set to 'foo'

    foreach($args->args as $arg)
        // Do something with each argument.
        // Args will be bar, apricot, and orange.

So that’s it. The class doesn’t assume much. It just picks out what you give it. It’s up to you, the programmer, to interpret those results.

As always, the code is released under the MIT License and available here.

Update: Want to stream your iTunes music over the internet? Try Highwire! It streams your iTunes library and a whole lot more 🙂

I keep all of my iTunes music stored on a Drobo attached to an Airport at home. This frees up valuable space on my laptop and lets me listen to it via Front Row on my TV as well. It’s a much more convenient solution all around.

The only problem is that I lose access to my music when I’m on the road or at work. I’ve seen other services or 3rd party apps that let you stream your music online. They work well enough, but I’ve always thought it silly to pay for a feature that iTunes used to have. In honor of iTunes’ new 8.0 release this morning, here’s my simple workaround.

First off, you’ll need a Mac at home with iTunes open that is always connected to the net. In my case, that’s a Mac Mini in our living room. It also needs to have “Remote Login” (ssh) enabled in System Preferences → Sharing.

Then, poke a hole in your router’s firewall to that machine on TCP port 3689. Here’s a screenshot of my router’s settings:

(Note: your computer’s IP address might be different than mine.)

Then, with that done, anytime you want to listen to your music elsewhere, run these two commands in Terminal on the Mac you’re listening with — not your Mac at home.

ssh your_username@your-home-ip-address -N -f -L 3689:your-home-ip-address:3689

That creates a tunnel from port 3689 on your local machine to port 3689 on your Mac at home. (That’s why you needed to open the hole in your firewall.)

mDNSProxyResponderPosix 127.0.0.1 squeal "My Music" _daap._tcp. 3689 &

That creates an iTunes Bonjour broadcast notification locally which points back to your Mac at home. In other words, it tricks your copy of iTunes into thinking there’s a Mac nearby sharing its iTunes library. When iTunes tries to connect, the traffic is automatically rerouted to your other Mac.

Sidenote: if you don’t have the mDNSProxyResponderPosix command installed on your Mac, I have download links at the bottom of this post.

So that’s it. No need for third party software. When you open iTunes, you should see your music library appear in the “Shared” section of the sidebar. (Much to my excitement, the newly announced Genius playlists appear as well!) You can close the Terminal window once you’ve connected to your music library. (The mDNSProxyResponder command needs to stay active until then.)

Making it Better

To speed things up a bit, I’d put those two command into a shell script and place it in your ~/Library/Scripts folder. I call mine music-tunnel. That way, you can run that one command and have everything up and running automatically.

Or, if you’re feeling adventurous, you could wrap the whole thing inside a simple Cocoa app with a nice On / Off button. But that’s a project for another day…

Downloading mDNSProxyResponderPosix

Here’s a binary download of the mDNSProxyResponderPosix command for Intel. Place it somewhere in your path. And here’s the source if you’d like to compile it yourself for PowerPC.

I think everyone has a core group of artists they grew up listening to — that made an indelible mark on their lives. LeRoi Moore was one of mine.

If you’re lucky enough to have a Grand Central account, here’s a quick PHP class that will login to your account and dial a phone number. This is probably one of the more random bits of code I’ve ever written, but I think it’s useful.

<?PHP
    $gc = new GrandCentral('gc_username', 'gc_password');
    $gc->call($your_number, $their_number);

And that’s it. Grand Central will call $yournumber and connect you to $theirnumber.

So, you can dial phone numbers from PHP. Now what?

Well, for starters, here’s an AppleScript Address Book plugin that will let you dial a contact from your Mac. Or, if you’re like me and have lots of conference calls each week, you can attach a script to your iCal events to dial you into the conference number a minute or two prior to the call start time.

As usual, the code is MIT Licensed and available in GitHub.

Contact me if you have any questions.

Astonishing.

One of the major challenges in web scraping is figuring out which page to scrape in the first place. Here’s a scenario: Say you need to pull some information for the film 30 Days of Night off IMDB. It would be great if you knew in advance what the URL was — something you could construct programatically — unfortunately, it’s actually http://www.imdb.com/title/tt0389722/. How can you possibly figure that out?

One solution would be to scrape IMDB’s built-in search feature and from there extract the correct URL. For IMDB, that works, but what about a site that doesn’t have a search feature? Or one that does, but it doesn’t work very well?

I’ve been web scraping for years and the hands-down, best solution I’ve come up with is to simply let Google do the work for you.

The trick is to take advantage of their “I’m Feeling Lucky” feature. Clicking that button, instead of the standard “Google Search” button, skips the results page and takes you directly to the first result. If you construct your query properly, it will almost always be the page you’re looking for.

Going back to the IMDB example, if you run an I’m Feeling Lucky search for “site:imdb.com $movie_title”, Google will send you a 302 Redirect to the appropriate page within IMDB. Voila! Not only does this get us where we need to be, but (since we’re relying on Google’s ever improving search index) it will also adjust for spelling mistakes or even partial movie titles. It’s a great technique for scraping any sort of online “encyclopedia” like IMDB, TVRage, Wikipedia, etc.

Here’s the code. Pass it a search query and it’ll extract the redirect Google sends back.

<?PHP
    function feelingLucky($q)
    {
        ob_start();
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, "http://www.google.com/search?hl=en&q=" . urlencode($q) . "&btnI=I%27m+Feeling+Lucky");
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
        curl_setopt($ch, CURLOPT_HEADER, 1);
        curl_setopt($ch, CURLOPT_NOBODY, 1);
        curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1) Gecko/20061024 BonEcho/2.0");
        curl_setopt($ch, CURLOPT_REFERER, "http://www.google.com");
        curl_exec($ch);
        curl_close($ch);
        $head = ob_get_contents();
        ob_end_clean();
        return (preg_match('/Location:(.*?)$/ms', $head, $matches) == 0) ? false : trim($matches[1]);
    }

I don’t know how I ever missed this Apple Mail plugin, but you absolutely have to give MsgFiler a try if you’re a heavy keyboard user. It lets you move messages into any folder in your mailbox using only the keyboard.

Press ⌘9 to pull up a TextMate-like list of your mailbox folders. Then, select a folder by typing the first few letters in its name and move the currently selected message(s) into it with ↩.

You can also jump to a folder with ⌘O instead of ↩. Awesome.

Today I’m proud to announce the release of two new open source projects: Appcaster and OpenFeedback. I’ve been working on them off and on for over nine months, so I’m very excited to finally see them out the door.

Appcaster, which I’ve written about before, is a web-based dashboard for indie Mac developers. It’s designed to manage payment and order processing and generate license files for your users. It even handles your product’s revision history in Amazon S3 and can produce reports from your users’ demographic info. It also serves as a central location to collect user feedback, bug reports, and support questions.

OpenFeedback is a Cocoa framework written in Objective-C that collects feedback from your users directly within your application. Instead of sending your users to a website or asking them to write an email, OpenFeedback gives them a simple window where they can ask support questions, file bug reports, or suggest new features. Their data is automatically sent to Appcaster for you to review. They never have to leave your application.

Collectively, I’m calling the two projects Appcaster since they’re designed to work closely with one another (and since I wanted them to be part of the same Google Code project). However, OpenFeedback can send data to any server-side script that accepts HTTP POST requests — you can easily integrate it into your existing bug tracker or reporting system.

Appcaster

When I first built Appcaster last year, I wrote a detailed overview of the application here. Aside from cleaning up a few bugs and upgrading it to use the latest version of the Simple PHP Framework, the only major additions have been adding support for OpenFeedback and graphing user demographic data using the Google Charts API.

Google’s Chart API is such a slick, clever way of doing things that I couldn’t pass by the opportunity to use it in a project. After aggregating your data, you simply craft it into a special URL and use that as the source of an <img> tag. Google parses your data out of the URL and returns a PNG formatted chart.

It took all of half an hour to create some basic stats from the Sparkle update data Appcaster collects. It would be trivial for other developers to add their own custom reports in the future.

OpenFeedback

The idea for OpenFeedback came from Cultured Code‘s task management application Things. Clicking on the Support menu item in their Help menu brings up a dialog where you can submit questions and feedback from inside the app — no need to visit a website or open an email.

I thought their implementation was a great idea and emailed them to ask if I could recreate that functionality as a Cocoa framework for other developers to use. They were nice enough to say yes 🙂

Adding OpenFeedback to your application is trivial. Like Sparkle, there’s no code required. You simply link your app against the framework and hook-up the appropriate actions in Interface Builder. In under five minutes you can have an elegant way to encourage users to provide feedback.

My long term goal is to see the Mac developer community standardize around OpenFeedback much like they have around Sparkle. Not only would it save time for developers, but it would provide users with a consistent interface for submitting feedback. That should help improve the dialogue between developers and our users — improving Mac software all around.

Speaking of Feedback

Your feedback is always welcome. This is my first open source Cocoa project, so I’m very much flying by the seat of my pants. Suggestions, improvements, bug reports — they’re all welcome. You can send them directly to me or file an issue in our bug tracker.

For an upcoming project, I need to pull in metadata about movies and TV shows — genres, plot summaries, actors, etc. The de-facto source is, of course, IMDB. Unfortunately, they’re behind the times and don’t offer an API to access their data. (At least not one that I’ve ever found.)

So, here’s a quick PHP class that takes a movie title (doesn’t have to be exact) or a filename (!) and scrapes IMDB for the relevant info.

Using the scraper is simple.

<?PHP
    $m = new MediaInfo();
    $info = $m->getMovieInfo('American Beauty');
    print_r($info);

will output:

Array
(
    [kind] => movie
    [id] => tt0169547
    [title] => American Beauty
    [rating] => 8.6
    [director] => Sam Mendes
    [release_date] => 1 October 1999
    [plot] => Lester Burnham, a depressed suburban father in a mid-life crisis...
    [genres] => Array
        (
            [0] => Drama
        )
    [cast] => Array
        (
            [Kevin Spacey] => Lester Burnham
            [Annette Bening] => Carolyn Burnham
            [Thora Birch] => Jane Burnham
            [Wes Bentley] => Ricky Fitts
            [Mena Suvari] => Angela Hayes
            [Chris Cooper] => Col. Frank Fitts, USMC
            [Peter Gallagher] => Buddy Kane
            [Allison Janney] => Barbara Fitts
            [Scott Bakula] => Jim Olmeyer
        )
)

At the moment, the class only returns data for movies. For TV shows I’m planning on pulling data directly from the database I’ve created for Schmooze.TV (which, in turn, scrapes its info from TVRage).

You can download the source from my Google Code project. As always, this code is released under the MIT License. Comments and suggestions are always welcome.