Creating GitHub Issues (with image attachments!) From an Email

I’m very meticulous about logging all of the feedback I receive from my customers. Whether it’s a bug report or a feature request, I want all of that information captured in a single place where I can plan and act on it. For me, that place is the Issues section in my app’s GitHub repo.

Normally, when I get a customer email, my workflow is to reply back to them with any clarification I need, and then once we’ve finished with any back and forth, create a new GitHub issue with the relevant info from their email and a note reminding me to email them back when their issue is resolved.

This takes all of a minute to do. But it still means opening a browser, navigating to the repo, clicking on “Issues”, then “New Issue”, and copy and paste the email details. Further, if the user supplied any screenshots, I have to save those out of the email and upload them to GitHub as well. Like I said, it only takes a minute or so, but it adds unnecessary friction.

Today I decided to automate all of that.

I use the fantastic Postmark service to send all of my company‘s transactional emails. They also have an equally awesome inbound service that will parse any emails you forward to them and POST the details as a JSON blob to your web hook.

So, I created a forwarding rule in Fastmail to forward any emails sent to product1@mydomain.com and product2@mydomain.com to my secret Postmark inbound address.

Postmark receives the forwarded email and POSTs the data to my server, which runs a small PHP script (embedded below) that downloads any image attachments contained in the email and creates a new GitHub issue in the appropriate repo with the contents of the email and image attachments.

It all works great! What used to be a slightly annoying process to do a couple times a day, now takes all of three seconds in my email client – whether I’m at my desktop or out and about on my phone.

Maybe you’ll find this script helpful.

A Simple, Open-Source URL Shortener

tl;dr One evening last week, I built pretty much the simplest URL shortening service possible. It’s simple, fast, opinionated, keeps track of click-thru stats, and does everything I need. It’s all self-contained in a single PHP script (and .htaccess file). No dependencies, no frameworks to install, etc. Just upload the file to your web server and you’re done. Maybe you’ll find it useful, too.

Anyway…

I run a small software company which sells macOS and iOS software. Part of my day-to-day in running the business is replying to customer support questions – over email and, sometimes, SMS/chat. I often need to reply to my customers with long URLs to support documents or supply them with custom-URL-scheme links which they can click on to deep-link them into a specific area of an app.

Long and non-standard URLs can often break once sent to a customer or subsequently forwarded around. I’ve used traditional link shortening services before (like bit.ly, etc), but always worried about my URLs expiring or breaking if the 3rd party shortening service goes out of business or makes a system change. Even if I upgraded to a paid plan which supports using a custom domain name that I own, I’m still not fully in control of my data.

So, I looked around for open-source URL shortening projects which I could install on my own web server and bend to my will. I found quite a few, but most were either outdated or overly-complex with tons of dependencies on various web frameworks, libraries, etc. I wanted something that would play nicely with a standard LAMP stack so I could drop it onto one of my web servers without having to boot up an entirely new VPS just to avoid port 80/443 conflicts with Apache. Out of the question was anything requiring a dumb, container-based (I see you, Docker) solution just to get started. Nice-to-haves would be offering basic click-thru statistics and an easy way to script the service into my existing business tools and workflows.

Admittedly, I only spent about an hour looking around, but I didn’t find anything that met my needs. So, I spent an evening hacking together this project to do exactly what I wanted, in the simplest way possible, and without any significant dependencies. The result is a branded URL shortening service I can use with my customers that’s simple to use and also integrates with my company’s existing support tools (because of its URL-based API and (optional) JSON responses – see below).

Requirements

  • Apache2 with mod_rewrite enabled
  • PHP 5.4+ or 7+
  • A recent version of MySQL

Install

  1. Clone this repo into the top-level directory of your website on a PHP enabled Apache2 server.
  2. Import database.sql into a MySQL database.
  3. Edit the database settings at the top of index.php. You may also edit additional settings such as the length of the short url generated, the allowed characters in the short URL, or set a password to prevent anyone from creating links or viewing statistics about links.

Note: This project relies on the mod_rewrite rules contained in the .htaccess file. Some web servers (on shared web hosts for example) may not always process .htaccess files by default. If you’re getting 404 errors when trying to use the service, this is probably why. You’ll need to contact your server administrator to enable .htaccess files. Here’s more information about the topic if you’re technically inclined.

Creating a New Short Link

To create a new short link, just append the full URL you want to shorten to the end of the domain name you installed this project onto. For example, if your shortening service was hosted at https://example.com and you want to shorten the URL https://some-website.com, go to https://example.com/http://somewebsite.com. If all goes well, a plain-text shortened URL will be displayed. Visiting that shortened URL will redirect you to the original URL.

Possibly of interest to app developers like myself: The shortening service also supports URLs of any scheme – not just HTTP and HTTPS. This means you can shorten URLs like app://whatever, where app:// is the URL scheme belonging to your mobile/desktop software. This is useful for deep-linking customers directly into your app.

iOS Users: If you have Apple’s Shortcuts.app installed on your device, you can click this link to import a ready-made shortcut that will let you automatically shorten the URL on your iOS clipboard and replace it with the generated short link.

Viewing Click-Thru Statistics

All visits to your shortened links are tracked. No personally identifiable user information is logged, however. You can view a summary of your recent link activity by going to /stats/ on the domain hosting your link shortener.

You can click the “View Stats” link to view more detailed statistics about a specific short link.

Password Protecting Creating Links

If you don’t want to leave your shortening service wide-open for anyone to create a new link, you can optionally set a password by assigning a value to the $pw_create variable at the top of index.php. You will then need to pass in that password as part of the URL when creating a new link like so:

Create link with no password set: http://example.com/http://domain.com

Create link with password set: http://example.com/your-password/http://domain.com

Password Protecting Stats

Your stats pages can also be password protected. Just set the $pw_stats variable at the top of the index.php file.

Viewing stats with no password set: http://example.com/stats

Viewing stats with password set: http://example.com/stats/your-password

A Kinda-Sorta JSON API

This project aims to be as simple-to-use as possible by making all commands and interactions go through a simple URL-based API which returns plain-text or HTML. However, if you’re looking to run a script against the shortening service, you can do so. Just pass along Accept: application/json in your HTTP headers and the service will return all of its output as JSON data – including the stats pages.

Contributions / Pull Requests / Bug Reports

Bug fixes, new features, and improvements are welcome from anyone. Feel free to open an issue or submit a pull request.

I consider the current state of the project to be feature-complete for my needs and am not looking to add additional features with heavy dependencies or that complicate the simple install process. That said, I’m more than happy to look at any new features or changes you think would make the project better. Feel free to get in touch.

Rockwell – Sort of like a private Foursquare meets Fire Eagle

Back in 2008, when I worked for Yahoo!, I had the good fortune of chatting with Tom Coates a few times about the now defunct Fire Eagle location brokerage service. Fire Eagle was my absolute favorite product to come out of Yahoo! during my time there. I’ve always been fascinated by real-time location data and sad that Fire Eagle’s intersection of privacy and ease of use never caught on.

Anyway, fast-forward to last Summer, I was bummed that although so much of my life is documented through photos, tweets, and journal entries, my location data was missing. A few products tried to solve this problem post-Fire Eagle. Google Latitude (née Dodgeball) gave you a way to seamlessly track your location and view your history, but they’ve since sunsetted that product. And, besides, it was a little creepy having Google track your every step. (Which I realize they still are via Google Now, but that’s another conversation.) There was also other apps like Rove and Foursquare, but none of them offered quite the feature set I was looking for. In 2010, I even went so far as to reverse engineer Apple’s Find My iPhone API so I could remotely query my phone’s location. That worked great, but doing so on a regular basis killed my battery life. But, with iOS 7’s advances in background location reporting, I knew there had to be a better way. I wanted something that would automatically track my location as precisely as possible, respect my phone’s battery life, keep that data private by default, yet still offer the ability to granularly share my location as I saw fit.

So I did what I always seem to do. I built my own app. It’s called Rockwell, and it’s available on GitHub.

Rockwell consists of two components. An iPhone app that monitors your location and allows you to annotate where you are using Foursquare’s database of named locations. And a PHP/MySQL web service you can install on your own server that the app talks to.

As you go about your day, the iPhone app uses iOS’ significant location change reporting feature to ping the web app with your location. The web app stores your location history and allows you to go back and search your history either by date or by location.

Further, since the website knows your most recent (current) location, you’re able to share that with others. In my opinion, one of the reasons Fire Eagle failed was it was (by design) very difficult to get location data out of the service. You had to go through an intricate OAuth dance each time.

With Rockwell, you simply choose the location level you’re comfortable sharing – either precise lat/lng coordinates, current city, current state, etc – and a format – plain text or JSON – and Rockwell generates a short url you can pass around. You can use that URL to embed your plain text location on your blog, or you can use the JSON version to do something more API-ish with it. There’s no OAuth shenanigans to deal with. You can have as many short geo-links as you want, and each one can be revoked at any time.

One more thing I’d like to explain. The iPhone app reports two types of check-in data to the web service. The first kind is dumb. Just your latitude, longitude, and timestamp. Many apps like Rove and Foursquare use this data to try and generate automatic location check-ins. Based on your past history, your friends’ location, and your location, they try and guess where you might actually be at a given time. Doing this well is the holy grail of location reporting. The problem is that I’ve yet to see any service get it right. In a dense urban area, with hundreds if not thousands of locations per square mile, there’s just no reliable way to figure out where you really are with precision. (At least not without some serious machine learning, which I assume Foursquare is working on.) Rockwell dances around this problem by allowing you to augment your dumb check-ins with annotated ones. Just launch the app, tap “Check-in”, and Rockwell pulls a list of nearby locations from Foursquare. Just tap on one and you’re done. It’s saved to the web service alongside the rest of your location history.

Rockwell is working great for me currently. All the basics work. The majority of the remaining work is around coming up with nice ways to show your location history in a useful way. The code is available on GitHub. I’d love it if you gave it a try, sent feedback, and maybe even a pull request or two.

Importing Jekyll Posts into WordPress

Nearly four years ago I switched my main site over to Jekyll. It’s been great. But late last year I decided to make that site and its blog purely about my software business and move all of my non-work posts over to my tyler.io domain so I could have a personal site again. To encourage myself to write more, I built the site with WordPress so it would be easy to publish. That meant I needed a way to convert and import all of my old Jekyll Markdown posts into WordPress. I found a few scripts that exported WordPress into Jekyll, but not the other way around. So I hacked together my own script, which I’ve pasted below. Hopefully this will help anyone wanting to make the same transition.

The script takes a directory of Markdown posts in the following format, reads their header meta-data, and imports them into your WordPress database.

date: 2013-04-08 20:57:14
title: PebbleCam
layout: post
permalink: /blog/2013/04/pebblecam/index.html
slug: pebblecam
---
Post content...

Automatically Compressing Your Amazon S3 Images Using Yahoo!’s Smush.it Service

I’m totally obsessed with web site performance. It’s one of those nerd niches that really appeal to me. I’ve blogged a few times previously on the topic. Two years ago, (has it really been that long?) I talked about my experiences rebuilding this site following the best practices of YSlow. A few days later I went into detail about how to host and optimize your static content using Amazon S3 as a content delivery network. Later, I took all the techniques I had learned and automated them with a command line tool called s3up. It’s the easiest way to intelligently store your static content in Amazon’s cloud. It sets all the appropriate headers, gzips your data when possible, and even runs your images through Yahoo!’s Smush.it service.

Today I’m pleased to release another part of my deployment tool chain called Autosmush. Think of it as a reverse s3up. Instead of taking local images, smushing them, and then uploading to Amazon, Autosmush scans your S3 bucket, runs each file through Smush.it, and replaces your images with their compressed versions.

This might sound a little bizarre (usless?) at first, but it has done wonders for mine and one of my freelance client’s workflows. This particular client runs a network of very image-heavy sites. Compressing their images has a huge impact on their page load speed and bandwidth costs. The majority of their content comes from a small army of freelance bloggers who submit images along with their posts via WordPress, which then stores them in S3. It would be great if the writers had the technical know-how to optimize their images beforehand, but that’s not reasonable. To fix this, Autosmush scans all the content in their S3 account every night, looking for new, un-smushed images and compresses them.

Autosmush also allowed me to compress the huge backlog of existing images in my Amazon account that I had uploaded prior to using Smush.it.

If you’re interested in giving Autosmush a try, the full source is available on GitHub. You can even run it in a dry-run mode if you’d just like to see a summary of the space you could be saving.

Also, for those of you with giant S3 image libraries, I should point out that Autosmush appends an x-amz-smushed HTTP header to every image it compresses (or images that can’t be compressed further). This lets the script scan extremely quickly through your files, only sending new images to Smush.it and skipping ones it has already processed.

Head on over to the GitHub project page and give Autosmush a try. And please do send in your feedback.

Generating Strong, User Friendly Passwords in PHP

For an upcoming project, I needed a quick PHP function that would generate strong passwords. It’s an easy problem on the surface, but it has some quirky nuances that appear if you spend any length of time thinking about it.

For example, it’s not enough to merely pick characters at random — you have to include at least one character from each set (lowercase, uppercase, digits, symbols) to minimize the chances of someone guessing the password.

Another problem are ambiguous characters. Many times, users won’t (or can’t) cut and paste the password you generate for them. This happens quite often on mobile devices. They’ll manually transcribe the password from an email or even read it aloud. Differentiating between I, l, 1, 0, and O can be a nightmare — for them and for you when they email support because their password won’t work.

Complicating matters further, long, random passwords are often mistyped because users have to look back once or twice while typing the password. For long strings, it’s easy to lose your place and duplicate or leave out a character or two.

The method I finally decided on solves each of these problems. It generates a strong password of a specified length, without any ambiguous characters, and can optionally include dashes between groups of characters to help users retain their place. You can also customize which sets of characters the password will contain, e.g. alphanumeric, no uppercase letters, etc.

The code is hosted on Github. Go forth and fork.

<?PHP
// Generates a strong password of N length containing at least one lower case letter,
// one uppercase letter, one digit, and one special character. The remaining characters
// in the password are chosen at random from those four sets.
//
// The available characters in each set are user friendly - there are no ambiguous
// characters such as i, l, 1, o, 0, etc. This, coupled with the $add_dashes option,
// makes it much easier for users to manually type or speak their passwords.
//
// Note: the $add_dashes option will increase the length of the password by
// floor(sqrt(N)) characters.

function generateStrongPassword($length = 9, $add_dashes = false, $available_sets = 'luds')
{
    $sets = array();
    if(strpos($available_sets, 'l') !== false)
        $sets[] = 'abcdefghjkmnpqrstuvwxyz';
    if(strpos($available_sets, 'u') !== false)
        $sets[] = 'ABCDEFGHJKMNPQRSTUVWXYZ';
    if(strpos($available_sets, 'd') !== false)
        $sets[] = '23456789';
    if(strpos($available_sets, 's') !== false)
        $sets[] = '!@#$%&*?';

    $all = '';
    $password = '';
    foreach($sets as $set)
    {
        $password .= $set[array_rand(str_split($set))];
        $all .= $set;
    }

    $all = str_split($all);
    for($i = 0; $i < $length - count($sets); $i++)
        $password .= $all[array_rand($all)];

    $password = str_shuffle($password);

    if(!$add_dashes)
        return $password;

    $dash_len = floor(sqrt($length));
    $dash_str = '';
    while(strlen($password) > $dash_len)
    {
        $dash_str .= substr($password, 0, $dash_len) . '-';
        $password = substr($password, $dash_len);
    }
    $dash_str .= $password;
    return $dash_str;
}
Posted in PHP

Open Source Updates

On this lazy Sunday afternoon I thought I’d take the opportunity to mention a few open source projects I’ve recently updated. GitHub makes sharing code so ridiculously easy, it’s a shame not to call attention to it occasionally in case other people might find something useful.

Sosumi 2.0

First up is Sosumi 2.0. Last year, when Apple launched the Find My iPhone component of MobileMe, I immediately saw an opportunity to grab persistent location information from my phone — without background processing. Although Apple didn’t supply an API for this information, it turned out to be easy enough to scrape their site and wrap it up nicely into a PHP class. Nat Friedman even used it as a way to automatically update his Google Latitude position in his playnice project, and I built a similar script for Yahoo!’s Fire Eagle service. It all worked well enough, but it was slow and prone to breaking whenever Apple updated me.com.

Fast forward to last week, Apple released an official Find My iPhone client for iPhone and iPad. The mere fact that they released this means there had to be a hidden “official” API somewhere. After a few hours messing around in WireShark I found their API end point and re-wrote Sosumi to talk to their API just like the client app. The result? Dramatically faster location updates (10x) and a solid script that’s immune to changes on MobileMe’s website.

This new version of Sosumi is available on GitHub and extremely easy to use:

<?PHP
include 'class.sosumi.php';
$ssm = new Sosumi('your-username', 'your-password');
$location = $ssm->locate();

That’s it. $location will be an array populated with your phone’s latitude, longitude, and a few other useful data points. What you do with this information is up to you!

PHP HTML Compressor

Like the name says, this project is a small PHP class that accepts an HTML document and minifies its filesize by removing unnecessary whitespace and blank lines. It takes care not to touch fragile areas like <pre> blocks. The result is HTML that renders exactly the same in the browser but (in my testing) can be up to 15% smaller. In today’s increasingly mobile world, every byte over the wire counts — and this is a simple way to speed up your page load times.

The compressor can be used in three ways:

  1. Pass it an string containing HTML and it’ll return the minified version.
  2. As full fledged command line utility. Pass it a filename or pipe content to it via stdin and it will send the minified version back over stdout. This is super useful for adding automatic compression into your deploy/build scripts.
  3. Or as a WordPress plugin that automatically minifies all of your posts and pages. Combine it with wp-super-cache and you’re well on your way to a speedy site — even on a shared host.

For an example of the type of HTML the compressor produces, just take a look at the HTML source of this site. Every page is piped through the compressor before being saved as a static file on my server.

Google Search Shell

My google-shell project is another small command line utility. It’s a simple interface to Google’s search results that talks to their AJAX Search API. It lets you easily pull down the top results for any query — including the result’s URL, title, and a brief abstract from the page. It has quite a few options that allow you to customize the output to be either human readable or digestible to other scripts. For example, here’s an ugly, ugly shell command that shows off the power of what having Google at your fingertips can do:

URL=`gshell -fun1 "imdb american beauty"`; curl $URL | \
sed -n 's/.*\([0-9]\.[0-9]\)\/10.*/\1/p' | head -n1

In case you don’t speak nerd, that tells google-shell to return only the URL of the first result for the query “imdb american beauty”. In other words, the same thing as Google’s “I’m Feeling Lucky” option. It then takes that URL, downloads it, and pipes it through a messy sed and head command that extracts the IMDB rating for American Beauty. Granted, that’s quite a lot to type — especially considering you could open a web browser and google it yourself much faster. However, if you were to add that long command as an alias in your Bash profile. you could very quickly write a command like

imdb "american beauty"

That would instantly return you the rating of whichever movie you specify. Nerdy, but cool, right?

Anyway

As always, the three projects above and all my open source code are available on GitHub. Hopefully you’ll find something useful. If you do, I’d love to hear about it — and I always welcome bug fixes and other contributions.

Shine – An Indie Mac Dashboard

Two years ago, shortly after I released VirtualHostX 1.0, I wrote about Appcaster – a web dashboard for Mac developers I built that manages my application updates, payment processing, etc. With the release of VHX 2.0 and Incoming!, I decided it was time to rewrite Appcaster as the original code was hurried and hastily patched over the last few years.

Today I’m happy to officially announce Shine, a revamped version of Appcaster re-written from the ground up. The goal of Shine (more on the name in a bit) was to provide clean, easy to use dashboard for indie Mac developers and also to build a stable foundation that provides for future improvement down the road.

I chose the name Shine because, at it’s heart, it’s a complimentary product to Andy Matuschak’s Sparkle project. (Inevitable tagline: Your app already Sparkles, now make it Shine.) The core functionality, like Appcaster before it, is to automatically generate appcast feeds for your product updates. But it does a whole lot more, too.

Shine Screenshot

Shine handles order processing using PayPal’s IPN service. That includes generating the license information (using either Aquatic Prime or your own, custom scheme), emailing it to the user, and managing the database of orders. It also computes aggregate stats based on your users’ Sparkle update requests, collects user feedback (bug reports, feature requests, questions), and automatically stores your application updates in Amazon S3.

In short, Shine manages my entire Indie Mac developer workflow.

The code is based on two of my other open source projects: the Simple PHP Framework and YUI App Theme. SPF provides a clean, lightweight, active record pattern to model the data, and yui-app-theme is an admin area CSS template built on top of the YUI Grids framework. Combining these two projects let me build Shine in record time (about 24 working hours).

The code for Shine is free to use (MIT license) and available on GitHub. Feel free to email me with any questions or feedback.

(Thanks to Steven Degutis of Thoughtful Tree software for his feedback on this project.)

Persistant Location Updates From iPhone to Fire Eagle

Location Based Services are hot. They add an extra layer of usefulness on top of the web sites and products we’re already using. The trick is keeping your location updated in the cloud as frequently, comfortably, and securely as possible.

Fire Eagle fulfills the security requirement — brokering your whereabouts only to parties you’ve authorized. And iPhone applications like Sparrow and Voila make it a cinch to update Fire Eagle on-the-go. But they’re limited to manual updates as they can’t run in the background. (And you wouldn’t want them to because of the drain GPS has on battery life.)

For me, the holy grail has always been a way to update your location persistently from iPhone. And until Apple offers their own solution (fingers crossed) I’d like to present mine. It’s a dirty hack (the best always are), and has the added benefit of working with any AT&T phone — not just iPhone.

To do this, we’ll be scraping AT&T’s new Family Map service and then pushing the data we retrieve into Fire Eagle ourselves.

(Family Map is an overpriced add-on to your monthly plan that lets you track the phones on your account using AT&T’s website. It’s limited, but surprisingly good considering it came from within the bowels of a cellphone company.)

There will be three parts to this hack.

  1. Scraping our location data from AT&T’s website.
  2. Pushing that data to Fire Eagle.
  3. Making the script run automatically.

Let’s get started.

Scraping the Data

The Family Map website uses Microsoft’s VirtualEarth maps plus some other AJAXy fanciness. I briefly poked around to see if there were any JSON or XML data sources I could hijack but didn’t see anything. Instead, I opted to directly scrape their mobile website as it’s plain vanilla HTML.

I won’t go into the details of scraping the data (you can see the code for yourself), but it was pretty simple. Login, send a “locate my phone” request, wait for the data, and parse out our coordinates.

Pushing Data to Fire Eagle

Fire Eagle is an excellent API to work with. They’ve got a clear spec and tons of example code. The only tricky part is handling the initial OAuth setup. I’ve included a simple web page (setup.php) you can use to do the authentication. It’s based on Fire Eagle’s PHP API kit example.

Once OAuth is setup, it’s only one line of code to publish our location.

Making it Automatic

Running this script automatically will vary depending on your setup. In my case, I’ve created a cron jon that runs the included update.php script every five minutes.

Download the Code

And that’s it. This code is only a few hours old, but it seems to work well so far. I watched as Fire Eagle was updated with my location this morning on the way to work. That said, it’s definitely not user friendly — clearly something only someone familiar with a command line would want to setup. But if you’re interested in developing a friendlier solution, let me know and perhaps we can work together.

Grab the code from GitHub.