Wouldn’t it be awesome if we could use OAuth in JavaScript-only apps? JS is a powerful, expressive programming language, and the browser engines are getting faster and faster all the time. Why not use JavaScript to conduct your API calls and parse your data? In many cases, it is unnecessary to maintain a server-side proxy if all it is doing is making API calls for you and hiding your OAuth keys.

Think about this… If you don’t need any server-side processing, your applications suddenly become infinitely scaleable, right? We could host on the cheapest of cheap commodity hosting. Heck, if all we’re doing is serving static HTML/CSS/JS files, just throw it on a CDN like S3 or CloudFiles and pay per GB.

Before you get too excited, realize that there is a fundamental problem with OAuth in JS. Because JavaScript in the browser is “view-source”, you are always forced to expose your consumer key pair, which compromises the security of your application. sigh

For example, when Twitter recently deprecated their Basic Auth services, that left OAuth as the only authentication method. It was supposed to be the death of JS-only Twitter apps. This was unfortunate for quite a few developers who leveraged the browsers ability to do Basic auth, to help with scaling their Twitter apps. I know, I was one of them.

So then I began to think what if you weren’t forced to expose your keys? What if your JS app could talk to any web API out there, in a secure, user-authenticated way?

Is that actually possible? Yup.

Backstory

Unknowingly at the time, my quest for a JS only OAuth app began two years ago.

When TechCrunch covered the launch of my Twitter client, the app pretty quickly died from the traffic they were sending my way. The problem is 90% of it was written in PHP and used a relational database to store waaaaaay to much data. Neither of them were designed to scale to 20k users in just a few minutes. After days of tweaking and optimizing, I finally gave up on the design. I realized I didn’t need PHP to parse the data, or a database to host the data, so I began a rewrite with the goal of removing as much server-side code as possible. I threw away the database, moved off expensive EC2 and onto commodity hosting where it worked great for the next year or so with some occasional tweaking. As hard as I tried, I never thought I’d be able to completely get rid of the backend because I needed a proxy to securely handle the OAuth requests to Twitter. “That’s ok, close enough” I thought.

One day I was reading the Yahoo Query Language documentation, and I came across a section about using YQL’s storage API to hide authentication info to be used in your queries. Ah ha! Could I actually use that for OAuth? I set to find out. I began learning the ins & outs of OAuth, which includes reading RFC 5849: The OAuth 1.0 Protocol many, many times, and staring at the OAuth Authentication Flow diagram for loooooong time. By the end of the weekend, I had successfully modified my recently rewritten Twitter client’s code-base (now YUI3 based) to remove all server-side programming.

Finally! A secure, pure JavaScript solution to OAuth.

Some Prep Work

So let’s crack the code of what is necessary to do OAuth securely in JavaScript.


  • You cannot store your consumer keys inside your JS code. Not even obfuscated. But it has to be stored somewhere web-accessible so your JS code can talk to it.

  • Because of the same-origin policy, that ‘somewhere’ has to be the same domain as your JS app. Unless of course you only rely on HTTP GET, in which case you can do JSONP.

  • Your storage location cannot transmit your consumer key pair back to you. So that means it needs to do the OAuth request on your behalf.


So hmm…. what is web accessible, can talk to APIs, and also has data storage? YQL.

Yahoo Query Language

YQL is an expressive SQL-like language that lets you query, filter, and join data across web servers. Along with YUI, it is by far my favorite product Yahoo has for developers. Both are simply amazing tools. I won’t go into detail on the specifics of what YQL is in this post, and instead point you to slides from one of my recent talks on the subject here (best viewed in Chrome). All you need to know for this post is that you can use it to access any web-accessible API. In the case of this post, we’ll talk to the Twitter API.

So now that we know it is possible, let’s see it in action.

How It Works

First let’s take a look at how you would call your Twitter friends timeline via YQL w/ OAuth. Using my @derektest user, I created a new OAuth app at dev.twitter.com and used the keys it generated for my user/app combo to generate this YQL query.

SELECT  FROM twitter.status.timeline.friends
WHERE oauth_consumer_key = ‘9DiJt6Faw0Dyr61tVOATA’
AND oauth_consumer_secret = ‘XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA’
AND oauth_token = ‘18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’
AND oauth_token_secret = ‘D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’;

So take that query, URL encode it, and throw it into a URL querystring. Like so…
https://query.yahooapis.com/v1/public/yql?q=select%20%20from%20twitter.status.timeline.friends%20where%20oauth_consumer_key%20%3D%20’9DiJt6Faw0Dyr61tVOATA’%20AND%20oauth_consumer_secret%20%3D%20’XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA’%20AND%20oauth_token%20%3D%20’18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’%20and%20oauth_token_secret%20%3D%20’D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’%3B&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

That unique URL will give you a list of the people @derektest follows (which is only @derek). You can play around with the query in the YQL Console, or view the results in an XML feed.

But there’s a problem using that query, because? You guessed it, you’ve exposed your consumer key-pair. So let’s work on hiding those.

First step, turn the embedded parameters into environment variables by using the SET command.

set oauth_consumer_key=’9DiJt6Faw0Dyr61tVOATA’ on twitter;
set oauth_consumer_secret=’XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA’ on twitter;
set oauth_token=’18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’ on twitter;
set oauth_token_secret=’D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’ on twitter;
select * from twitter.status.timeline.friends;

Now that we’ve turned all the parameters into environment variables, the next step is to throw the consumer key pair into YQL’s storage so only YQL can access it.

To do this, create a YQL environment file, similar to this one, http://derekgathright.com/code/yahoo/yql/oauthdemo.txt

As you’ll see, it’s just a regular text file where I pasted my consumer key pair, along with importing the YQL community tables using the ENV command. Since we’re replacing the previously included env file (store://datatables.org/alltableswithkeys) with our own, we need to chain-load it back in because it includes the Twitter tables. If you miss that step, you’ll get a “No definition found for Table twitter.status.timeline.friends“ error.

Before we store the env file in YQL, let’s test it with this new query:

set oauth_token=’18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’ on twitter;
set oauth_token_secret=’D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’ on twitter;
select * from twitter.status.timeline.friends;

Also, you’ll have to change the env file loaded in the querystring to “?env=http://derekgathright.com/code/yahoo/yql/oauthdemo.txt“

(View: YQL Console - Results)

Now that we have our environment file created and tested, let’s tell YQL to import it. To do that, we’ll construct a YQL query similar to:

insert into yql.storage.admin (name,url)
values (“oauthdemo”,”http://derekgathright.com/code/yahoo/yql/oauthdemo.txt")

Which returns:


       store://derekgathright.com/oauthdemo

[hidden]</pre>

You now have 3 keys pointing to your data, and each does something different (think: unix permissions, R/W/X). For more information on what each of the 3 does, Using YQL to Read, Update, and Delete Records.

For this example we want the execute key, which is really just an alias to our stored env file. So if we change our query’s URL to ?env=store://derekgathright.com/oauthdemo and use the same YQL query as last time, you’ll see we have now hidden our consumer key pair from the public.

(View: YQL Console - Results)

Well there you have it, an example of how to hide your consumer key pair, which now allows you to use YQL as your server-side proxy as opposed to writing & maintaining your own!

A Pure JS Twitter Client is Born

When I started at Yahoo, I wanted an excuse to learn YUI3 and expand my knowledge of YQL. So porting my jQuery/PHP based Twitter client seemed like a logical choice. The result of this work is an open-source project I call Tweetanium. I’m not going to argue it is the most polished or feature-rich Twitter client. In fact, it is quite buggy, and will likely always be that way. It’s just something I toy around with occasionally to try out new things. But feel free to use it if you like. You can play around in it at tweetanium.net.

As proof that there is no server-side JS, you can even use a version of it hosted on Github Pages, which is a static file host (no PHP, Ruby, Python, etc…). Hosting off Github Pages was a neat test for it, which basically proves you can host JS-only apps on commodity hosting. If you actually need to process data externally, you can use YQL tables for any APIs on the web, even your own custom-built ones (See: YQL Open Data Tables). Any scaling bottlenecks have now been offloaded to Github and Yahoo. The best part about this solution? It’s free!

Post some comments if you have questions.

UPDATE: A few people have asked, “But can’t I execute YQL queries with your consumer keys now?“ The answer is, yes. But that isn’t as bad as you think because you only have half of the keys necessary. You are missing the unique keys assigned to a user on behalf of my application, and without those, you cannot make authenticated calls. If you get those, well… there’s a whole other security issue of you having physical access to their computer or man-in-the-middle attacks.

“Ok, but can’t I authenticate new keys posing as your app?“ To my knowledge, Twitter does not currently support the oauth_callback parameter, which allows the requester to Twitter to redirect the user to the URL of their choice. So if EvilHacker tries to authenticate InnocentUser using my consumer keys, InnocentUser will just be directed back to my app’s preset URL stored in Twitter’s database. In the future, who knows how the OAuth spec, or Twitter’s implementation of it, will change. This is mostly a proof-of-concept hack at this point.