How-to: Secure OAuth in JavaScript

Wouldn’t it be awesome if we could use OAuth in JavaScript-only apps? JS is a powerful, expressive programming language, and the browser engines are getting faster and faster all the time. Why not use JavaScript to conduct your API calls and parse your data? In many cases, it is unnecessary to maintain a server-side proxy if all it is doing is making API calls for you and hiding your OAuth keys.

Think about this… If you don’t need any server-side processing, your applications suddenly become infinitely scaleable, right? We could host on the cheapest of cheap commodity hosting. Heck, if all we’re doing is serving static HTML/CSS/JS files, just throw it on a CDN like S3 or CloudFiles and pay per GB.

Before you get too excited, realize that there is a fundamental problem with OAuth in JS. Because JavaScript in the browser is “view-source”, you are always forced to expose your consumer key pair, which compromises the security of your application. *sigh*

For example, when Twitter recently deprecated their Basic Auth services, that left OAuth as the only authentication method. It was supposed to be the death of JS-only Twitter apps. This was unfortunate for quite a few developers who leveraged the browsers ability to do Basic auth, to help with scaling their Twitter apps. I know, I was one of them.

So then I began to think what if you weren’t forced to expose your keys? What if your JS app could talk to any web API out there, in a secure, user-authenticated way?

Is that actually possible? Yup.

Backstory

Unknowingly at the time, my quest for a JS only OAuth app began two years ago.

When TechCrunch covered the launch of my Twitter client, the app pretty quickly died from the traffic they were sending my way. The problem is 90% of it was written in PHP and used a relational database to store waaaaaay to much data. Neither of them were designed to scale to 20k users in just a few minutes. After days of tweaking and optimizing, I finally gave up on the design. I realized I didn’t need PHP to parse the data, or a database to host the data, so I began a rewrite with the goal of removing as much server-side code as possible. I threw away the database, moved off expensive EC2 and onto commodity hosting where it worked great for the next year or so with some occasional tweaking. As hard as I tried, I never thought I’d be able to completely get rid of the backend because I needed a proxy to securely handle the OAuth requests to Twitter. “That’s ok, close enough” I thought.

One day I was reading the Yahoo Query Language documentation, and I came across a section about using YQL’s storage API to hide authentication info to be used in your queries. Ah ha! Could I actually use that for OAuth? I set to find out. I began learning the ins & outs of OAuth, which includes reading RFC 5849: The OAuth 1.0 Protocol many, many times, and staring at the OAuth Authentication Flow diagram for loooooong time. By the end of the weekend, I had successfully modified my recently rewritten Twitter client’s code-base (now YUI3 based) to remove all server-side programming.

Finally! A secure, pure JavaScript solution to OAuth.

Some Prep Work

So let’s crack the code of what is necessary to do OAuth securely in JavaScript.

  • You cannot store your consumer keys inside your JS code. Not even obfuscated. But it has to be stored somewhere web-accessible so your JS code can talk to it.
  • Because of the same-origin policy, that ‘somewhere’ has to be the same domain as your JS app. Unless of course you only rely on HTTP GET, in which case you can do JSONP.
  • Your storage location cannot transmit your consumer key pair back to you. So that means it needs to do the OAuth request on your behalf.

So hmm…. what is web accessible, can talk to APIs, and also has data storage? YQL.

Yahoo Query Language

YQL is an expressive SQL-like language that lets you query, filter, and join data across web servers. Along with YUI, it is by far my favorite product Yahoo has for developers. Both are simply amazing tools. I won’t go into detail on the specifics of what YQL is in this post, and instead point you to slides from one of my recent talks on the subject here (best viewed in Chrome). All you need to know for this post is that you can use it to access any web-accessible API. In the case of this post, we’ll talk to the Twitter API.

So now that we know it is possible, let’s see it in action.

How It Works

First let’s take a look at how you would call your Twitter friends timeline via YQL w/ OAuth. Using my @derektest user, I created a new OAuth app at dev.twitter.com and used the keys it generated for my user/app combo to generate this YQL query.

SELECT * FROM twitter.STATUS.timeline.friends
WHERE oauth_consumer_key = '9DiJt6Faw0Dyr61tVOATA'
AND oauth_consumer_secret = 'XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA'
AND oauth_token = '18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y'
AND oauth_token_secret = 'D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI';

So take that query, URL encode it, and throw it into a URL querystring. Like so…

https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20twitter.status.timeline.friends%20where%20oauth_consumer_key%20%3D%20’9DiJt6Faw0Dyr61tVOATA’%20AND%20oauth_consumer_secret%20%3D%20′XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA’%20AND%20oauth_token%20%3D%20’18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’%20and%20oauth_token_secret%20%3D%20′D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’%3B&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

That unique URL will give you a list of the people @derektest follows (which is only @derek). You can play around with the query in the YQL Console, or view the results in an XML feed.

But there’s a problem using that query, because? You guessed it, you’ve exposed your consumer key-pair. So let’s work on hiding those.

First step, turn the embedded parameters into environment variables by using the SET command.

SET oauth_consumer_key='9DiJt6Faw0Dyr61tVOATA' ON twitter;
SET oauth_consumer_secret='XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA' ON twitter;
SET oauth_token='18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y' ON twitter;
SET oauth_token_secret='D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI' ON twitter;
SELECT * FROM twitter.STATUS.timeline.friends;

Now that we’ve turned all the parameters into environment variables, the next step is to throw the consumer key pair into YQL’s storage so only YQL can access it.

To do this, create a YQL environment file, similar to this one, http://derekgathright.com/code/yahoo/yql/oauthdemo.txt

As you’ll see, it’s just a regular text file where I pasted my consumer key pair, along with importing the YQL community tables using the ENV command. Since we’re replacing the previously included env file (store://datatables.org/alltableswithkeys) with our own, we need to chain-load it back in because it includes the Twitter tables. If you miss that step, you’ll get a “No definition found for Table twitter.status.timeline.friends” error.

Before we store the env file in YQL, let’s test it with this new query:

SET oauth_token='18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y' ON twitter;
SET oauth_token_secret='D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI' ON twitter;
SELECT * FROM twitter.STATUS.timeline.friends;

Also, you’ll have to change the env file loaded in the querystring to “?env=http://derekgathright.com/code/yahoo/yql/oauthdemo.txt

(View: YQL ConsoleResults)

Now that we have our environment file created and tested, let’s tell YQL to import it. To do that, we’ll construct a YQL query similar to:

INSERT INTO yql.storage.admin (name,url)
VALUES ("oauthdemo","http://derekgathright.com/code/yahoo/yql/oauthdemo.txt")

Which returns:

       store://derekgathright.com/oauthdemo
<select>store://VfoIoYWhLWLxYzRTcrbvNb</select>
 
       [hidden]

You now have 3 keys pointing to your data, and each does something different (think: unix permissions, R/W/X). For more information on what each of the 3 does, Using YQL to Read, Update, and Delete Records.

For this example we want the execute key, which is really just an alias to our stored env file. So if we change our query’s URL to ?env=store://derekgathright.com/oauthdemo and use the same YQL query as last time, you’ll see we have now hidden our consumer key pair from the public.

(View: YQL ConsoleResults)

Well there you have it, an example of how to hide your consumer key pair, which now allows you to use YQL as your server-side proxy as opposed to writing & maintaining your own!

A Pure JS Twitter Client is Born

When I started at Yahoo, I wanted an excuse to learn YUI3 and expand my knowledge of YQL. So porting my jQuery/PHP based Twitter client seemed like a logical choice. The result of this work is an open-source project I call Tweetanium. I’m not going to argue it is the most polished or feature-rich Twitter client. In fact, it is quite buggy, and will likely always be that way. It’s just something I toy around with occasionally to try out new things. But feel free to use it if you like. You can play around in it at tweetanium.net.

As proof that there is no server-side JS, you can even use a version of it hosted on Github Pages, which is a static file host (no PHP, Ruby, Python, etc…). Hosting off Github Pages was a neat test for it, which basically proves you can host JS-only apps on commodity hosting. If you actually need to process data externally, you can use YQL tables for any APIs on the web, even your own custom-built ones (See: YQL Open Data Tables). Any scaling bottlenecks have now been offloaded to Github and Yahoo. The best part about this solution? It’s free!

Post some comments if you have questions.

UPDATE: A few people have asked, “But can’t I execute YQL queries with your consumer keys now?” The answer is, yes. But that isn’t as bad as you think because you only have half of the keys necessary. You are missing the unique keys assigned to a user on behalf of my application, and without those, you cannot make authenticated calls. If you get those, well… there’s a whole other security issue of you having physical access to their computer or man-in-the-middle attacks.

Ok, but can’t I authenticate new keys posing as your app?” To my knowledge, Twitter does not currently support the oauth_callback parameter, which allows the requester to Twitter to redirect the user to the URL of their choice. So if EvilHacker tries to authenticate InnocentUser using my consumer keys, InnocentUser will just be directed back to my app’s preset URL stored in Twitter’s database. In the future, who knows how the OAuth spec, or Twitter’s implementation of it, will change. This is mostly a proof-of-concept hack at this point.

  • http://kentbrewster.com Kent Brewster

    Nicely done, Derek; please see if they’ll link to this from the YQL/YDN blog. One tiny quibble: I’m much more likely to try out a GitHub-hosted example that does not ask for write permissions to my Twitter stream.

  • Eugene

    Holy Crap.
    All that’s left is a way to pay Yahoo for higher rate limits.

  • Derek

    Kent, thanks. Will keep you suggestion in mind for future hacks. Good point.

    Eugene, YQL’s current rate limit is 100k calls per day (if you call w/ your access keys). If you expect you need to go over that, try sending an email to yql-questions@yahoo-inc.com. More info @ http://developer.yahoo.com/yql/guide/usage_info_limits.html

  • Eugene

    Derek, I’m not at the rate limit yet but the trajectory is there. I certainly intend to send that email soon.

    BUT, I’d feel much better not being an edge case but part of a growing pool of customers who provide a recurring revenue source for Yahoo. Amazon has stuff in this vain like Amazon SimpleDB, which if I were inclined to use I could do without worrying that it might go away (or terms changed) tomorrow.

  • http://markandey.com Markandey Singh

    AWESOME !!!!! U really solved it !!

  • http://blog.getify.com Kyle Simpson

    Well written article, and cool stuff.

    But I must protest, this is NOT pure JS solution to secure OAuth. YQL has a huge server-side backend, and you’ve just shifted the burden from your backend to their backend. It’s the shell-game approach.

    The other thing is (ironically), if you want to use YQL with higher rate limits (and not be in some special exception) you have to use…. you guessed it, OAuth to sign your YQL queries. Chicken-and-the-egg…

    That having been pointed out, YQL is awesome and I’m glad you wrote this great article about how to use it in this way.

  • http://twitter.com/drewlesueur Drew LeSueur

    OAuth 2.0 has a “User-Agent” profile that addresses OAuth in pure client-site JavaScript.

  • http://unclehulka.com/ryan/ Ryan Kennedy

    Derek…if you fancy another solution to this problem, dig around internally for “Fulcrum”. I built it while I was still in Y!OS specifically to solve this problem. It started life as Yahoo! Connect (not the current Yahoo! Connect, more of a direct Facebook Connect competitor) and ended up morphing into OAuth Connect…a way to securely talk to OAuth 1.0/1.0a protected resources from the client. It protects against the final holes in your implementation…namely the ability to authorize users with your consumer key and secret.

    The code is in source control…I forget if it went into CVS or SVN. I can’t remember if I ever wrote a twiki document for it or not, either.

  • http://soundcloud.com/matas Matas Petrikas

    @Drew: the OAuth 2.0 ‘User-Agent’ flow works super. The only thing you have to keep in mind that this case the authorization expires in one hour (less or more depending on the implementation) It’s nice for the some immediate interaction, but not something you’d like to build e.g. a Twitter client, where a persistant authorization is needed.

    we are using the flow in our JS API wrapper, you can check out the code on GitHub:
    http://github.com/soundcloud/SoundCloud-API-jQuery-plugin

  • http://www.feedic.com/ Felix

    I already implemented a similar script a couple of weeks ago, I just got no project for it (for now). Some thoughts I had when reading your implementation:

    -YQL rate limits depend on IP adresses, so when a user accesses your script, it wouldn’t lower any limits. (I don’t expect a user to login a billion times, but even just he would be blocked.) You can even push data to your or other services inside the requests as often as you want.
    -Twitter actually got a callback_url parameter, and it works pretty well. It’s just that the tables you use don’t (as far as I know) support it. I used the OAuth tables in the github repo, they offer these params. (I tested my script inside my public Dropbox and redirected the user to a child domain, so that worked out.)
    -If you don’t want your user to get any keys, just store his keys in a seperate DB as Cloudcache

    I hope that helped in any way :D

  • Greg Thorson

    I’ve tried this a bit… and it _sometimes_ works. But more often than not, I see this message:

    “Table _____ requires https but requested through http.”

    What am I overlooking here?

  • Derek

    Greg, the YQL URL you are requesting is likely “http” instead of “https”. Double-check that.

  • Greg Thorson

    It’s definitely not a matter of calling query.yahooapis.com without an https… I have refreshed the page, and sometimes gotten the expected results, sometimes the error message. I am certain of this, because I have simply refreshed without any change to the URL.

    I _always_ get the error message when I try to run this in the context of a web page, though.

  • rahul

    thanks for this article..
    one more thing i want to know i have downloaded your twitter client code from git. i dont know how to create yql tables..and the tables that are already present where i have to mention consumer key , consumer secret and oauth token key and secret for my twitter application
    waiting for your response

  • Francois Lafortune

    I have not even finished reading the article but I must say: congrats! That is one of the niftiest workarounds I’ve seen when it comes to OAuth and Javascript. YQL… why didnt we all think of that? Well, you did and I thank you for it. Off to hack this up!

    P.S.: I regret not having found this article when it was fresh!

  • errumm

    Nice. This should come in really handy on multiple future projects.

  • Douglas Hiles

    Thanks for the YQL/oauth introduction! However, something bothered me about your solution to using oauth in javascript. This is the problem:

    Once the link to your ‘store://derekgathright.com/oauthdemo’ is public, there is nothing stopping a rogue programmer from using your store any program. Imagine if the program pretends to be written by you. Trusting in your excellent reputation, the unsuspecting person running ‘your’ program sees the twitter authorization screen matching your name and naturally gives the program the right to access his or her twitter account. I discovered this nagging problem when I was exploring your ‘Tweetanium’ program.

    What disturbed me was that I was able to call your store from my test program, display your Tweetanium twitter authorization screen and then have twitter return back to any url on my calling site by setting the oauth_callback in twitter.oauth.requesttoken.

    You have hidden your consumer key and secret but have by sharing your store, you have exposed the ability to display your twitter authorization screen and return back to the calling program. This is like giving your car keys to an untrusted valet who won’t give up the car keys, but who can be told by anyone to drive your car anywhere.

    What I did to solve the problem was to create a YQL open data table, for example, myauth.xml, which returned the results of this query:
    var q = y.query(‘select * from twitter.oauth.requesttoken where oauth_callback=”http://www.mywebsite.com/index.html”‘);

    I appended myauth.xml to a YQL env file similar to your oauthdemo.txt and made a store out of it.

    By hiding the oauth_callback in a yql open data table, nobody will hijack my store because the twitter authorization screen can only return to the url that is defined in the store, not in the calling program. In this scenario, my ‘valet’ can be told to drive the car, but he is forced to return the ‘car’ back to a defined location. This added security makes YQL a completely viable solution for javascript oauth.

  • Douglas Hiles

    The page that twitter calls back to can be ‘locked in’ by setting oauth_callback as an environment variable along with oauth_consumer_key and oauth_consumer_secret in the store. This is a much better solution than passing in oauth_callback from another xml open data table.

  • Talha

    Could you please give vimeo a try as well. I came across problem that my boss wants me to go through a vimeo channel which has 150+ videos and pull the one that have specific keywords in them. Problem is that basic api allows to pull only 60 videos and 20 videos per page. I ended up in making three calls to load all in one big array of 60 videos.
    In order to query all 150+ i will need to use advance api that uses OAuth. Luckily YQL has oauth tables defined that you can use in playground but I cant get it to work with YUI . I can pull result in YQL playground. Please see if you can spare few minutes on this.

  • Beppo

    Is there anything stored on your Server ? Because in twitter.js there is a line params.env = “store://tweetanium.net/tweetanium06″; ?

  • http://www.budporn.org budporn

    Hey! I know this is kinda off topic however , I’d figured I’d ask.

    Would you be interested in exchanging links or maybe guest writing a blog article or vice-versa?
    My blog discusses a lot of the same topics as yours and I think we could
    greatly benefit from each other. If you happen to be interested feel free
    to send me an e-mail. I look forward to hearing from you!
    Awesome blog by the way!