YUI Benchmark is a new toolkit for JavaScript performance testing. Despite “YUI” being in the name, it can be used for any type of JavaScript application, including vanilla JS, YUI, Dojo, jQuery, Node.js, and anything else you can think of. The project was something I was working on at Yahoo to help with YUI’s CI performance testing needs, and since its quiet open-sourcing a few weeks ago, I’ve had some time to clean it up, fix bugs, and introduce some new functionality.

Before we get into what it is, let’s first look at why it is useful.

Prelude

I believe the current state of JavaScript performance testing is a little inadequate. While there are certainly some developers who use tools like JSLitmus, [Benchmark.js])http://benchmarkjs.com/), or manual profiling to gather performance metrics, most don’t actively test performance of their code. Instead, most developers write performant code to the best of their ability and call it a day. I don’t think this is due to lack of interest in the idea. Rather, I just think it is due to a lack of flexible and easy-to-use tools. For instance, JSPerf.com is awesome at what it does, but due to it being web-only, its usefulness is a bit limited. What about developers who want the ease of JSPerf, but want something command-line driven? Perhaps in a CI environment? Benchmark.js is handy, but it requires lots of boiler-plate for browser-testing as well as a custom test runner for integration into CI. Shouldn’t this be easier?

We went through this before. If you look back 5+ years, few JavaScript developers were unit testing their JavaScript code. Even some of the popular JS libraries weren’t shipping fully tested code. We first started with JsUnit, then came YUI Test, QUnit, Jasmine, Mocha, Vows, and a variety of others. As the number of available tools grew, so did our acceptance of the idea that our code needed to be rock-solid stable. Now, often times the only way to deploy code is through continuous integration systems (such as Jenkins, Travis CI) that run your code through a gauntlet of unit tests. Heck, a few weeks ago I was talking to a fellow developer and he said “Know when code can be considered ‘legacy’? When it isn’t unit tested.”

Our priorities, expectations, and workflows have changed since 2008. So with JavaScript unit testing now a mostly solved problem, I think we can now turn our attention to the problem of performance testing. If this field is of interest to you, here’s a tool that can probably help you out.

What is YUI Benchmark?

Think of YUI Benchmark as the glue that combines Node.js, Benchmark.js, Phantom.js, Yeti, and was designed around the workflow of YUI developers and CI testing environments. To dive in a bit more, it is a Node.js application that utilizes Benchmark.js to measure performance of a given function in various environments, including Node.js and web browsers.

Here’s a quick demo executing this simple test.

// vanilla.js - A test suite to compare array creationvar suite = new PerfSuite({    name: 'Simple test',    tests: [        {            name: 'new Array()',            fn: function () {                var arr = new Array();            }        },        {            name: '[]',            fn: function () {                var arr = [];            }        }    ]});

The idea is that you provide simple performance tests (examples), which contain only the code you want to test performance for, and you get to forget all the boiler-plate code. YUI Benchmark will read your test file, then compile it to either a .html file (for browser testing) or a .js file (for Node.js testing), execute it in the environment(s) of your choice, then dump the human readable results to the command-line or a raw JSON file. Flexible, and simple.

Bonus features include the ability to test in multiple browsers at the same time (thanks Yeti!), as well as multi-version testing for development on the YUI project. In addition to built-in support for YUI, it also offers the ability to test jQuery and Dojo as well. Here’s an example of that.

So that’s the quick intro. I could go into more details here, but it would be duplicating much of the details and demos that can be currently found in the README. So go check that out for more information on the project and details on how to get started. (Hint: npm install -g yui-benchmark)

Roadmap

YUI Benchmark is still a young project and has only been used in YUI’s CI system as well as a few select developers. While the feedback so far has been great, there’s probably still some rough edges to iron out. So aside from improving stability and usability, I’d love to eventually add some features to the project so it can realize it’s full potential. Here’s a few ideas…

• Generalize multi-version testing for all projects, not just YUI. Multi-version support is valuable in performance testing because duplicating testing conditions for comparative analysis when results are gathered sometimes months apart, is not ideal. It’s often times far easier to execute tests against select versions of your code in the same test run and compare your results.

• Machine state intelligence. I believe this is the most exciting aspect of an application like this. With the possibility of YUI Benchmark code executing in the client and server while your performance tests are executing, insight can be gathered about the state of your machine and factored into test results. Did your CPU max during testing? Are you swapping memory? Are your objects leaking? Any of these problems could be detected to inform the user that the results are less than ideal.

• Phantom.js cluster support. Currently tests are run serially, one after another, after another, after another. If your project is like YUI, you can have dozens of tests suites containing hundreds of test cases, and at approx 6 seconds per test, this pretty quickly fails to scale. One solution could be to farm out the test files to phantom.js clusters on low-powered VMs so the tests could be run synchronously. Important to note that while performance testing on low-powered VMs seems counter-intuitive, it’s important to realize that this type of testing is comparative, not discrete. The key metric you are looking for is not how many operations per second your code ran, instead, it is how much faster it ran than your other result. Also, I’d target “low-powered VMs” because that is commonly what you get in CI environments.

• Introduce threshold support to trigger regression/failure in CI. Currently multi-version results require manual analysis to determine if your code is regressing. It isn’t possible to always trigger a failure if your test is running slower due to the fact that swings of +/- 5% are not uncommon. My suspicion is this will require the introduction of a threshold to accommodate highly variable tests. In the future, more intelligence can be used to compare against historical deltas and automate the process of determining the regression threshold value.
  

• More intelligent analysis of gathered statistics. Any performance testing toolkit should provide accurate and detailed statistics so the developer can make the most informed decision for appropriate fixes. Currently, YUI Benchmark simply relays the statistics Benchmark.js calculates, but greater analysis can be done with multiple –iterations to expunge outliers and re-execute the suite.

• And more!

Thanks for reading, and I look forward to your feedback and contributions!

Some highlights

• Attending, speaking, and mentoring hackers at dozens of conferences and events. The YDN crew is world-class, and despite all the free food and nice benefits, working with them is one of the best perks of being an engineer at Yahoo. At least I think so. Special thanks to Anil, Havi, and Stacy for all their support at Hackdays, and helping organize SoCal meetups.

• I feel fortunate to have been around for the beginning stages of what everyone hopes will be Yahoo’s “big turnaround”. We probably won’t know for another 2 or 3 years if it will ultimately succeed, but the changes made in the first 18 months have been exciting to witness. While I don’t envision myself taking over a struggling Fortune 500 company anytime soon, learning some lessons for what it takes to turn that ship around been invaluable. Communicate, set goals, accomplish, repeat. Simple, right? We’ll find out.

• I also feel fortunate to have had the unique opportunity to work on a project that reaches, what… 1/7th of the Earth’s population in a given month? Kinda mindblowing when you think about it. And to top it off, you get to do it while working with a really talented and passionate group of people.

• It was especially great being able to see the internals of how to effectively manage a large open-source project. Since YUI is on a never-ending quest to open up as much as possible, looking back on where it was in 2009 compared to now is pretty remarkable. Some of that praise should of course be given to Github, for producing such an amazing product for open-source organizations and letting developers focus on developing. The entire culture around open-source has evolved greatly in the last few years, and I think YUI has responded remarkably well.

• Hacks! Especially ones with YQL. Between things like the YQL Executor, Secure OAuth in JavaScript, hacking YUI inside of YQL, SoCal Hackdays, Hacker Movie Nights, late-night scripting competitions with Dan Beam, and my projects from internal Hackdays. I couldn’t tell you if Paul Graham was right about Yahoo’s dead hacker culture in 1998, but it certainly doesn’t apply in 2013. As one of my hack-discuss memos once said, “Let a thousand hacks bloom!”

• The Purple Kitten Tracker. One of these days, purple kittens will make an appearance in URLs, and it will be glorious.

• Y! Sports Karaoke.

• But most of all, the biggest highlight was simply setting a career goal, accomplishing it, then again.

To conclude

It was a pleasure to work with so many smart and passionate people, which was the entire reason why I joined Yahoo. If you are the smartest person in the room and not the CEO, go find a smarter room. If you are working for a company that doesn’t believe in you, it’s time for a change. Ultimately, your career is about you, and the best way to succeed is to have the support of a company that can help you reach your goals. Yahoo gave me an immense amount of support over the years, and I’ll always be grateful. So thanks to my managers Michael, Dan, Thomas, and Jenny.

What’s Next?

Believe it or not, in my four years at Yahoo I only took one vacation. So, I’m going to spend a few months off, Hawaii next week, travel more after that, hack on new projects, contribute to open-source, and eventually find my next adventure! If you are interested in speaking to me about opportunities, you can connect with me on LinkedIn, and find any other relevant links at derek.io.

Well Yahoo… So long, and thanks for all the fish!

If there’s one thing we’ve learned in 20+ years of building Web sites, it’s that performance matters. Browsers need to fetch, parse, and render pages quicker. JavaScript needs to execute faster, HTTP servers need to generate and serve content more efficiently, and caching layers need to be more pervasive. All in effort to make the browsing experience more enjoyable for the user so they browse, click, engage, and buy more. The effect performance has on the bottom-line has been very well documented over the years, and the interest in “fast” is doing anything but slowing down.

So every year, the industry’s experts on speed gather at the Santa Clara Convention Center for Velocity, a conference all about Web performance and operations. Myself, along with other members of the YUI and Mojito teams were fortunate to be on hand again this year to discuss our experiences and learn from others. It was an insightful and inspirational 3 days, and in this post I’ll recap and share some of what we learned with you.

Day 0

(Steve Souders | “The Illusion of Speed”)

The first day of Velocity-related events consisted of workshops and short Ignite talks, some of which you can find on O’Reilly’s Ignite Velocity YouTube playlist.

One particular talk to highlight is Steve Souders‘ “The Illusion of Speed”, which was especially eye-opening on the importance of speed, or rather the importance of perceived speed. “As much as I love data and metrics, that’s the tail wagging the dog. The real thing we’re after is to create an experience that people love and they feel is fast. We might be front-end engineers, we might be devs, we might be ops, but what we really are, is perception brokers.” In other words, the winner is the experience that makes users the happiest, not necessarily the fastest. Often those are correlated, but not always. Checkout out the video of his talk to learn why.

Day 1

Johan Bergström - Lund University

(Johan Bergström | photo credit)

Velocity Conference 2013 kicked off with Johan Bergström‘s presentation about risk management. Bergström is an Associate Professor at Lund University, and while he isn’t an expert on anything Web-related, he is an expert in the field of safety science. He was brought to Velocity to enhance and enlighten our understanding of risk factors in linear and relational systems and provide us with tips to minimize incidents that can lead to errors and downtime in our Web applications.

The main takeaway I got from his 25-minute keynote (which is available on YouTube) was that safety is just an illusion and no systems are actually safe, they are just made less unreliable. Since risk is never truly gone, you should never ignore its presence and continually discuss it amongst your team.

I believe that final point is an important one. Often times we don’t acknowledge risk until something actually fails, and much more can be done to minimize the damages if more proactive steps are taken to account for inevitable failures. Proactive vs reactive.

Kyle Rush - Obama for America

(Kyle Rush | photo credit)

The opening day keynotes continued with other presentations by industry experts, including Kyle Rush‘s excellent talk “Meet the Obama Campaign’s $250 Million Fundraising Platform”. Rush’s keynote was based off a blog post he wrote last year. It recapped the donation platform his team built during 2011 and 2012 when he served as Director of Front-end Development at Obama for America.

When Rush joined the campaign he inherited a slow, error-prone Web application. He rebuilt it using a variety of techniques and tools focused on front-end performance, such as serving static sites (built with Jekyll), optimize.ly, WebPageTest, and utilizing JavaScript to off-load as much as possible to the browser. The result was a 60% faster TTP metric (time to paint), a 49% increase in donation conversion rate, and a platform that was processing up to $3 million dollars an hour.

You can find a video of Rush’s talk here. It’s a worthwhile viewing that can give you some insight into what a high-scale, high-performance front-end web application can look like. Also, the Hacker News discussion response to his blog post offers some insightful follow-up conversation.

Arvind Jain - Google

Day 1 keynotes concluded with Google’s Arvind Jain, whose talk (video) was an analysis to answer the question of “Is the Web Getting Faster?”. Using data gathered from Akamai, industry studies, browser benchmarks, and the HTTP Archive, Jain concluded that no, the Web is not getting faster. While the core infrastructure (networks, browsers, protocols, etc.) are getting faster, the Web is primarily getting richer to take advantage of the extra performance capabilities, which results in a nearly equal slowdown.

Andrew Betts - Financial Times Labs

After a few lightning demos consisting of new products from Akamai, AT&T, and Appurify, the first day’s sessions kicked off. The first one I attended was a talk by Andrew Betts, a software engineer at Financial Times Labs who spoke on the topic of making Web applications as performant as native apps. In his talk “Escaping the Uncanny Valley”, Betts focused on the challenges his team faced when building the Financial Times HTML5 Web app and how to properly avoid the “uncanny valley“ (the point where a Web app tries to be too similar to a native app, but falls short and appears as a clunky, cheap imitation).

(Andrew Betts | photo credit)

Some of Betts’ key points:

• Ensure smooth scrolling with sub-16ms frame rates
• Ensure any UI pauses are sub-100ms
• Intelligently use CSS selectors
• Don’t use flexbox!
• Disable hover effects
• Intelligent use of local storage technologies
• Cache, cache, cache
• Text re-encoding (see: FT Labs blog post)
• XHR batching
• Fast Click
• Always provide UI response feedback

I recommend a review of the slides for anyone working on mobile Web applications. In them, you’ll find quite a few priceless nuggets of information in there to help save time investigating performance issues, in addition to tips on preventing any bottlenecks.

(An overview of storage options)

Stephen Woods - Yahoo

(Stephen Woods | photo credit)

Following Betts’ talk was another mobile-centric presentation from Yahoo’s Stephen Woods. In Woods’ talk (slides), he echoed some of the same lessons learned while building new Flickr products that Betts did with the Financial Times app, including ensuring that you achieve the magical <16ms frame rate (which equates to 60FPS), and tips for intelligent caching (e.g., “cache values, not DOM nodes”).

Main takeaways included:

• Cache data where possible
• Update the UI on user actions, use transforms where possible
• Multitask when you can, with threads and yielding

Colt McAnlis - Google

(Colt McAnlis | photo credit)

A little later in the day was an insightful talk from Google’s Colt McAnlis in which he described the inner-workings of Chrome’s rendering engine, harnessing the power of 3D acceleration, and the tools available in Chrome that can help you create more efficient applications.

The takeaway from this talk was to understand how the rendering process occurs and to design pages with the GPU in mind. As Betts also alluded to in his talk, if you misuse advanced techniques, you can end up in worse shape than where you started. Understand, analyze, then enhance.

Here’s a pdf of his slides.

Day 2

Dylan Richard - Obama for America

(Dylan Richards | photo credit)

Day 2 of Velocity kicked off with an insightful keynote from Dylan Richard, who recently spent 18 months as Director of Engineering at Obama for America. In his devops-centric talk, Richard shared some of his experiences and insights during the time when he was tasked with rebuilding the organization’s technology infrastructure, as well as sharing tips for building resilient systems and teams.

Richard reminded us that uptime for critical systems is the primary objective, because “No feature is cool enough to not have a working app.” He is also a strong believer that “You can’t learn unless you are simulating real things”, which of course lead into some discussion of destructive testing.

While this talk certainly was more on the operations side of the Velocity spectrum, there are still valuable insights for software engineers of any kind and is worth a viewing. You can find a video of his talk here.

Ilya Grigorik - Google

(Ilya Grigorik | photo credit)

During the Lightning Demos portion of Day 2, Google’s Ilya Grigorik introduced some new tools to help you find answers to Web performance-related questions. Grigorik began his talk (video) with Indira Gandhi’s famous quote “The power to question is the basis of all human progress” to emphasize the role curiosity plays in our success as engineers. Grigorik then proceeded to list off various questions many Web developers have such as “Are websites getting faster?”, “What CSS frameworks are the most popular?”, “Is Flash adoption on the decline?”, and more.

It turns out, these previously difficult questions to answer are now relatively easy thanks to the availability of the HTTP Archive data in Google’s Big Query. If you are familiar with SQL, you can now throw together some simple queries and analyze gigabytes of HTTP data scooped up from a crawl of a million websites. Grigorik goes into more details in his blog post.

So naturally, I became curious on the version breakdown of YUI usage and realized the HTTP Archive can help answer that question for me. After some data crunching I was able to generate this chart:

Ariya Hidayat - Sencha

(Ariya Hidayat | photo credit)

(“The Unix Philosophy”)

One of the more educational talks of Velocity 2013 was Ariya Hidayat‘s “Emerging JavaScript Tools to Track JavaScript Quality and Performance”. In this talk (slides), Hidayat began with describing composable tools, “Write something once that does its job very well”, and then presents his favorite JavaScript tools while referencing some topics and tips covered from many of his blog posts.

Some of the JavaScript-related tools mentioned:

• Phantom.js - A headless WebKit scriptable with a JavaScript API
• Esprima - A high performance, standard-compliant ECMAScript parser
• MetaJS - A CPS Javascript metacircular interpreter that visualizes script execution
• JSComplexity - A complexity analysis for JavaScript projects
• Plato - A code complexity visualization tool
• Defs.js - An ES6 const and let transpiler to ES3.
• Istanbul - A code coverage tool for JavaScript

Hidayat closed with the following bits of advice:

• “Build composable tools”
• “Automate any tedious parts of code review”
• “Incorporate code quality metrics into dashboards”

Nicholas Zakas - Box

(Nicholas Zakas | photo credit)

Velocity’s Web Performance track concluded with a talk (slides) from Nicholas Zakas on the over-reliance of JavaScript. In “Enough with the JavaScript already”, his point was that as JavaScript developers, we have the tendency to try to do too many things in JavaScript. As an example of this, Zakas cited the plethora of client-MVW frameworks (Model-View-Whatever) and client-side rendering as JavaScript-gone-too-far. This is something Twitter and airbnb have realized, which should serve as a reminder to take a step back and rethink things before you try to outsmart a web browser.

As an example of proper and intelligent use of JavaScript, Zakas pointed to Alex Sexton’s wonderful Deploying JavaScript Applications blog post which mentions the idea of listening for mouse movements to trigger loading JS. Consider the fact that you have two types of scripts you typically run on a page, 1) Render/Functionality-critical scripts, and 2) Supplementary functionality scripts. If a script isn’t required for primary functionality of your page, you should defer loading it.

Speaking of when and where to execute your JavaScript, here’s his categorization of the four suggested locations to place your Javascript:

Zakas’ talk concluded with some observations about building and using libraries, arguing that it’s an economy of scale rule, meaning the cost per component decreases the more components you use. So for example, if you are using YUI for only node, that’s really expensive, and maybe not the best investment. But, if you begin using other YUI components as well, the more you use, the cheaper your investment becomes.

The takeaway here is that bytes are costly in bandwidth, performance, and maintenance, so scrutinize your decisions when selecting libraries to use. Don’t just blindly invent or throw a new library at the problem.

You can find some insightful discussion on these slides over on Hacker News.

Overall Takeaways

Here are some common themes observed at Velocity 2013:

• Tracking browser metrics in CI is becoming popular, both due to availability of Navigation Timing APIs and greater awareness of paint timings. As Paul Hammond said in his Ignite Velocity talk, “If it moves, track it.”
• Your software will make mistakes, but it’s difficult to predict what and where. So, focus on making resilient systems that help you detect problems faster to mitigate the damage.
• The point of improving performance is to improve the experience of the end user. They actually don’t care that things are metrically faster, they just like the feeling they get from their perception that things are faster.
• Alex Sexton’s excellent “Deploying JavaScript Applications“ blog post, and his follow-up “Front-End Ops“ seemed to be commonly cited among Velocity talks.
• And finally, when you think you are done with performance, you probably aren’t.

As a two-time attendee of Velocity, I’m comfortable saying that it continues to be one of the most educational and inspiring events out there today. Tim O’Reilly stated in his 2011 keynote, “You might think you are just helping to run some random site, but you are engaged in an exploration that is central to the future of humanity”, and after learning about all the interesting problems being tackled by Velocitians, I completely agree. So, cheers to the Velocity organizers, presenters, and community, for building a better, faster, stronger Web!

P.S. You can find all my session notes here. Also, if you are an O’Reilly Safari subscriber, keep an eye out for videos of the 2013 talks. In the meantime, you can view previous years 2011 and 2012, as well as finding select videos from 2013 on YouTube.

However, I’ll actually try and refrain from commenting too much about their recent changes. It’s already been discussed to death, everyone hates it, including me, and frankly, I’m not surprised. It’s just yet another move to close off and control their once free, open, and awesome API. But whatever, they can do what they like.

The reason it did prompt me to begin thinking about the topic though, is the larger significance of the announcement. Twitter’s move is just one more instance of yet another company turning its back on the developer community. But more importantly though, it’s a move that made me realize something… the mashup era is over.

Let’s rewind back to 2004. Google was still a private company, Yahoo was best known as a search engine, Facebook was still running on a computer in a Harvard dorm room, and most important to this story, Microsoft thought the Web had reached its pinnacle. They won the “browser wars”, Internet Explorer had a 92% market share and Netscape was dead, so (as rumor has it) they disbanded the IE team. Subsequently, innovation on the Web stalled. It was a dark time as very little work was being done (successfully) to evolve the Web’s core technologies (browsers, HTML, CSS, JavaScript). The Web, as an application platform, had failed.

Well, that’s what everyone thought. At least until Google released two very important products that proved to everyone that there was still a lot that can be done with the Web. Those being GMail (2004), which popularized Ajax, and Maps (2005), which unintentionally led to the invention of the mashup and contributed to the rise of “Web 2.0”.

HousingMaps.com (2005) typically gets credit as the first Web mashup, which blends Craigslist with Google Maps, and this was before Maps even had an official API. ChicagoCrime.org followed shortly after, and the rest was history. Mashups were popping up by the thousands, and JavaScript was becoming “cool” for the first time, because it could make your once bland and stale Web page come alive!

Over the next 3 or 4 years, Web companies left and right were distributing their data far and wide through a variety of data formats. “Want XML? Here ya go! RSS for everything!! JSON? Sure. We’ll even set up JSONP for you too. Authentication? Nope, not necessary, we trust you to do the right thing with this data. Happy coding and let us know if you build anything cool!” It was a very innovative era, and the extent of what you could build were only limited by your knowledge and the amount of time you had to hack something together.

But as you’ve heard 1,000 times before, all good times eventually come to an end. Nowadays, it’s tough to find an RSS or JSON feed on most sites that aren’t blogs. Sure many of them have that same data available via formal APIs, but that requires an account for authentication, documentation to be read, knowledge of technologies required, and (very likely) limitations placed on the amount of data you can consume. It’s a lot of work. Also, an API is no longer thought of as a necessity to building a successful product.

Why did this happen? It’s tough to say really, but here’s a few theories as I think aloud…

• With the rise of JSON and the death of RSS, there was no standard/popular feed format for every Web service to implement, which meant it was a bit more difficult to open up a public API since you had to actually think about your data structure. On the client-side, while I would absolutely rather work with JSON over XML/RSS, the tools and data are less interchangeable because they are all using custom structures, which leads to more work to build a mashup.

• The bar was raised. No longer are we amused and entertained by Google Maps mashups or a Web-based Twitter client. Our expectations are higher, requiring more unique ideas and more resources towards development. With a higher barrier to entry and higher expectations,

• Facebook and Apple proved you could make $billions off partially-closed environments, which made previously open companies start looking at closed environments for their data. For example, Google, a previously strong advocate of full and open APIs, still has yet to release a full API for Google+ and lets us know we shouldn’t expect one anytime soon. Yahoo, who previously would try to out-Open Google, was never able to properly execute on the “Open” concept (see: YAP, and many of the APIs from the 2006-2010 era are either gone or no longer useful.

• When companies are young, they are usually very open to the idea of “Open”, because that’s a great way to gain developers and new ideas while you work on creating your business. Eventually though, all those investors that gave you millions of dollars will want to be paid back (preferably 100x over), so you need to eventually settle on a business model.

RIP mashups.

I spent a few days last week in New York City at Yahoo’s Open Hack All-stars event. At this hack day, I was mentoring a team of 3 students from the University of Texas who set out to create a hack that allows you to control a media experience on your TV by using your iPad.

For this hack, they needed to talk to search APIs from 4 different services (Youtube, Justin.tv, Flickr, Netflix), parse the results, and display a thumbnail for each item with a link to play/view it. Traditionally, this would be a rather bulky iPad application where you’d have to include all the code and logic to communicate with the various JSON, XML, & ATOM service APIs, parse the results, combine them, and finally render the content. Likely, the HTTP calls would be synchronous, which would certainly present some issues as you get to 5+ APIs and you have to wait for one response to return before making the next.

Alternatively, you could create an API service that will do all of this for you. When that option was presented, I immediately realized YQL would be perfect for this task. Why?

• It can communicate with any HTTP-based APIs, asyncronously, so your response time is always as fast as the slowest API you have to talk to


• Use custom JavaScript to parse the results and form the return set


• Reduces the number of requests your client makes to a single HTTP request

So, I strapped on the headphones and began coding. A few hours later, here’s the result. It’s YQL datatable that heavily uses the feature, which allows you to run arbitrary JavaScript. Within , you get a simple library that allows you to do things like parse JSON, make HTTP calls, and create XML structures with E4X. The datatable code is pretty straight-forward really. Here’s the service to talk to, the URLs to send the search query to, and the callback to parse each result set. Now go!

The beauty of this YQL datatable is that you have now created a fully-functional high-performance API server without the need for a server of your own to run it on.

Here’s a JSFiddle of the script in action. Click the play button to see the combined search results.

You can also toy around with the query in the YQL console here.

If you are interested in learning more fun stuff you can do with YQL, here’s another post, How-to: Secure OAuth in JavaScript

So you want to learn JavaScript huh? I can’t blame you, it’s a pretty rad programming language. Well lucky for you, it’s a really easy language to pick up and learn. You can get started with the language without spending a penny on a compiler, an IDE, or any instructional material. Heck, you already have a computer capable of running JavaScript. I know that because every modern web browser has one, and that’s how you are viewing this blog post.

I’m approaching this post as an introduction to JavaScript for someone who is already a programmer (novice or advanced, doesn’t matter). An introduction to programming would be an entirely different post.

Without further adieu…

Tip #1: Start by Reading the Wikipedia Entry

Tip #2: Learn it Outside of a Browser!!!

I’m going to assume you have some experience with another programming language. Odds are likely it is PHP, Ruby, or Python. Those are all scripting languages, which means it isn’t compiled prior to runtime, instead it is read by an interpreter and executed on the fly. Well, JavaScript works the same way, there’s no compilation step you have to do prior to running a program. So, download Node.js and start writing some basic scripts. At this moment, Node.js only runs on OSX and Linux, so if you are running Windows, then ok… You can either visit jsbin.com or use the JavaScript console that comes built in to Chrome. Windows support for Node.js is coming soon though (late-summer 2011).

The reason I say learn it outside of a web browser is because you should approach JavaScript just like you would any other programming language. Only focus on its standard library to start. If you start toying with it in a web browser, you all of a sudden have access to the DOM and the BOM APIs, and then you’ll be distracted by learning HTML, CSS, which will likely be frustrating and lead you to a library like jQuery or YUI. Just don’t do it. Once you get comfortable with the syntax, scoping, and prototypal nature of the language, then proceed to use it inside of a web application.

Tip #3: Read Some Books

I highly recommend Eloquent JavaScript to start, because it’s a great book, and it’s free! After that, check out a few other of my favorites; The Good Parts, High Performance JavaScript, and Pro JavaScript Techniques. You can buy them all used off of Amazon for under $15 each.

Tip #4: Watch some videos

Tip #5: Get Involved in the Community

Tip #6: Ignore W3Schools!

There is a plethora of ancient JavaScript tip sites that haven’t been updated in 10+ years. They often contain horrible, and sometimes incorrect, examples of JavaScript code. Instead, visit the Mozilla Developer Network’s JavaScript documentation. W3Schools is so bad, it even spurred W3Fools.com.

Tip #7: Be Curious

There are more discussions about JavaScript and code posted per day than time you have to actually read it. That means there is an unlimited supply of information out there about the language, seek it out.

–

Well, those are the tips that immediately come to mind. There’s quite a few more ideas, blog posts, tools, resources, services, etc… that I’d love to recommend, but you have more than enough to get started. Now go get coding!

I’ve tweeted “#ie6mustdie” more than a few times. I’ve cheered as Douglas Crockford shows “IE6 Must Die” slide during his presentations. I even own a t-shirt that has the phrase. As happy as I’d be if the browser disappeared off the face of the earth. But unfortunately that’s not going to happen and no amount of wishing on my part can accelerate its demise.

So as IE6 celebrates it’s 10th year anniversary (happy birthday you P.O.S.), I’m going to throw this out there… “IE6 Must Die” must die. We’re at a turning point now where no amount of polyfills can provide the level of features that a “modern browser” can provide. In the past, browsers all did mostly the same thing, implementation just varied a bit. Nowadays, that’s not the case. Sure you can use a Flash fallback for something like Websockets. But IndexedDB? Hah! CSS3? Psshh… Geolocation? Yeah… no.

But the simple truth is that there is still a very large amount of users on the world that are using IE6. While it may be a non-factor for some niche sites (e.g. Github), for other sites it may represent more than half of their traffic. In China, 40% of the web traffic (“Why Microsoft Can’t Get Rid Of IE6 Yet“) is using the browser. Globally, we’re still above 10% IE6 market-share (ie6countdown.com) and it’s going to take years before that drops below 5%.

My advice? Embrace it. Stop fighting IE6 and understand that all web pages do not have to look or act exactly the same. Remember, you are a professional web developer building web pages. While IE6 has its faults, it is still a standards compliant browser that is designed to view (X)HTML web pages. There is no reason your blog cannot support IE6

Last week, the YUI team announced an update to its Graded Browser Support guidelines (“Graded Browser Support Update“). There has been some confusion in the community as to why IE6 is still on it and what that inclusion implies.

My take? I think it’s a great, thoughtful, step in the right direction. If IE6 were excluded from the EBS, that means dropping all support for the browser and just ignoring it even exists. As much as we’d love for that to be a reality, we’re still far from it happening. Ignoring IE6 even exists means you don’t even consider it when building your web applications. Even some of the more advanced web apps from Yahoo, Google, Facebook, etc… still consider IE6. “Considering” doesn’t mean fully supporting, it just means you provide some tailored experience for those users. That experience could even be a splash page that says “Sorry, you need to upgrade to another browser in order to view this page.” According to the latest GBS, that is absolutely an acceptable experience.

With the rise of HTML5, the world that the GBS exists in today is much different than it was even a year ago. In the past, all browsers had more-or-less the same capabilities, the variance came in implementation and quirks, which libraries helped you work around. There were a few exceptions (e.g. png transparency, performance) but for the most part, they all did the same thing and had similar capabilities. Those days are long gone though. Polyfills can only get you so far until you run into things that simply cannot be done in non-HTML5 browsers (FF<4, IE6-9). Because there’s such a wide-variety of browser capabilities today and it’s especially hard to predict capability support (see: now-you-see-me-now-you-don’t websockets), it’s impossible for the GBS to insist that you provide specific experiences for a set of browsers. The best we can do at this point is just tell you which browsers matter, and the metric for that is market share.

There are some parts of the world (northern Europe) where IE6 traffic is under 1%. Fortunately for them, (in my opinion) they can safely ignore IE6. However, in other parts of the world, such as China, IE6 still accounts for up to 40% of the traffic (“Why Microsoft Can’t Get Rid Of IE6 Yet“). Globally, we’re still above 10% IE6 market-share (ie6countdown.com). Because it still represents a significant market-share, it would be careless to completely drop support for the browser, especially for a JS library.

Along with YUI, jQuery and Prototype both continue to support IE6, along with most other JS libs as well. Why? We have to. If a JS library doesn’t support IE6, you can’t even include it on your pages without fear of seriously impacting the experience for your users. We shouldn’t be the ones who tell you what browsers you can and can’t support, that should be your decision. It’s your product, they are your users, you should decide what is best for them. If you choose not to support a full experience for IE6 users, more power to you. In many cases, I advocate that approach. But, you should at least be confident that if you still wish to provide support for IE6 users, even minimal support, you have confidence that your JS library will work.

While I’m absolutely in favor of pushing the web forward and reducing the usage of legacy browsers, we as web developers have to accept reality. IE6 is the baseline web browser we’re going to have to target for years to come. When you build your next web application and build in basic support for IE6, do not think of it as “Supporting IE6”, think of it as supporting “the web”. We develop atop a platform that considered backwards compatibility to be paramount, and the things we build should work decades from now.

Life has definitely taught me to learn to love the things you must do. In this case, I’m forcing myself to learn to love IE6 all over again. I no longer view it as an annoyance, and instead look at it as a fundamental stepping stone on our journey to build a better web. Years from now, I’m certain we’ll all look back at IE6 with a bit more reverence.

“Work with the smartest people you can find, do something you’re not ready to do, find an environment in which you’re very comfortable so you can find your voice, and work for someone who believes in you – because when they believe in you, they’ll invest in you.”

I couldn’t agree more. For some people, that’s Google or Facebook. For others, that is a startup. For me, that encompasses every reason why I’m at Yahoo.

Royals fans are a great bunch. I’ll always view Kansas City as a baseball town, no matter how good the Chiefs are any given season. That’s especially remarkable considering the fact that we’ve only had a single winning season since we won the World Series. FYI, that championship happened 25 years ago, if you didn’t know. Us Royals fans know. It’s always in the back of our minds as we hover around the .500 mark anytime after April. “This is the year, we’re gonna do it!” is what we think. Most fans say that about winning the division. Not us. We just want 82 wins in one year, a winning season. Just once, and then we’ll build from there. Baby steps. Progress.

Over the stretch of a 162 game season, most baseball fans will watch or listen to dozens and dozens of games. There’s a connection you feel with the guys in the booth, a one-way bond that is developed over hundreds of hours of listening to their narrative. I can literally hear the Royals announcers in my head as I write this and make their voices say anything I want. I know them that well. Their catch phrases. Their quirks. Their jokes. I feel like I’ve known them my whole life, yet never met them once. I’m not even sure I’d recognize them if seen in person, but I could recognize their voice from across a crowded room.

Tuesday night, one of our announcers lost a battle with cancer. When the news came out last week that Paul “Splitt” Splitorff was ill, the rumors swirled that he had less than a week to live. It was a shock. We all suspected he wasn’t in great health, but no one thought it was something this bad. Less than a month ago he was calling a game. Two weeks ago he was doing the Royals postgame show. People don’t just… go like that. Do they?

Unfortunately, yes.

As I watched the Royals vs Baltimore game the day of his passing, the Royals TV broadcast observed an inning of silence for a man who gave 2/3s of his life to the organization. He was one of the first players drafted by the new expansion Royals in 1968, and pitched wearing royal blue all 16 years until his retirement. After his retirement, he began an even longer 24-year career as a broadcaster with the team.

He wasn’t the greatest pitcher the team ever had, but he still holds the record for most won games. He wasn’t the greatest announcer the game has seen, but he was ours. He was mine. He was a constant. Every night you know you can turn on the ball game and listen to the same guys call a game.

Broadcasters are special. So much has changed in your life, but they’re still doing the exact same thing they were decades before. The day you graduated high school, they were calling a Royals game. The day you got married, they were calling a game. The day you had your first child, they called a game. The day your kids graduated high school, those same guys are calling yet another game from the exact same booth for the exact same team.

Inevitably though, change must occur, and the next generation is given their chance. Sad that this time it happened sooner than it should have.

We’ll miss you Splitt.

Needless to say, sales managers don’t really like that attitude.

So as a 24 year old (f)unemployed person, how do you spend your time? Trying to sell a professional sports team, of course.

Alright, that statement deserves a little explanation. If you know me, you know I’m a huge soccer fan. I’d been attending Kansas City Wizards games since Major League Soccer’s inception (1996) and was very involved in the KC team’s supporters group. Well, a few days after I was given a pink slip, on Dec 9th, 2004 the Wizards ownership group announced they had pretty much given up hope on the team and were putting it up for sale. That was pretty devastating for us fans. We knew there was little chance someone was going to come in and buy an MLS team in Kansas City. If they did, they were buying it to move to another city. There wasn’t much hope in keeping our team, but we weren’t going to lose it without a fight.

Being without a job, I had more than enough time to take the initiative and get the ball rolling. I registered our name (Heart of America Soccer Foundation), wrote forum posts, sent out emails, made phone calls, hosted a wiki for us to organize, etc… It was a busy time for a lot of us. Within a few weeks, we were having meetings, internally, with politicians, and with potential ownership groups. Our goal was to make the city and fanbase as attractive as possible to anyone interested in making a significant investment in the team. We of course weren’t the ones selling the team, but we could make a pretty convincing argument that Kansas City was the right home for whoever was buying it. It was a collective voice that needed to be heard.

While we organized & communicated mostly online, early-on we were without a website, so that needed to change. Wordpress of today would have been perfect for the task, but this was 2004, so it was pretty unknown at the time and certainly wasn’t as robust as it is now. I had built a few web sites before, dabbled in PHP, HTML, and CSS, but had never built anything too special and certainly nothing professional appearing. However, that didn’t discourage me, and I was excited for the challenge of learning how to build the site from scratch. Luckily, we had a talented designer in the group, so all I had to do was the coding. Off I went.

Because of how much fun I was having building the HASF website, I started to have the confidence that maybe I could turn my long-time programming hobby into a career. Within a few weeks, I had my first interview for a web developer position at a local startup. During that interview, I was asked all the typical questions, some technical, and some non-technical. I was doing ok. Well, maybe. My lack of professional experience was pretty evident, cause I had none. Towards the end I was asked, “Do you have a website you’ve built that you could show us?“

Luckily…

“Sure, I just built one a few weeks ago“, and we pulled up hoasoccer.org on the projector.

I was pretty proud of it, and they could tell as I enthusiastically talked about all the things I wanted to eventually do with it.

Later that night, I got a phone call, and a job offer. Accepted!

As I ventured into a new career, my involvement with HASF had to take a backseat. Oh, I was still involved, but I just didn’t have enough time to spend the majority of each day as the COO of a grassroots organization. At that point, more qualified individuals took leadership roles and did an amazing job. It was a long process, but on August 31st, 2006, the team was finally sold to a local ownership group for $20 million. Mission accomplished!

Well, mission accomplished as far as HASF was concerned. There was still a massive uphill battle for the new owners to get a stadium built for the team, ensuring its long-term home is Kansas City. Luckily for them, some of those heavily involved with HASF joined the new ownership group filling a variety of roles. It’s a very talented, passionate group of individuals. While I worked on building my career, they continued on the journey to build a world-class soccer stadium right in the heartland, in Kansas City. I was rooting for them every step of the way.

At the same time, in 2007 I was working as a developer at an advertising agency, and not having very much fun there. Jeff, a friend from HASF, emailed me one day about a meeting with a family friend of his who was working on a startup. We heard the pitch, and the challenge was exciting. I was in the process of buying my first home, so leaving a steady job and going month-to-month contract at a web startup was a pretty dumb idea. My mom disliked it, can’t blame her. But, I took it, and had an extremely satisfying 2.5 years there. It even directly led to the next chapter in my career.

Fast forward to today, a journey that started on Dec 9th, 2004 is going to conclude on June 9th, 2011 with that first game at LIVESTRONG Sporting Park, home of Sporting Kansas City (formerly Kansas City Wizards). That’s 2,374 days of hard work one group of people have done to accomplish something uniquely special. I now live in Los Angeles, but am making the trip back for the first game in the new stadium this week. For myself, it’s going to be pretty amazing walking into that stadium for the first time. There’s a personal connection. But I can’t even imagine what it will be like for those people who have been involved with every step of the process.

It boggles my mind to think what I’d be doing if I didn’t wake up one day and decide I wanted to help save a sports team. If I don’t co-found a non-profit organization that had nothing to do with technology, I likely don’t get that first programming job at a startup. If I don’t work at that startup where I learned what it takes to build the engineering side of a company, I don’t get the experience to actually lead the next startup. Well, that next one is what directly led to where I am today and the new position I’ll be starting in a few weeks.

The takeaway from this story is that you need to be driven by your passions. This stadium’s story very likely could have happened without my involvement early on. But the fun part is, it did happen with my involvment. When you are motivated by things that you love, you’ll make awesome things happen. Continually plant seeds for for the future, because eventually some of those seeds will turn into trees.

You never know, someday one of those trees might look like this…

Photo credits:
Facade: Ramsey Mohsen - Flickr
Grass: Flickr

Well, the next chapter is about to begin.

(Note: I’ve since learned it is actually “NorCal”, not “NoCal”. Me = California noob.)

What is YUI?

The reason we need JavaScript libraries is because web browsers have inconsistent implementations of the CSS & ECMAScript (JavaScript) standards. Some browsers render something one way, while others do it in another way. We need to level the field and have a standard way to make stuff happen in the browser, so it’s the job of the library to abstract out all the differences and give developers a sane way to make your browser do something.

You can read more about the concept on Wikipedia: JavaScript Library.

Why is YUI awesome?

• It’s an open-source project with a very liberal BSD license. Yay!


• It’s widely used both inside of Yahoo, and externally as well.


• It’s a very exciting time for JavaScript as it invades the server (thanks to projects like Node.js) and YUI runs amazingly well in a server environment. Better than any other full-featured JS library.


• YUI’s module system is one of the best out there. It allows you to create lean, fast, and portable applications.


• YUI is way more than just a DOM manipulation library, it is a platform for JavaScript applications. For that reason, I think YUI is in a great position to really make an impact as JavaScript is used in new & innovative ways.


• It’s one of the most important projects Yahoo is working on and gets immense support from within the company. It even has its own conference.


• The YUI team and the YUI community are equals. The YUI Gallery allows anyone to contribute to the collection of modules YUI can use, and we’ll even host the code for you.


• YUI has some of the brightest, and most passionate people in the industry working on the project. It’s an honor to be considered among them.

I have a ways to go before I feel I’m at the level a YUI’er should be, but I can’t wait to get started.

“…the discipline, art and profession of acquiring and applying scientific, mathematical, economic, social, and practical knowledge to design and build structures, machines, devices, systems, materials and processes that safely realize solutions to the needs of society.”

This morning I was watching Carol Bartz’s interview at the Web 2.0 Summit, and John Battelle asked her the question about acquiring engineering talent in the software/web world. When he asked that question, I stopped what I was doing and focused on the conversation because it’s important to me that our CEO understands what motivates us. If she doesn’t “get” it, then the company has a serious problem. Here was her answer:

“[Engineers] want a really interesting job to do, they want to have the tools to do it, they want to work on large data sets and large problems, and [Yahoo] has that.“

Exactly what I wanted to here. And it’s not just lip-service, there is really awesome stuff we’re working on at the moment, things that haven’t been done elsewhere, so it is clear she does get it.

Then, a theme began to emerge for the day. A friend that works in developer relations at HP/Palm tweeted…

And that’s what makes Lisa great at what she does, she understands developers very well. She knows what makes us tick.

I work with an engineer (@danbeam) is so compulsive with solving problems that sometimes just to mess with him I’ll say “I bet you can’t figure this out.” I know that after 30 seconds of resisting, he has no choice but to drop everything and focus on solving it. Seriously, it works every time, it’s pretty funny. We once spent hours battling back & forth trying to generate the shortest shell script that will output every Friday the 13th. Why? Because it was Friday the 13th and we were curious. (Solution: echo -e {$((2e5)).1}13\n|date -f-|grep ^F)

I am obviously not immune to it myself. Recently, a friend sent me a link to the SeatGeek applicant hacking challenge, I had no choice but to drop what I was doing and “hack” into the system even though I’m happily employed. 15 minutes later, mission accomplished, back to work. It was fun for a few minutes, and I learned a couple things. (Tip: Try using nothing but cURL. Using a browser is cheating)

Solving these little problems is what keeps the engineer occupied. It is our form of entertainment. But if you are working with engineers, you don’t want them to simply be entertained, you want them to be inspired. You want them so full of determination that they’ll stop at nothing to figure out a solution. How do you do that? Give them something that isn’t possible. That’s like a mother telling her 13 year old son he can’t do something. Oh yeah? Watch. (sorry mom!)

For instance, last year I wanted to figure out the best method to do OAuth using nothing but client-side JavaScript. It can’t actually be impossible to do, can it? After a few weeks of hacking around, I came up with a solution and wrote a post about it, How-to: Secure OAuth in JavaScript. Yeah, it is using a proxy (YQL), but my code is all JS and I can host it anywhere. Close enough.

There was little practical reason for doing it, other than I just wondered how it could be done. There is a lot to learn by challenging yourself to do something you don’t know how to do. The biggest lesson I learned from that project was to quit trying to coerce a library into doing something for you it wasn’t intended to do. It’s usually quicker to sack up, read the spec, and write a solution yourself. Me & the OAuth 1.0 spec are like best buddies now.

Anyways, later on that same day, yet another relevant tweet rolled across my timeline…

That, is an excellent teacher. I’m a huge science nerd, and the NASA tweet reminded me of my favorite scene from Apollo 13. Yes, I’m so nerdy that the scene where the engineers save the day is my favorite. In it, the crippled Lunar Module is flying back to Earth when engineers determine that the astronauts may soon die of carbon dioxide poisoning unless they devise a solution.

Here’s the clip…

Those are the moments the engineer lives for. Solving interesting problems keep us occupied. Solving impossible problems is why we’re engineers.

Today, 2 billion people are on the internet. That means nearly a third of the world’s population has access to Wikipedia, Google, Yahoo, Twitter, Facebook, etc… The Web has finally given us the chance to globally communicate. The only thing restricting us at this point isn’t a technical barrier, but instead is a language barrier. Linguists estimate there are between 5,000-6,000 currently spoken languages on the planet. Yet, very few of us speak more than one or two of those. More than ever we need free & accessible ways to easily translate between languages, allowing us to communicate with a global audience.

Since 2008, we’ve have that. The Google Translate API. It’s an API (application programming interface) that allows you to programmatically send text to Google, and they will translate it for you, for free. It’s an amazingly useful service that I have in one of my Twitter clients to instantly translate any tweets I can’t read.

Sadly, yesterday they posted this message on the API’s page.

Important: The Google Translate API has been officially deprecated as of May 26, 2011. Due to the substantial economic burden caused by extensive abuse, the number of requests you may make per day will be limited and the API will be shut off completely on December 1, 2011. For website translations, we encourage you to use the Google Translate Element.

In other words… “This is why we don’t have nice things”

Very disappointing, especially for a company who set out to “Organize the World’s Information” a decade ago. One would assume the goal of organizing is to make it accessible. Guess not.

When Google implemented the Speech Input API in Chrome, my mind instantly blew up with ideas. I loved knowing that we were so close to having an application for your mobile device that allows you to say something, and it will instantly start translating it into text of any language.

Here’s an example of how easy it could be.

<html>    <input type="text" id="foo" speech>    <textarea id="translation"></textarea>    <script src="https://www.google.com/jsapi"></script>    <script type="text/javascript">        function translate() {            google.load("language", "1");            google.setOnLoadCallback(function() {                var native_text     = document.getElementById('foo').value;                var native_language = "en"; //english                var target_language = "es"; //spanish                google.language.translate(native_text, target_language, native_language, function(result) {                    document.getElementById("translation").innerHTML = result.translation;                });            });        }        document.getElementById("foo").addListener("change", translate);    </script></html>

You could solve a problem humans have had for thousands of years, in about 5 lines of JavaScript code.

Well, starting Dec 1st, 2011, you’ll have to use another service.

There’s no doubt in my mind we’ll get this type of capability in the near future, I’m just disappointed Google decided it didn’t want to be involved in solving these kinds of real, human problems.

Hopefully Google reconsiders this decision. To help, go post a comment on the Google Code blog.

Some developers are understanding, others are irate, and many are still pretty confused about what exactly this all means. I guess I’d put myself in the 3rd (“confused”) group. But I guess the one thing I do know, is that it is clear these “suggestions” and TOS updates are directed at UberMedia in order to prevent them from forking their portion of the Twitter user base. But, now the rest of the community has now unfairly been dragged into the mix.

As someone who has been doing development on Twitter-related projects since 2007, I figured I’d throw my 2 cents into the mix and give Twitter an idea of where they went wrong, and how they can fix it. I figured it was worth a repost here on my blog.

Link to post.

My 2 cents…

The reason for the perceived mixed messages for some of us is because many developers don’t, and never have been interested in doing Twitter development as a business. I’ve created a dozen Twitter clients & apps over the last 5 years, some of which received enough users and press coverage that I could have attempted to turn it into a business, but I didn’t. Why? Because it doesn’t interest me. I do it for the challenge and the learning experience.

So, the things we hear Twitter saying are “Don’t build clients anymore” as well as “Client apps make bad business”. Well, first, as long as the APIs are active and it’s not against the TOS, I’m still going to build, develop, and use my own clients. Second, I don’t care that it makes “bad business”, that’s isn’t a concern to me. Third, developers can determine for themselves what seems like a smart business decision or not. Fourth, frankly, Twitter Inc has never been regarded as an expert in monetization strategies.

Plus, this is info we already knew. For the most part, building a company whose main product is a Twitter client hasn’t been a good business decision for a few years (if ever, outside of a lucky few). But on the other hand, there are still markets where it could be good business. For example, where is the official Twitter client for webOS? Messages like “Don’t build clients anymore” and no official Twitter app on webOS does nothing but hurt the ecosystem for thousands of users. If I were a developer for one of the popular webOS clients, I’d be pretty pissed right now. Heck, as a webOS user I’m not thrilled. I’m sure this is applicable to other ecosystems too.

The point is, Twitter should be more vocal about what it is going to do as opposed to coy suggestions to developers (which some perceive as threats) about what they shouldn’t do. Twitter is going to heavily focus on front-end user experiences across all platforms? Great! Leave it at that. Let developers decide for themselves what are good/bad ideas. Just arm us with the knowledge of your plans, and we’ll worry about our own.

Finally, Twitter, you should be excited to compete with your developers. Much of the innovation over the years has been a product of the developer & user community. Things like mentions & hashtags came from your users. Features like saved searches, lists, trends, and ajax driven clients were inventions of developers years before they made it into Twitter.com. Essentially, “New” Twitter is just a compilation of the best features from all the 3rd party clients. Do not be hostile. Do not attack them with your TOS. Do not suspend tokens without working with the developer first. Doing these things hurts the community, which in turn hurts you. Your users are your product. Not your platform. Not your website. Not your ads. Your users.

- @derek

In short, it’s a easiest way to get Node.js modules installed on your system. It is Node’s equivalent to Ruby’s Gems, Ubuntu’s APT, PHP’s PEAR, and Perl’s CPAN. So instead of manually downloading libraries/modules, explicitly including them in your source code, and having to manually resolve dependency issues, you can just let npm handle that for you. Now, installing a new module is as easy as typing npm install <module>. The version of Node.js that webOS 2.0 ships with (at the moment) is v0.1.102, which is rather old. The build scripts for the latest npm installer does not work with older versions of Node.js, so with trial and error, the most recent version I’ve been able to install on webOS 2.0 is npm v0.1.23. Luckily it’s pretty easy to install that specific version, so here’s how you do it on your webOS device.

If you are looking for a list of packages, check out http://npm.mape.me/. Or, you can just type npm ls. Are there any plans for npm to be included in webOS? HP/Palm engineers confirmed at the webOS Developer Day event a few weeks ago that there are no plans for npm to ship with webOS. That’s fine with me. Modules should be included in the application package anyways.

Since I came across Paul Buccheit’s “Serendipity Finds You“ blog post yesterday, it has been difficult to stop thinking about it. It’s a great read, so definitely go check it out. Go ahead, I’ll wait. While you are at it, read through the Hacker News comments on the article, they are just as enlightening as the article itself.

For those of you that don’t know, Paul is an engineer at Facebook, and has had extraordinary success in his relatively young career. As one of the early employees at Google, he was in at the ground floor of one of the most successful companies of the last decade, and he was one of the reasons for their ginormous success. It almost seems unreal when you hear that one single person came up with GMail, AdSense (Google’s primary source of revenue), and their famous “Don’t be evil” motto. That is quite a trifecta. Paul eventually left Google to co-found Friendfeed, which was acquired by Facebook.

You’d probably say Paul has been lucky in his career, and he may agree with you to an extent. However, there are various interpretations of what defines “luck”. Crediting his success to mere random chance discredits Paul’s knack for putting himself in fortunate positions. Some peoples’ version of “luck” is when opportunity meets preparation, and they pounce.

I’m not even going to try to compare my career to Paul’s here, they’re on completely different levels. But what I will say is that in my mind I have achieved a level of success in my career that I am happy with, so far. When I look back at what brought me to my current position, it has been a series of events that separately, seem pretty random. In reality though, they are all woven together by a continued pursuit towards things in life that I am passionate about. One, leading to the next, leading to the next, leading…

1. I started playing soccer at the age of 5 and didn’t slow down until 10 years later, so soccer is something that I loved growing up. (Thanks for your encouragement Mom & Dad)
2. That interest in soccer turned me into a rabid Kansas City Wizards fan and a member of their supporters group. (Thanks for your passion Sam)
3. As a side-project we created a non-profit advocacy organization, Heart of America Soccer Foundation. We needed a website, so I volunteered my time to create one. The result of that was the hoasoccer.org website, and that was first website that really demonstrated my skills as a web developer. (Thanks for your help Chad)
4. When I interviewed for my first programming job out of school, I had just completed the hoasoccer.org website, and was able to use that as evidence that I knew how to webdev. That evidence was key to me being hired. (Thanks for your confidence Hillary)
5. Fast forward a few years, a friend of mine from the Wizards games became a web designer and we occasionally did freelance work together. One client we did work for eventually hired me as the lead developer for his new company because of my experience in the startup world. (Thanks for your entrepreneurial spirit Jeff)
6. It was at that startup where I convinced my boss that sending me to Zendcon ‘08 would be well worth the expense. He agreed, so off I went. (Thanks for your generosity Fred)
7. That just so happened to be the weekend of Yahoo! Open Hackday ‘08. It was at that hackday on the Y! campus where I met the recruiter who eventually got me into the interview process and hired. More details @ Return to Sunnyvale. (Thanks for the opportunity Shawn)

So, this all began with a 5 year old chasing a ball around a soccer field? Well, that is certainly one interpretation.

Often times in life, you might not know what awaits you at the end of a journey. But as long as the roads are leading to fun & challenging adventures, you are doing something right. Keep doing it, and do more of it when you can. We are naturally driven towards what we seek, even unconsciously. If you seek nothing, you’ll find nothing. Serendipity is everywhere, so maximize your chances of finding it.

Think about this… If you don’t need any server-side processing, your applications suddenly become infinitely scaleable, right? We could host on the cheapest of cheap commodity hosting. Heck, if all we’re doing is serving static HTML/CSS/JS files, just throw it on a CDN like S3 or CloudFiles and pay per GB.

Before you get too excited, realize that there is a fundamental problem with OAuth in JS. Because JavaScript in the browser is “view-source”, you are always forced to expose your consumer key pair, which compromises the security of your application. sigh

For example, when Twitter recently deprecated their Basic Auth services, that left OAuth as the only authentication method. It was supposed to be the death of JS-only Twitter apps. This was unfortunate for quite a few developers who leveraged the browsers ability to do Basic auth, to help with scaling their Twitter apps. I know, I was one of them.

So then I began to think what if you weren’t forced to expose your keys? What if your JS app could talk to any web API out there, in a secure, user-authenticated way?

Is that actually possible? Yup.

Backstory

Unknowingly at the time, my quest for a JS only OAuth app began two years ago.

When TechCrunch covered the launch of my Twitter client, the app pretty quickly died from the traffic they were sending my way. The problem is 90% of it was written in PHP and used a relational database to store waaaaaay to much data. Neither of them were designed to scale to 20k users in just a few minutes. After days of tweaking and optimizing, I finally gave up on the design. I realized I didn’t need PHP to parse the data, or a database to host the data, so I began a rewrite with the goal of removing as much server-side code as possible. I threw away the database, moved off expensive EC2 and onto commodity hosting where it worked great for the next year or so with some occasional tweaking. As hard as I tried, I never thought I’d be able to completely get rid of the backend because I needed a proxy to securely handle the OAuth requests to Twitter. “That’s ok, close enough” I thought.

One day I was reading the Yahoo Query Language documentation, and I came across a section about using YQL’s storage API to hide authentication info to be used in your queries. Ah ha! Could I actually use that for OAuth? I set to find out. I began learning the ins & outs of OAuth, which includes reading RFC 5849: The OAuth 1.0 Protocol many, many times, and staring at the OAuth Authentication Flow diagram for loooooong time. By the end of the weekend, I had successfully modified my recently rewritten Twitter client’s code-base (now YUI3 based) to remove all server-side programming.

Finally! A secure, pure JavaScript solution to OAuth.

Some Prep Work

So let’s crack the code of what is necessary to do OAuth securely in JavaScript.

• You cannot store your consumer keys inside your JS code. Not even obfuscated. But it has to be stored somewhere web-accessible so your JS code can talk to it.


• Because of the same-origin policy, that ‘somewhere’ has to be the same domain as your JS app. Unless of course you only rely on HTTP GET, in which case you can do JSONP.


• Your storage location cannot transmit your consumer key pair back to you. So that means it needs to do the OAuth request on your behalf.

Yahoo Query Language

YQL is an expressive SQL-like language that lets you query, filter, and join data across web servers. Along with YUI, it is by far my favorite product Yahoo has for developers. Both are simply amazing tools. I won’t go into detail on the specifics of what YQL is in this post, and instead point you to slides from one of my recent talks on the subject here (best viewed in Chrome). All you need to know for this post is that you can use it to access any web-accessible API. In the case of this post, we’ll talk to the Twitter API.

So now that we know it is possible, let’s see it in action.

How It Works

First let’s take a look at how you would call your Twitter friends timeline via YQL w/ OAuth. Using my @derektest user, I created a new OAuth app at dev.twitter.com and used the keys it generated for my user/app combo to generate this YQL query.

SELECT  FROM twitter.status.timeline.friends
WHERE oauth_consumer_key = ‘9DiJt6Faw0Dyr61tVOATA’
AND oauth_consumer_secret = ‘XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA’
AND oauth_token = ‘18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’
AND oauth_token_secret = ‘D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’;

That unique URL will give you a list of the people @derektest follows (which is only @derek). You can play around with the query in the YQL Console, or view the results in an XML feed.

But there’s a problem using that query, because? You guessed it, you’ve exposed your consumer key-pair. So let’s work on hiding those.

First step, turn the embedded parameters into environment variables by using the SET command.

set oauth_consumer_key=’9DiJt6Faw0Dyr61tVOATA’ on twitter;
set oauth_consumer_secret=’XBF9j0B2SZAOWg44QTu6fCwYy5JtivoNNpvJMs6cA’ on twitter;
set oauth_token=’18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’ on twitter;
set oauth_token_secret=’D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’ on twitter;
select * from twitter.status.timeline.friends;

To do this, create a YQL environment file, similar to this one, http://derekgathright.com/code/yahoo/yql/oauthdemo.txt

As you’ll see, it’s just a regular text file where I pasted my consumer key pair, along with importing the YQL community tables using the ENV command. Since we’re replacing the previously included env file (store://datatables.org/alltableswithkeys) with our own, we need to chain-load it back in because it includes the Twitter tables. If you miss that step, you’ll get a “No definition found for Table twitter.status.timeline.friends“ error.

Before we store the env file in YQL, let’s test it with this new query:

set oauth_token=’18342542-NkgUoRinvdJVILEwCUQJ3sL2CIm2ZwzS5jjj2Lg7y’ on twitter;
set oauth_token_secret=’D6ewAzsueTzQmrAJGFH0phV5zgWT88FOtcMeqW4YeI’ on twitter;
select * from twitter.status.timeline.friends;

(View: YQL Console - Results)

Now that we have our environment file created and tested, let’s tell YQL to import it. To do that, we’ll construct a YQL query similar to:

insert into yql.storage.admin (name,url)
values (“oauthdemo”,”http://derekgathright.com/code/yahoo/yql/oauthdemo.txt")

       store://derekgathright.com/oauthdemo

[hidden]</pre>

You now have 3 keys pointing to your data, and each does something different (think: unix permissions, R/W/X). For more information on what each of the 3 does, Using YQL to Read, Update, and Delete Records.

For this example we want the execute key, which is really just an alias to our stored env file. So if we change our query’s URL to ?env=store://derekgathright.com/oauthdemo and use the same YQL query as last time, you’ll see we have now hidden our consumer key pair from the public.

(View: YQL Console - Results)

Well there you have it, an example of how to hide your consumer key pair, which now allows you to use YQL as your server-side proxy as opposed to writing & maintaining your own!

A Pure JS Twitter Client is Born

When I started at Yahoo, I wanted an excuse to learn YUI3 and expand my knowledge of YQL. So porting my jQuery/PHP based Twitter client seemed like a logical choice. The result of this work is an open-source project I call Tweetanium. I’m not going to argue it is the most polished or feature-rich Twitter client. In fact, it is quite buggy, and will likely always be that way. It’s just something I toy around with occasionally to try out new things. But feel free to use it if you like. You can play around in it at tweetanium.net.

As proof that there is no server-side JS, you can even use a version of it hosted on Github Pages, which is a static file host (no PHP, Ruby, Python, etc…). Hosting off Github Pages was a neat test for it, which basically proves you can host JS-only apps on commodity hosting. If you actually need to process data externally, you can use YQL tables for any APIs on the web, even your own custom-built ones (See: YQL Open Data Tables). Any scaling bottlenecks have now been offloaded to Github and Yahoo. The best part about this solution? It’s free!

Post some comments if you have questions.

UPDATE: A few people have asked, “But can’t I execute YQL queries with your consumer keys now?“ The answer is, yes. But that isn’t as bad as you think because you only have half of the keys necessary. You are missing the unique keys assigned to a user on behalf of my application, and without those, you cannot make authenticated calls. If you get those, well… there’s a whole other security issue of you having physical access to their computer or man-in-the-middle attacks.

“Ok, but can’t I authenticate new keys posing as your app?“ To my knowledge, Twitter does not currently support the oauth_callback parameter, which allows the requester to Twitter to redirect the user to the URL of their choice. So if EvilHacker tries to authenticate InnocentUser using my consumer keys, InnocentUser will just be directed back to my app’s preset URL stored in Twitter’s database. In the future, who knows how the OAuth spec, or Twitter’s implementation of it, will change. This is mostly a proof-of-concept hack at this point.

On a side note, as I mentioned, I’m new to jQuery. In fact, my jQuery expertise goes back a full 24 hours. The website I’m coding now did use prototype and scriptaculous, but once I looked at the 1.2 version of jQuery, I knew I’d found a better solution (for me, anyways). So I spent the rest of the day recoding the site for jQuery. It was this post that convinced me I needed to change, “Why jQuery’s Philosophy is Better“

In the post, Yehuda Katz compares a relatively young jQuery to Prototype.js. Rewinding back to 2007, I was a Prototype user and it was apparently that post that sold me on jQuery being a better library. For the next 2 years I was a (mostly) happy jQuery user.

When I started at Yahoo, that was the beginning of my journey into the world of YUI. I was excited to learn something new, and pretty quickly learned to love the library because there is so much that it does right. Sure there’s a little big of a learning curve compared to jQuery, but if you know where to look for help, you can easily get answers to whatever questions you have. YUI has a full-time staff, and part of the team’s job is to help you with whatever issues/questions you have about the library.

Before I get started, I at least wanted to clarify something. I’m not going to make an argument that as a whole, one library is “better” than the other, they’re both good. The title of this post is mostly just an homage to Yehuda’s post that inspired me almost 5 years ago. On the first day of my “Introduction to Programming: QBASIC” course in high school, I vividly remember my teacher telling me “In programming, there are a hundred ways to solve every problem. Pick the one that is best for you.“ That is especially true when it comes to JavaScript libraries.

YUI and jQuery do share some similar functionality, but they approach the problem from a completely different perspective. jQuery is about simplicity and usability. YUI is about performance and scalability. Neither ignores those other qualities, they just take a backseat to the primary goal of the library.

So here we go, my list of things I like about YUI compared to jQuery. Unlike Yehuda’s post, I’m not going to focus much on specific API differences, because those can be sugared up as much as you like. Besides, just take a look at JSRosettaStone.com and you’ll see that many of the APIs are very similar, if not identical.

What I will instead focus on are fundamental differences between the libraries and why I think YUI is the best JS library out there for building web pages and modern web applications.

Modules

The biggest difference between the libraries is the organization of the code. While most other programming languages natively support packages & modules, JavaScript won’t see those widely adopted in a browser for many more years. So it was left up to the developer community to create our own module pattern.

Modules are little bits of code that serve a limited purpose and can be combined and swapped out as you see fit within your application. In one of his YUIConf talks on “Scalable JavaScript Application Architecture“, Nicholas Zakas compared this to space station modules, which I think is rather accurate. In order to get a space station in outer space, you don’t construct it on the ground and launch the entire thing all at once. You build it as many separate, smaller modules, launch them up, and construct it in space. You only use what you need, when you need it.

You should take the same approach when constructing your web application. Break it up into as many small bits as possible and only include exactly what you need. In doing so, you will save on download and execution time, and it will be much less cluttered. It’s just good design, and as long as you know what you are doing, there’s no limit to how small you should break up your code into modules. You can even go as far as 1 function per file. The topic of “ultra-modularization” once came up on the RequireJS mailing list.

Within jQuery, you are basically including 1 super-module, jQuery itself. There’s no (good) way to just pull jQuery’s Ajax component if that is all you need. Too much of it is inter-dependent. This design of course led to jQuery’s simplicity, at the risk of flexibility. A good trade given its goal of ease-of-use.

Within YUI, all the functionality you include comes from various modules. In YUI’s core set, there are 300+ modules. Things like “Calendar”, “DD” (drag and drop), “Graphics”, etc… In the YUI Gallery (community contributed modules) contains even more than that! 431 at the time of this writing, and continually growing.

To use any of these modules, just include them in your YUI().use() statement.

If you don’t include any modules, by simply executing YUI().use( function(Y){ } ), you only start with a few core ones. Specifically: DOM, Event, Node, and OOP (Object Oriented Programming).

To use anything else you need, it’s simply

YUI().use("someModule", "anotherModule", function(Y) {    // Do something});

And to make things really easy, all your dependencies will be calculated and included for you.

In jQuery, you don’t really have a native concept of modules, so the library is all or nothing. Sure the jQuery is relatively small (32k gzipped), but that doesn’t make up for the fact that you are always going to be downloading and executing more than you need. Few people use -all- of jQuery on every page.

Now, one of the problems you can run into if you break up your application into dozens of separate files is that you’ll execute dozens of HTTP requests to include all those via <script> tags. Well, here comes the concept of combo-loading.

CDN Combo-loading

All YUI modules, whether they are YUI Core or YUI Gallery (community contributed) modules are by default loaded from the Yahoo CDN, all as a single request. This is an amazing feature. Sure you get URLs that look like:

http://yui.yahooapis.com/combo?3.4.1/build/oop/oop-min.js&3.4.1/build/event-custom-base/event-custom-base-min.js&3.4.1/build/dom-core/dom-core-min.js&3.4.1/build/dom-base/dom-base-min.js&3.4.1/build/selector-native/selector-native-min.js&3.4.1/build/selector/selector-min.js&3.4.1/build/node-core/node-core-min.js&3.4.1/build/node-base/node-base-min.js&3.4.1/build/event-base/event-base-min.js&3.4.1/build/event-custom-complex/event-custom-complex-min.js&3.4.1/build/event-synthetic/event-synthetic-min.js&3.4.1/build/event-mouseenter/event-mouseenter-min.js&3.4.1/build/event-hover/event-hover-min.js&3.4.1/build/event-delegate/event-delegate-min.js&3.4.1/build/node-event-delegate/node-event-delegate-min.js&3.4.1/build/intl/intl-min.js&3.4.1/build/array-extras/array-extras-min.js&3.4.1/build/attribute-base/attribute-base-min.js&3.4.1/build/attribute-complex/attribute-complex-min.js&3.4.1/build/base-base/base-base-min.js&3.4.1/build/base-build/base-build-min.js&3.4.1/build/escape/escape-min.js

But who cares? You aren’t the one constructing it, YUI’s loader does it for you. The loader even takes into account URL size limits and properly chunks it up into multiple requests if need be.

In jQuery, you are typically either hosting the jQuery.min.js file yourself, or pulling it from a CDN somewhere. You then also typically host the plugins yourself. Also, if you want to minify and concatenate your JS & CSS for optimal performance, you have to do that yourself by using something like YUI Compressor as part of a build process.

Why not just let Yahoo do all of that for you? Use the exact same process, the same CDN, and the same files that Yahoo properties use themselves. You can build entire applications hosting nothing more than a single HTML file yourself because everything you need is hosted on Yahoo’s CDN. And as a bonus, due to it being served off Yahoo’s CDN, you are guaranteed uptime equivalent to Yahoo.

Loader

My favorite part of YUI is the Loader component. This is the utility that you use to dynamically include whatever JS and CSS you need in the most optimal method possible. Gone are the days of loading in every js script and css stylesheet that you need using HTML tags. Take a look at the new Delicious <head> tag since they moved from YUI to jQuery. It’s a pretty typical jQuery page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<head>
  <link rel="stylesheet" type="text/css" media="screen" href="/static/css/del-new.css?v=307" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/css/flick/jquery-ui-1.8.13.custom.css?v=307" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/css/jcrop/jquery.Jcrop.css" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/css/960grid.css" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/css/selectbox/selectbox.css" />
  <script type="text/javascript" src="/static/js/jquery-1.6.1.min.js"></script>
  <script type="text/javascript" src="/static/js/jquery-ui-1.8.13.custom.min.js"></script>
  <script type="text/javascript" src="/static/js/jquery.json-2.2.min.js"></script>
  <script type="text/javascript" src="/static/js/jquery-hoverintent.js"></script>
  <script type="text/javascript" src="/static/js/jquery.Jcrop.min.js"></script>
  <script type="text/javascript" src="/static/js/jquery.metadata.min.js"></script>
  <script type="text/javascript" src="/static/js/jquery.swapimage.min.js"></script>
  <script type="text/javascript" src="/static/js/jquery.selectBox.min.js"></script>
  <script type="text/javascript" src="/static/js/tiny_mce/jquery.tinymce.js"></script>
  <script type="text/javascript" src="/static/js/jcarousellite_1.0.1.min.js"></script>
  <script type="text/javascript" src="/static/js/del.js?v=307"></script>
</head>

11 scripts + and 5 stylesheets = 15 HTTP requests. And because they placed it all in the <head>, all 15 files need to be downloaded prior to the page being rendered. Yeesh.

So let’s look at a small YUI app

1
2
3
4
5
6
7
8
9
10
11
12
13

<html>
  <head>
    <!-- Include the YUI seed -->
    <script src="http://yui.yahooapis.com/3.5.1/build/yui/yui-min.js"></script>
  </head>
  <body>
    <script>
      YUI().use("json", "graphics", "calendar", "yql", function(Y){
        // Do something magical
      });
    </script>
  </body>
  <html>

The combination of those 4 modules and all the dependencies those require, actually loads in 49 modules. Again, this is all done for you! YUI’s loader is very mature and is used on some of the largest Yahoo and non-Yahoo websites on the internet.

One of the great things about the YUI Loader is that you can even use it to include non-YUI modules. Here’s a jsFiddle to demonstrate using it to include jQuery, a jQuery plugin, a CSS stylesheet, and combining it with some YUI modules. (hint: click the Play button)

Environment Agnostic

There is no “YUI for Mobile” or “YUI for Node.js”. Why? Because YUI is a JavaScript library and does not assume it is running in any specific environment. This has allowed the project to be nimble and flexible. This also allows you to reuse code you’ve already written for one environment in other places.

If server + client-side JavaScript is something that seems useful, then YUI can be a great solution. Here’s an example site that Dav created to demonstrate the power of using YUI and ExpressJS to use standard YUI widgets to pre-render the HTML markup on the server instead of the browser. The advantage here is that you can get JavaScript-like behavior, without needing JavaScript enabled, and using all the same code.

YUI is continually getting even easier to use inside Node.js. Dav Glass is the owner of the Loader component (previously mentioned), and he’s done some amazing stuff to get YUI to “just work” inside Node.js. Here’s a recent video of some work he’s done to make YUI even easier to use on the server. Yahoo uses YUI, in Node.js, on some of our largest sites, and it’s largely due to YUI being an awesome all-around JavaScript library. YUI is not just a DOM manipulation library, like jQuery.

YUI has spent, and will continue to spend considerable resources to ensure it runs just as good server-side as well as in the browser. There are some announcements coming within the next few months that will make it very apparent that Yahoo’s web-serving platform of the future is JavaScript, and YUI is a big part of that.

How does YUI work so well across all environments? We target the lowest levels possible, and build up from there.

Lower-level

I’ve never been a fan of rigid frameworks. I toyed with Rails, and Django, and CakePHP, and CodeIgnitor, and was never really happy, so I eventually ended up rolling my own PHP framework. Very simple, and possibly only appeals to me, but it’s exactly what I need to get started. Nothing more. Customizing someone else’s framework is typically more trouble than it is worth. Heck, that’s one of the reasons why I have never spent much time documenting or making Derek’s Anti-Framework Framework more friendly, because I don’t really want others to use it. That defeats the purpose of why I made it.

jQuery certainly isn’t as rigid as most frameworks. It’s much more of a library after-all. Still, my feelings are the same, freedom is more important to me than being told how to do something. For that, I appreciate that YUI is a little lower-level than jQuery. jQuery and many plugins for it ignores edge cases, and they can get away with it because their target is the 90% use-case. Well, YUI’s target is the 99%. YUI needs to accommodate all the various use-cases and customizations requirements needed for use on Yahoo websites. This is a strength of the project as it means it will be as robust and thought-out as possible.

YUI works along-side Yahoo projects to develop components for the library. For example, the new Y! Mail uses the exact same base rich text editor that is available as the Editor module. At first glance, you aren’t going to see a fancy editor with 80 buttons that does everything under the sun (and usually poorly). You are going to see a solid foundation for which to build an editor on top of. For customization, building up is usually easier and more maintainable than hacking something apart and cramming in your functionality.

In jQuery-land, if you need an editor, you are likely going to search around for “jQuery rich text editor” and come across an article like “12 JQuery Based Rich Text Editor“ (sic) and play around with each of them, figure out which one works best for you, integrate, then launch it. Then 3 months in, your boss comes to you and asks for a new feature to be added to the editor. So you crack open the plugin code and start hacking around. Maybe you hack it in there, but more often, you give up digging through the convoluted mess of code that you didn’t write, and go back to searching for a new editor that does what you need. Repeat this process every 3-6 months. I know, because I’ve been in that exact situation. Eventually you might go with something like TinyMCE because it supports the kitchen sink, but isn’t even jQuery-based (oh sure, there’s a plugin for that). So you are now duplicating all this code you are making your users download. Multiply that times an RTE, a date-picker, an uploader, a carousel, a lightbox, and you eventually end up with something like the Delicious code I pasted above.

Conclusion

As I mentioned above, I’m not claiming that YUI is definitively better than jQuery, or any other library/framework. They each solve the problem of web development differently. What I will say is that if you give YUI enough time to get over an admittedly steeper learning curve, you’ll be on a fantastic track to build highly performant, scalable, web applications. C’mon, we are programmers, developers, front-end engineers. We should strive to be more than cut, paste, then glue-together artists. Let’s build stuff!

When I saw on NCZ’s blog that he was writing a new book on JavaScript performance techniques, I instantly went to pre-order it. Having partially read through High Performance JavaScript by now, I figured I’d start writing a review of this excellent book.

Since JavaScript is such an expressive language, there are dozens of different ways to do the same thing. Some of them good, some mediocre, and a lot of them bad. It’s amazing how much awful JS info is on the web, all leftover from the dark ages of JS (‘96 - ‘05). Up until this point, we haven’t had an authoritative source on the topic of how to write JavaScript that performs well, both in and out of the browser. Sure we’re had great books about web performance (High Performance Web Sites is my favorite), but we haven’t had anything specific to JavaScript. Now we do.

Nicholas is widely known as one of the best minds in the JavaScript world today. He joined Yahoo! in 2006 as a front end engineer and has been working on one of the most trafficked pages on the interwebs, the Yahoo! home page. His blog (nczonline.net) is a treasure trove of information on all things JavaScript & web performance. Some recent gems include Interviewing the front-end engineer & Writing maintainable code. It goes without saying that he knows his stuff when it comes to JavaScript & performance. As his books and blog posts have shown, he’s also a very skilled technical writer, keeping topics fresh, concise, & relevant.

I’m writing this as I read along, so the verbosity of this post is due to me taking reference notes on interesting things as I go.

Chapter 1: Loading & Execution

Specifically:

• Why is it important to put your <script> includes just above the closing </body> tag?


• What is the browser doing while loading those external files?


• Why should you put all your in-page JS code above your CSS includes? (A: If you put it after a </link> tag referencing an external stylesheet, the browser will block execution while waiting for that stylesheet to download)


• How you can use the defer attribute in <script> tags to delay non-essential execution of code.


• A thorough look at dynamic script loading to import & execute your JS without blocking the browser.


• An overview of some of the common JS loaders used today (YUI3, LazyLoader, & LABjs).

While much of the content in this chapter contains common knowledge among experienced developers, it is important knowledge to cover as it serves as the foundation for the rest of the book. Don’t worry, we’ll get more advanced.

Chapter 2: Data Access

Major topics covered in this chapter:

• Why do global variables perform so slowly?


• Why creating data as local variables as opposed to object properties is 10%-50% faster (depending on the browser).


• Why is it a good idea to create local instances of global variables?


• Why with, try/catch, and eval are bad ideas from a performance perspective. (A: they augment the scope by inserting themselves first on the tree)


• What truly happens under the hood when a variable is found to be undefined?


• Closure scope and why they can cause memory leaks.


• How prototype’s work and performance issues related to traversing up the prototype chain.


• Why is it bad to use deeply nested object members (i.e. foo.bar.baz.bop())?

There were so many “Ah hah! I get it now!” moments in this chapter for me that it alone was worth the price of the book. It took me about 5x as long as it should have to get through this chapter because I was too busy playing with Firebug as I began to learn some of these concepts.

Chapter 3: DOM Scripting

Many web developers don’t understanding what exactly “DOM scripting” is, even though they likely do it on a daily basis. Many could tell you what the acronym stands for and that it represents the structure of an (X)HTML/XML document, but most don’t know that it also represents the API part of how you interact with the document. When you are using document.getElementById(“foobar”) or myelement.style.color = “blue”, you are utilizing a DOM API function accessible via JavaScript, but it has nothing to do with the ECMAScript (aka: JavaScript) standard.

This chapter is chalk-full of great best practices & overviews of DOM principles. The first thing we learn is that accessing the DOM is so slow because we’re crossing the bridge between JavaScript and native browser code. Jumping between the two is expensive, and should be kept to a minimum. There are a lot of tricks & tips that are very under-utilized by most developers when DOM scripting.

For example:

• Using the non-standard innerhtml is way faster than creating nodes via the native document.createElement() method.


• When looping through a NodeCollection you should cache the length of the node in a local variable because it’s own length property is very slow.


• Iterating through nextSibling() can be 100x faster than using childNodes()

This chapter also goes into a detailed explanation of what repaint & reflow are, when they occur, and how understanding them will improve your application performance. The realization I had after reading the R&R explanation is we do stupid stuff all the time simply because we don’t understand how the browser renders and updates our pages. You know how you’ve always heard using margin-left & margin-right as separate styles is a bad idea? Well, here you find out why. Oh, and did you know there was a cssText property you can use to batch your CSS modifications? I didn’t.

As JS libraries get more & more popular, knowledge of good DOM scripting is becoming increasingly rare. Take event delegation for example. Many developers just presume jQuery’s live() or YUI3’s delegate() methods are just magic. They’re far from it, and are actually easy to understand concepts. When interviewing candidates for front end jobs at Yahoo!, this is one of the primary concepts we expect candidates to understand. They may have never used it, but the good ones will figure it out as they are whiteboarding and we walk them through the challenges.

JS libraries are awesome, but it’s because they abstract out the cross-browser differences & fix a flawed language, not because they allow you to forget what it actually going on under the hood.

Chapter 4: Algorithms & Flow Control

Next up are conditionals, such as if-else and switch. We learn when it is appropriate to use each one, and when they can be ditched for a much faster method, like using arrays as lookup tables.

Finally we come to the topic of recursion. We skip the basics of “What is recursion” and jump straight into browser limitations with call stacks and advanced recursion topics such as memoization to cut out the fat in your stack.

Since the majority of our time spent coding is inside of loops, conditionals, and (if we really want to optimize) recursion, this chapter has great, basic information on efficient shortcuts that will set you apart from the other developers on your team. Techniques learned in this chapter extend beyond the scope of JavaScript and can be used in just about every other programming language.

Chapter 5: Strings and Regular Expressions

Along with loops, another very common task in JavaScript is string manipulation, most commonly one by concatenation or regular expressions, so it makes sense to have a whole chapter to itself.

When most people start out with JS, their concatenation method is likely var str = “foo”; str = str + “bar”; //str = “foobar”, then they discover the += operator and it becomes var str = “foo”; str += “bar”; //str = “foobar”. It turns out that one of those is more efficient when it comes to memory usage, and it happens to not be the latter. This chapter provides some memory allocation table diagrams to explain the difference and how different browsers perform that operation. It should also be noted that another alternate method of concatenation, [‘foo’,’bar’].join(‘’); is the preferred method in IE 6 & 7, so that should be considered depending on your userbase.

The second half of this chapter covers regular expressions, which usually make me cringe. I have no problem writing them, but they’re an absolute nightmare to maintain sometimes. Douglas Crockford has a saying, “If a regular expression is longer than 2 inches, find another method.” I couldn’t agree more.

In this review, I only covered the first half of the book. Here are the remaining chapters:

• Chapter 6: Responsive Interfaces


• Chapter 7: Ajax


• Chapter 8: Programming Practices


• Chapter 9: Building and Deploying high performance JavaScript applications


• Chapter 10: Tools

If you like what you’ve seen so far, go buy it!

Wavy distorted omg we’re going into a flashback. Begin narration

My first experience on the Yahoo campus was for Y! HackDay 2008. I remember coming to the campus, being totally lost, and overwhelmed, almost like your first day of High School or College. I wasn’t an employee or anything. I was just a dumb programmer who wanted a taste of what Silicon Valley was really like. Seriously, I come from the startup world in Kansas City, I was in absolute awe of the place. This is where the Internet happens. Holy shit.

I came to HackDay armed with an idea for a hack to build, but was totally unable to focus, so I just sat around, tweeting, talking, and having fun. The music, the hacks, the food, the beer. I was totally awestruck when I talked to someone who worked at Yahoo!, especially the ones working on products I had used. I knew at that moment this was a place I’d always strive to work at. I knew I just had to work here, and be the person on the other end of that conversation.

Through the course of that weekend, I met a recruiter who for one reason or another took interest in my skills and said he’d follow up with me. I didn’t expect he would and he was just being nice. A couple weeks later I got a call from him stating he was interested in setting up an interview. I was shocked. “Ok, yeah, umm.. sure, anytime” I was so nervous before that first call. I reviewed just about every book I owned on programming, and I own a lot. I got the call and was speaking with an engineering manager who started asking me all sorts of questions about web development. In retrospect, I totally bombed it, and knew it. Rejected.

Down, but not out, I was focused, I knew it was attainable, but I just needed more time. So, over the next year I did just about everything I could to get my skills up to the level they needed to be for another crack at an interview, always keeping that original interview experience in mind. I had a blueprint. A plan.

A year later I got an email… “I’m back at Yahoo! Want another interview?” It was the original recruiter. “Yeah, absolutely.” The only goal I had this time was getting further than the first. I wouldn’t be totally bummed out if I didn’t get the job, but I at least wanted an on-site interview, just as validation I was making progress. Off I went, studying my ass off for about a week straight, so focused on the lone objective of nailing that phone-screen. The phone rang, and we started chatting. These questions were totally different from the first time. But that’s ok, I knew them. Apparently I did well, and I got an on-site.

The on-site (at the Santa Monica office) went well, and I got an offer. It was a big step leaving Kansas City, but one that I’d always regret if I stayed. So off I went, off to sunny SoCal. I started at the Santa Monica office working with the Entertainment team in November. Due to some mix-ups, I never did make it up here to Sunnyvale for training & orientation. Beyond that, there was never much need for me to be up here in person as we have tele-conferencing equipment galore, and these virtual meetings are in our DNA because we have offices around the country, and around the world.

So 5 months go by and I finally get up here for my first time. I’m actually glad I didn’t get up here before. I get to experience my first day at Yahoo, twice. I knew it was going to be weird, a good weird, and I knew that first time I came here was going to start flashing back. So here I am, sitting in the same booth, sipping my (free) mocha cappucino, admiring the courtyard, the weather, and the conversations going on around me. This is awesome. I have somewhere to be right now. But, nope…

If you haven’t set goals for yourself, do it. Set big ones. Set life-changing ones. When you achieve those, set higher ones, and just keep rolling. If you don’t have goals, find them. I stumbled across this one because I saw a tweet about HackDay, thought it sounded fun, and stepped on a plane to fly out here almost 2 years ago. Random. Lucky… Bold.

It’s feelings like this that you wish you could just bottle up and relive whenever you want.

So, I guess that’s the reason I’m writing this. A 30 minute slice of awesomeness, carved into this blog.