Menu
March 25th, 2016

What The Left-Pad Incident Teaches Us About JavaScript Dependencies

Mobile App Testing Learn tips, techniques and trends for launching great mobile apps Get It Now
Like what you're reading? Subscribe to the weekly newsletter. Subscribe For Free

The power and pitfalls of open source programming.

In 2016, you can build basically any app or other piece of software without writing an original piece of code. Just plug in to your favorite code library or package registry and cut, paste and compile.

Nearly all developers do it. Why write your own functions for a database access layer (a fairly complex set of code) when you can just depend on the work that somebody has done before?

The problem is that today’s software—especially JavaScript—relies on a series of dependencies upon dependencies upon dependencies. When a function is performed, the program will call upon these dependencies—often from disparate sources—and they will all come together and make the program work. If any one of these dependencies break, the whole string goes to hell.

And these days, developers are relying on dependencies for even the simplest of functions.

The results of this system were on painful display this week when a developer with a relied-upon NPM module pulled all of his code off the NPM registry over a copyright naming issue with a popular messaging app. The module was only 11 lines of code—called left-pad—but it almost brought the Internet to its knees.

Mobile Beta Management Learn the benefits, best practices and toolkits for mobile beta management Get It Now

What Happened With Left-Pad

An open source developer named Azer Koçulu had 273 published modules on the NPM registry at npmjs.org. One of those modules was named “kik” which, as people will likely know, is the name of a popular messaging app with more than 200 million users worldwide.

Kik, the company, decided it wanted to publish a npm module of its own under its own name. So, Kik emailed Koçulu asking him for the name. Koçulu basically told Kik to go shove it (the email exchange has been published here by the head of messenger at Kik Mike Roberts).

left_pad_emails_screen_shot

So Kik went straight to the people that run the npm registry and used what npm co-founder Isaac Z. Schlueter calls the organization’s standard “package name dispute resolution policy.” In the end, npm awarded the kik module name to Kik the company, citing the company’s brand and trademark.

Schlueter wrote:

The policy’s overarching goal is this: provide npm users with the package they expect. This covers spam, typo-squatting, misleading package names, and also more complicated cases such as this one. Entirely on this basis, we concluded that the package name “kik” ought to be maintained by Kik, and informed both parties.

So far, this followed a process that is routine, though rare. What happened next, though, was unprecedented.

What happened next is that Koçulu, feeling like he was being pushed around by big companies and ultimately thrown under the bus by npm, Inc. pulled all of his modules from the NPM registry. This included left-pad.

npm_left_pad_unpublished

Koçulu explained on Medium:

This situation made me realize that NPM is someone’s private land where corporate is more powerful than the people, and I do open source because, Power To The People.

Summary; NPM is no longer a place that I’ll share my open source work at, so, I’ve just unpublished all my modules.

This is not a knee-jerk action. I love open source and believe that open source community will eventually create a truly free alternative for NPM.

Emphasis by Koçulu.

For its part, npm, Inc. is going to have to look at its processes to keep this type of disruption from happening in the future.

“NPM could begin overriding these decisions to unpublish a module, in which case, they would republish the module that the developer removed,”said Brian Rinaldi, developer content manager at Telerik, in an email to ARC. “However, with this and other potential solutions to reinforce the system, we get into the sticky issue of ownership rights, which a lot of people are concerned about.”

The Wild World Of JavaScript

At ARC, we write a lot about JavaScript. We’ve covered how prominent developers like Parse CTO Kevin Lacker thinks JavaScript is eating the world. How long time Intel developer and now head of the Open Connectivity Foundation Mike Richmond thinks that JavaScript is perfectly suited for the Internet of Things. How Microsoft built a JavaScript engine called ChakraCore and most recently how JavaScript has become the most-used programming language in the world, according to a 50,000 (or so) developer survey from Stack Overflow.

JavaScript is hot. And it is hot for a reason.

See also: JavaScript, HTML5 And The Web Made Big Comebacks In 2015

JavaScript, created in 10 days in 1995 by Mozilla co-founder Brendan Eich in 1995 while at Netscape, has long been a bit of a bane for developers. It has been the de facto language of Web browsers since basically the beginning of Web browsers, even though Microsoft refused to support it in Internet Explorer (which dominated all Web browsers for more than a decade) for years. Different browsers support varying practices for the same functionality in JavaScript.

In comparison to other programming languages, JavaScript is non-intuitive. Longtime developers call it a joke, a toy, a tool foisted upon them that they don’t really want. But, as yet, nobody has ever been able to supplant JavaScript and even the alternative scripting languages are often correlated to JavaScript.

The thing about JavaScript though is that these days it is the most effective language for cross-platform programming. Web pages and Web apps are built with JavaScript. Mobile and hybrid apps are built with JavaScript. The back-end processes running all of these apps are starting to be built with JavaScript through the modern marvel that is Node.js. React is a growing and popular framework based on JavaScript and championed by Facebook to build cross-platform functionality (the “write once, run everywhere” dream). In 2016, you can learn JavaScript and become a full-stack developer in one swoop … an unthinkable prospect just a couple years ago.

One of the major problems with JavaScript these days is the lack of what developers call “truths.” Despite the “Harmony” initiative and EMCAScript 5 and the still-developing EMCAScript 6, the same functions in JavaScript can be done in different ways. There is no standard library where you can plug-and-play certain basic functions and know for a certainty that it won’t break.

If you are building in Android, you can use the libraries in Android Studio or Eclipse and know that basic functionality will work, essentially bug free. Same with writing iOS apps in Objective-C or Swift through Apple’s Xcode or Windows apps in .NET through Visual Studio. Developers know that something as simple as a left-pad function (which takes a string of characters and adds padding of the string on the left) in those environments will just work.

JavaScript has many, many libraries and repositories like jQuery. The most important these days is NPM, run by npm, Inc. out of Oakland, California. What npm does is basically provide a platform (like an app store) for open source developers to publish modules that can be downloaded by anybody and used in their software projects.

When an NPM module is called upon from the registry, it will provide its functionality to the software that is making the call. This where dependencies come in. The problem is if a module is broken or, in the case of Koçulu’s left-pad completely de-listed, it will then break the software that is making the call.

The App Quality Imperative Creating Apps that Win - 5 Challenges and 5 Solutions Get It Now

Koçulu’s left-pad had been downloaded more than 2.55 million times in the last month. The left-pad NPM was included in Babel, a package of NPM modules that has become very popular. Babel’s creator now works at Facebook. The React framework depends on Babel. In turn, massive software operations like Netflix, Spotify, Facebook, Reddit, AirBnB, Mozilla, Flipboard, MongoDB, HubSpot, AuthO and more rely on Babel. When left-pad was pulled by Koçulu, Babel could no longer be downloaded when called … and thus the Internet “broke” for about an hour.

“Most people who install these dependencies tend to download and work with them locally, so to a certain extent, the level of failure may not have been as significant as many believe,” said Telerik’s Rinaldi.

“At the same time, because left-pad was a dependency of projects that were a dependency of other projects, the impact of it being unpublished was much more wide spread than it probably would have been otherwise. Specifically, people pushing code to do a build on a continuous integration server were impacted, since dependencies would fail when the server tried to pull them, meaning the entire build failed,” Rinaldi said.

The open-source community moved quickly to fix the left-pad issue and the NPM registry quickly added a place holder in its spot to make sure the issue did not last long.

The Problem With JavaScript Dependencies

Fatalistic developers like to make morbid jokes that the entire software ecosystem is built, basically, on a house of cards. The whole paradigm is extremely fragile.

The left-pad incident highlights this particular notion. One spat between an open source developer and a large messaging app brought the Internet to its knees.

At the same time, the thing that makes the system fragile is also what gives it strength: community. Like a human body mobilizes antibodies to fight infection, the developer community got together and fixed the problem. The speed of the self-correction by the organism was impressive.

At the same time, people are questioning how we got to this point in the first place. The left-pad module was 11 lines of code. Thousands of modules in the npm registry are smaller and serve equally minute functions, including many of Koçulu’s 272 other modules that he pulled (some being only one line of code). The question becomes: why rely on dependencies for tasks that should be covered by basic programming knowledge?

Popular engineering manager David Haney of Stack Overflow states:

Finally, stringing APIs together and calling it programming doesn’t make it programming. It’s some crazy form of dependency hacking that involves the cloud, over-engineering things, and complexity far beyond what’s actually needed.

What’s worse is that if any of your code (or the 3rd party library code) has a bug or breaks, you won’t know how to debug or fix it if you don’t know how to program.

On the other hand, developers are just making the best of what is inherently a bad situation, created by JavaScript itself. Unlike other programming languages, JavaScript does not easily provide many of the simple functions included in other languages. To learn every single function and exception in JavaScript would require years.

As Haney’s Stack Overflow colleague Adam Maras points out, developers have three options:

  1. Write your own function from scratch
  2. Take a dependency on a library that contains the function
  3. Copy and paste someone else’s implementation into your project

All three of those options are valid. And it highlights the fact that JavaScript is so big, influential and, yes, unruly, that entities like the NPM registry are important.

At the same time, Haney makes a good point in that many of the modules in the NPM registry are single function created with a few lines of code. These are not massive packages that would take developers a long time to create on an individual basis. If something like un-publishing of left-pad can have such a dramatic effect on the Internet, developers of the bigger packages (like Babel and React) are going to have to reassess how many of these single-purpose modules created by third-party developers and managed by third-party registries they actually include in their packages.

Lead image: “Crash” by Flickr user Le Vin Parfait, Creative Commons