The power and pitfalls of open source programming.
In 2016, you can build basically any app or other piece of software without writing an original piece of code. Just plug in to your favorite code library or package registry and cut, paste and compile.
Nearly all developers do it. Why write your own functions for a database access layer (a fairly complex set of code) when you can just depend on the work that somebody has done before?
And these days, developers are relying on dependencies for even the simplest of functions.
The results of this system were on painful display this week when a developer with a relied-upon NPM module pulled all of his code off the NPM registry over a copyright naming issue with a popular messaging app. The module was only 11 lines of code—called left-pad—but it almost brought the Internet to its knees.
What Happened With Left-Pad
An open source developer named Azer Koçulu had 273 published modules on the NPM registry at npmjs.org. One of those modules was named “kik” which, as people will likely know, is the name of a popular messaging app with more than 200 million users worldwide.
Kik, the company, decided it wanted to publish a npm module of its own under its own name. So, Kik emailed Koçulu asking him for the name. Koçulu basically told Kik to go shove it (the email exchange has been published here by the head of messenger at Kik Mike Roberts).
So Kik went straight to the people that run the npm registry and used what npm co-founder Isaac Z. Schlueter calls the organization’s standard “package name dispute resolution policy.” In the end, npm awarded the kik module name to Kik the company, citing the company’s brand and trademark.
The policy’s overarching goal is this: provide npm users with the package they expect. This covers spam, typo-squatting, misleading package names, and also more complicated cases such as this one. Entirely on this basis, we concluded that the package name “kik” ought to be maintained by Kik, and informed both parties.
So far, this followed a process that is routine, though rare. What happened next, though, was unprecedented.
What happened next is that Koçulu, feeling like he was being pushed around by big companies and ultimately thrown under the bus by npm, Inc. pulled all of his modules from the NPM registry. This included left-pad.
This situation made me realize that NPM is someone’s private land where corporate is more powerful than the people, and I do open source because, Power To The People.
Summary; NPM is no longer a place that I’ll share my open source work at, so, I’ve just unpublished all my modules.
This is not a knee-jerk action. I love open source and believe that open source community will eventually create a truly free alternative for NPM.
Emphasis by Koçulu.
For its part, npm, Inc. is going to have to look at its processes to keep this type of disruption from happening in the future.
“NPM could begin overriding these decisions to unpublish a module, in which case, they would republish the module that the developer removed,”said Brian Rinaldi, developer content manager at Telerik, in an email to ARC. “However, with this and other potential solutions to reinforce the system, we get into the sticky issue of ownership rights, which a lot of people are concerned about.”
— Chris Simpson (@CmdrShepardsPie) March 24, 2016
If you are building in Android, you can use the libraries in Android Studio or Eclipse and know that basic functionality will work, essentially bug free. Same with writing iOS apps in Objective-C or Swift through Apple’s Xcode or Windows apps in .NET through Visual Studio. Developers know that something as simple as a left-pad function (which takes a string of characters and adds padding of the string on the left) in those environments will just work.
When an NPM module is called upon from the registry, it will provide its functionality to the software that is making the call. This where dependencies come in. The problem is if a module is broken or, in the case of Koçulu’s left-pad completely de-listed, it will then break the software that is making the call.
Koçulu’s left-pad had been downloaded more than 2.55 million times in the last month. The left-pad NPM was included in Babel, a package of NPM modules that has become very popular. Babel’s creator now works at Facebook. The React framework depends on Babel. In turn, massive software operations like Netflix, Spotify, Facebook, Reddit, AirBnB, Mozilla, Flipboard, MongoDB, HubSpot, AuthO and more rely on Babel. When left-pad was pulled by Koçulu, Babel could no longer be downloaded when called … and thus the Internet “broke” for about an hour.
“Most people who install these dependencies tend to download and work with them locally, so to a certain extent, the level of failure may not have been as significant as many believe,” said Telerik’s Rinaldi.
“At the same time, because left-pad was a dependency of projects that were a dependency of other projects, the impact of it being unpublished was much more wide spread than it probably would have been otherwise. Specifically, people pushing code to do a build on a continuous integration server were impacted, since dependencies would fail when the server tried to pull them, meaning the entire build failed,” Rinaldi said.
The open-source community moved quickly to fix the left-pad issue and the NPM registry quickly added a place holder in its spot to make sure the issue did not last long.
Fatalistic developers like to make morbid jokes that the entire software ecosystem is built, basically, on a house of cards. The whole paradigm is extremely fragile.
The left-pad incident highlights this particular notion. One spat between an open source developer and a large messaging app brought the Internet to its knees.
At the same time, the thing that makes the system fragile is also what gives it strength: community. Like a human body mobilizes antibodies to fight infection, the developer community got together and fixed the problem. The speed of the self-correction by the organism was impressive.
At the same time, people are questioning how we got to this point in the first place. The left-pad module was 11 lines of code. Thousands of modules in the npm registry are smaller and serve equally minute functions, including many of Koçulu’s 272 other modules that he pulled (some being only one line of code). The question becomes: why rely on dependencies for tasks that should be covered by basic programming knowledge?
Popular engineering manager David Haney of Stack Overflow states:
Finally, stringing APIs together and calling it programming doesn’t make it programming. It’s some crazy form of dependency hacking that involves the cloud, over-engineering things, and complexity far beyond what’s actually needed.
What’s worse is that if any of your code (or the 3rd party library code) has a bug or breaks, you won’t know how to debug or fix it if you don’t know how to program.
As Haney’s Stack Overflow colleague Adam Maras points out, developers have three options:
- Write your own function from scratch
- Take a dependency on a library that contains the function
- Copy and paste someone else’s implementation into your project
At the same time, Haney makes a good point in that many of the modules in the NPM registry are single function created with a few lines of code. These are not massive packages that would take developers a long time to create on an individual basis. If something like un-publishing of left-pad can have such a dramatic effect on the Internet, developers of the bigger packages (like Babel and React) are going to have to reassess how many of these single-purpose modules created by third-party developers and managed by third-party registries they actually include in their packages.