Three Tales of Second System Syndrome

In the last decade, three major scripting languages embarked on project to produce a major revision to each language: Perl 6, Python 3, and PHP 6. Despite surface similarities, such as the problem of Unicode support, each language ended up on a radically different track.

With the Perl 6.0.0 release officially coming this year, it's a good time to reflect on how we got to this point, and to start thinking about what comes after the release.

PHP 6

So -- and I can't believe I'm writing this -- let's see if we can learn something from PHP. Andi Gutmans, who is now the CEO of Zend Technologies, gave an interview back in February 2008. In it, he said,

So we are anticipating a long rollout cycle for PHP 6, and we did not want to take the same route that the Perl project did, with project contributors still working on Perl 6 I think six years later. People make fun of Microsoft, but take a look at Perl 6. . . .

To which Andy Lester of PerlBuzz replied:

Sure, PHP 6 may have a shorter release cycle than Perl 6 has, but at the end of it all, we'll have Perl 6, and you'll still have PHP.

Just sayin'.
xoxo,
Andy

So how did those predictions work out? Well, after a little over six years of development, we discovered that we were never going to see a PHP 6 at all. Having seen how long Perl 6 had taken, and how long PHP 6 was taking, the number 6 is associated with failure. So they cancelled PHP 6 and voted to change the name to PHP 7. Problem solved! No, really, this is some of the actual reasoning given by people on the 6 to 7 RFC. (Someone should tell the ES6 folks before the curse strikes our browsers!)

But the main intent of the renumbering was to justify a much reduced scope of new features for the next major version of PHP. PHP 7 is slated to add:

• "Huge Performance Improvements" to the Zend engine (HHVM already pretty speedy)
• JIT to the Zend engine (but already available in HHVM)
• Abstract Syntax Tree (AST) generation
• Asynchronous IO and functions
• Standalone Multi-threading Web Server (HHVM)
  • Which is cool if you want a language to provide its own server, I guess

EDIT: People both here and on Hacker News have pointed out that this is the above feature list was from a bad source, and that much of PHP 6 was incorporated into 5.3.  See the better summary of PHP 7 features, including generator improvements, and new operators like ??.  However, much of the same analysis still applies -- the end result was very few backwards incompatible changes, not the major revision promised with major Unicode improvements.

Perl 6

Meanwhile Perl 6, which has taken 15 years to get to the 6.0.0 release slated for this Christmas.  I'm sure that there were some embarrassing quotes about when it's going to be done, but that was so long ago, I'll just link to this post forecasting that Perl 6 will be ready for production in 2027.

As it now stands, Perl 6 comes with this set of new features:

• A real type system (not just type hints as in PHP) everywhere
  • The ability to continue ignoring types most short scripting code
  • The ability to use static type checks to catch errors
  • Native types (C strings, unsigned ints, etc.) unlock new performance potential
  • Meta-object programming is available
• A sane native function calling interface
• Rakudo Perl 6 runs on multiple virtual machines (JVM, MoarVM), with more backends planned (Javascript)
  • Can compile P6 to bytecode or an AST
  • Takes advantage of VM's JIT
• Complete syntax refactoring
  • Fully backwards compatible with the use of Inline::Perl5
  • Consistent syntax throughout
• Native Unicode handling, with NFG (grapheme) strings by default
• Hygienic macros
  • As I said above, AST
• Regexes have evolved to Grammars, a first class language
  • PCRE are no longer Perl-compatible (except in Perl 5 mode), but are instead much easier to read
• Easy to use concurrency
  • Some operators will autothread
  • Junction types for set operations
• Module versioning to ensure that even if a module completely changes its API, your code will not break. If you declare a version, of course.

Honestly, there are a whole lot more of these features. This even excludes things that have already made back into the Perl 5 core, like subroutine signatures and smartmatching. And these are all things that are working today.

The eerie thing is that Andy's flippant prediction came true. At the end of it, we have Perl 6, and they still have the same old PHP. Let me repeat that: we have Perl 6. It works, it will get a major release this year, and it is going to come with many more features than originally promised.

Still, Perl 6 has had its share of doubters. Some people proposed, actually seriously, that Perl 5 should leapfrog ahead to Perl 7 with the next version, and Perl 6 can go on calling itself that if it wants. Right. While this idea was rejected by the general Perl community, PHP actually skipped a version a year later. I guess it's another example of PHP stealing the worst ideas from Perl.

Python 3

The Python group, on the other hand, has tried to stay mostly on the side of sanity. Python 3 introduced a much smaller set of breaking changes, in order to keep updates rolling out. It was introduced, well, six years ago in early 2009.

New features of Python 3 included:

• Sane Unicode handling
  • A breaking change that allowed all of the other breaking changes to happen
• Various name changes for style consistency
• Automatically loading C modules when available.
• Refactor of exceptions
• Support for ancient OSes dropped
• Old functions removed, along with generally bad APIs
• Statement form print removed in favor of function print(), ostensibly to make a consistent API but really just to mess with people.

So how's that working out? The latest version of python preinstalled on my fully updated MacBook is 2.7.6. At least Ubuntu gives me 3.4.0 — Apple is well known to be crap at updating OSS. But you'd think someone at Apple would have cared in six years would have cared enough to throw python3 in the XCode monster download; after all, Python does not have the kiss of death known as the GPLv3 license.

The flip side of availability is developer adoption; this story isn't much better. If you look at statistics from a last year and this month, Python 3 adoption rates are abysmal. Hell, even 23% of people inside the Python community still think Python 3 was a mistake. Despite obvious improvements, it's still considered a tough sell.

Second Deployment Syndrome

So the takeaway from all of this is that Second System Syndrome is a real problem, but not the only problem. Successfully executing major revisions to a language is difficult, but getting widespread adoption is just as difficult with breaking changes. Second Deployment Syndrome can be just as hard to face as making the new system in the first place.

So we have three software communities that took radically different approaches to building a second system. PHP is a complete zoo of awful design, begging to be tamed. Yet the PHP community effectively voted to give up, and only offer incremental change that doesn't address PHP 6's number one issue of Unicode support. The Python folks, bless their hearts, made a smaller set of achievable changes, implemented it in 3 years, and shipped the damn thing. And despite the truly useful improvements, only a few people came.

Perl decided to stick to its vision of "break all the things once", and it's taken 15 long years. That's almost as long as the HTML 5 spec. Over this time, the design has continued to evolve, incorporating more modern needs like easily multithreaded code that would have otherwise been missed. Although the complaint of "no final spec" is common, it has been learned the hard way that the spec is the very last thing that should be finalized.

It's easy to naively say that 15 years is a ridiculous amount of development time, but it's obvious from looking at second systems for the other scripting languages, Perl 6 was never going to complete such a major transition in less than a decade. What's still unclear is whether this transition is going to work out for Perl.

Nearly everyone who tries Perl 6 from a Perl 5 background likes it immensely, which is usually followed up by a "can this not be so slow?" Optimization is still getting there, just not prematurely. In general, reception has been a net positive. And unlike the breaking changes introduced in the other languages, Inline::Perl5 allows multiple versions of Perl to coexist in the same program.

Will this be enough? It's too early to tell. Perl 5 is going to last another 5 years at the minimum, if not forever, munging text output by a shell script written by a programmer from generations ago. Perl 6 will have an uphill battle with Perl 5 for ubiquity, legacy code, and language familiarity.

Adoption rate is the next big challenge facing Perl 6. There is a very real possibility that six years from now, Perl 5 will still be the dominant form of an ever shrinking faction of Perl users. After all, Python might be in the same boat right now. Perl needs to reverse an already existing downward trend, at least partially brought on by how frakking long Perl 6 took in the first place.

The best advice I can see for ensuring Perl 6's success is for Perl developers to start writing code in Perl 6. I mean now; it's definitely stable enough. Every module available within a year of release is going to be a major argument for people to try the new version. Getting Shit Done can win a lot of arguments.

After that, it's going to be a tough slog. Is it deployed enough places to distribute code in? Is there enough code written in it to deploy to more places? Package managers like apt and Homebrew are going to help with bootstrapping the user base, but to win Perl 6 going to have to get that killer app.

So for now, it's a giant gamble. In poker terms, Python 3 called, PHP 6 folded, and Perl 6 went all-in. It just might be possible that Perl 6's crazy long development process can produce the best-adopted second system around, if people decide that the overwhelming improvements are worth the hassle of upgrading.

I'll let you know how that went in six years.