Sunday, May 3, 2015

Three Tales of Second System Syndrome

In the last decade, three major scripting languages embarked on project to produce a major revision to each language: Perl 6, Python 3, and PHP 6. Despite surface similarities, such as the problem of Unicode support, each language ended up on a radically different track.

With the Perl 6.0.0 release officially coming this year, it's a good time to reflect on how we got to this point, and to start thinking about what comes after the release.


So -- and I can't believe I'm writing this -- let's see if we can learn something from PHP. Andi Gutmans, who is now the CEO of Zend Technologies, gave an interview back in February 2008. In it, he said,
So we are anticipating a long rollout cycle for PHP 6, and we did not want to take the same route that the Perl project did, with project contributors still working on Perl 6 I think six years later. People make fun of Microsoft, but take a look at Perl 6. . . .

To which Andy Lester of PerlBuzz replied:
Sure, PHP 6 may have a shorter release cycle than Perl 6 has, but at the end of it all, we'll have Perl 6, and you'll still have PHP.

Just sayin'.
So how did those predictions work out? Well, after a little over six years of development, we discovered that we were never going to see a PHP 6 at all. Having seen how long Perl 6 had taken, and how long PHP 6 was taking, the number 6 is associated with failure. So they cancelled PHP 6 and voted to change the name to PHP 7. Problem solved! No, really, this is some of the actual reasoning given by people on the 6 to 7 RFC. (Someone should tell the ES6 folks before the curse strikes our browsers!)

But the main intent of the renumbering was to justify a much reduced scope of new features for the next major version of PHP. PHP 7 is slated to add:
  • "Huge Performance Improvements" to the Zend engine (HHVM already pretty speedy)
  • JIT to the Zend engine (but already available in HHVM)
  • Abstract Syntax Tree (AST) generation
  • Asynchronous IO and functions
  • Standalone Multi-threading Web Server (HHVM)
    • Which is cool if you want a language to provide its own server, I guess
EDIT: People both here and on Hacker News have pointed out that this is the above feature list was from a bad source, and that much of PHP 6 was incorporated into 5.3.  See the better summary of PHP 7 features, including generator improvements, and new operators like ??.  However, much of the same analysis still applies -- the end result was very few backwards incompatible changes, not the major revision promised with major Unicode improvements.

Perl 6

Meanwhile Perl 6, which has taken 15 years to get to the 6.0.0 release slated for this Christmas.  I'm sure that there were some embarrassing quotes about when it's going to be done, but that was so long ago, I'll just link to this post forecasting that Perl 6 will be ready for production in 2027.

As it now stands, Perl 6 comes with this set of new features:
  • A real type system (not just type hints as in PHP) everywhere
    • The ability to continue ignoring types most short scripting code
    • The ability to use static type checks to catch errors
    • Native types (C strings, unsigned ints, etc.) unlock new performance potential
    • Meta-object programming is available
  • A sane native function calling interface
  • Rakudo Perl 6 runs on multiple virtual machines (JVM, MoarVM), with more backends planned (Javascript)
    • Can compile P6 to bytecode or an AST
    • Takes advantage of VM's JIT
  • Complete syntax refactoring
    • Fully backwards compatible with the use of Inline::Perl5
    • Consistent syntax throughout
  • Native Unicode handling, with NFG (grapheme) strings by default
  • Hygienic macros
    • As I said above, AST
  • Regexes have evolved to Grammars, a first class language
    • PCRE are no longer Perl-compatible (except in Perl 5 mode), but are instead much easier to read
  • Easy to use concurrency
    • Some operators will autothread
    • Junction types for set operations
  • Module versioning to ensure that even if a module completely changes its API, your code will not break. If you declare a version, of course.
Honestly, there are a whole lot more of these features. This even excludes things that have already made back into the Perl 5 core, like subroutine signatures and smartmatching. And these are all things that are working today.

The eerie thing is that Andy's flippant prediction came true. At the end of it, we have Perl 6, and they still have the same old PHP. Let me repeat that: we have Perl 6. It works, it will get a major release this year, and it is going to come with many more features than originally promised.

Still, Perl 6 has had its share of doubters. Some people proposed, actually seriously, that Perl 5 should leapfrog ahead to Perl 7 with the next version, and Perl 6 can go on calling itself that if it wants. Right. While this idea was rejected by the general Perl community, PHP actually skipped a version a year later. I guess it's another example of PHP stealing the worst ideas from Perl.

Python 3

The Python group, on the other hand, has tried to stay mostly on the side of sanity. Python 3 introduced a much smaller set of breaking changes, in order to keep updates rolling out. It was introduced, well, six years ago in early 2009.

New features of Python 3 included:
  • Sane Unicode handling
    • A breaking change that allowed all of the other breaking changes to happen
  • Various name changes for style consistency
  • Automatically loading C modules when available.
  • Refactor of exceptions
  • Support for ancient OSes dropped
  • Old functions removed, along with generally bad APIs
  • Statement form print removed in favor of function print(), ostensibly to make a consistent API but really just to mess with people.
So how's that working out? The latest version of python preinstalled on my fully updated MacBook is 2.7.6. At least Ubuntu gives me 3.4.0 — Apple is well known to be crap at updating OSS. But you'd think someone at Apple would have cared in six years would have cared enough to throw python3 in the XCode monster download; after all, Python does not have the kiss of death known as the GPLv3 license.

The flip side of availability is developer adoption; this story isn't much better. If you look at statistics from a last year and this month, Python 3 adoption rates are abysmal. Hell, even 23% of people inside the Python community still think Python 3 was a mistake. Despite obvious improvements, it's still considered a tough sell.

Second Deployment Syndrome

So the takeaway from all of this is that Second System Syndrome is a real problem, but not the only problem. Successfully executing major revisions to a language is difficult, but getting widespread adoption is just as difficult with breaking changes. Second Deployment Syndrome can be just as hard to face as making the new system in the first place.

So we have three software communities that took radically different approaches to building a second system. PHP is a complete zoo of awful design, begging to be tamed. Yet the PHP community effectively voted to give up, and only offer incremental change that doesn't address PHP 6's number one issue of Unicode support. The Python folks, bless their hearts, made a smaller set of achievable changes, implemented it in 3 years, and shipped the damn thing. And despite the truly useful improvements, only a few people came.

Perl decided to stick to its vision of "break all the things once", and it's taken 15 long years. That's almost as long as the HTML 5 spec. Over this time, the design has continued to evolve, incorporating more modern needs like easily multithreaded code that would have otherwise been missed. Although the complaint of "no final spec" is common, it has been learned the hard way that the spec is the very last thing that should be finalized.

It's easy to naively say that 15 years is a ridiculous amount of development time, but it's obvious from looking at second systems for the other scripting languages, Perl 6 was never going to complete such a major transition in less than a decade. What's still unclear is whether this transition is going to work out for Perl.

Nearly everyone who tries Perl 6 from a Perl 5 background likes it immensely, which is usually followed up by a "can this not be so slow?" Optimization is still getting there, just not prematurely. In general, reception has been a net positive. And unlike the breaking changes introduced in the other languages, Inline::Perl5 allows multiple versions of Perl to coexist in the same program.

Will this be enough? It's too early to tell. Perl 5 is going to last another 5 years at the minimum, if not forever, munging text output by a shell script written by a programmer from generations ago. Perl 6 will have an uphill battle with Perl 5 for ubiquity, legacy code, and language familiarity.

Adoption rate is the next big challenge facing Perl 6. There is a very real possibility that six years from now, Perl 5 will still be the dominant form of an ever shrinking faction of Perl users. After all, Python might be in the same boat right now. Perl needs to reverse an already existing downward trend, at least partially brought on by how frakking long Perl 6 took in the first place.

The best advice I can see for ensuring Perl 6's success is for Perl developers to start writing code in Perl 6. I mean now; it's definitely stable enough. Every module available within a year of release is going to be a major argument for people to try the new version. Getting Shit Done can win a lot of arguments.

After that, it's going to be a tough slog. Is it deployed enough places to distribute code in? Is there enough code written in it to deploy to more places? Package managers like apt and Homebrew are going to help with bootstrapping the user base, but to win Perl 6 going to have to get that killer app.

So for now, it's a giant gamble. In poker terms, Python 3 called, PHP 6 folded, and Perl 6 went all-in. It just might be possible that Perl 6's crazy long development process can produce the best-adopted second system around, if people decide that the overwhelming improvements are worth the hassle of upgrading.

I'll let you know how that went in six years.


  1. Maybe they should have just called the languages something new entirely...

  2. This comment has been removed by the author.

  3. The information on PHP 6 and 7 is not entirely accurate.

    PHP 6 ultimately became PHP 5.3 (without unicode support), and was followed by 5.4, 5.5, 5.6, and now the upcoming 7. A large rewrite or refactoring of the engine prompted the need for a major version release, which started the debate of calling it 6 vs 7 due to the history with PHP 6.

    Also, the list of features for PHP 7 is very incorrect, aside from AST. The article linked is utter garbage. See:

    1. Thanks Kristopher, and sorry about that. Updated the post to reflect what you said. There is a lot of bad information about PHP out there, and it seems I mainly came across that in my research. I don't keep up with PHP development as much as I should, I just write the occasional Mediawiki extension with it.

  4. Whelp... I'm *just about* starting to adopt Python 3 riight about now ...

  5. Perl6 is available as rakudo package in Debian Jessie and Ubuntu. The version is oldish (2014-07) and based on deprecated parrot virtual machine. Nevertheless, this version is good enough to start learning Perl6.

    Debian rakudo team is currently working at updating Perl6 with moar virtual machine.

    All the best

    1. Rakudobrew gets you the latest Rakudo pakages (Perl 6 compiler with MoarVM backend) and the Panda module management tool.

  6. I would say that 6 years isn't going to be long enough to see the adoption curve of perl6 versus perl5. I expect adoption has an inverse reaction to size of the existing code base. [Probably something to the square of the existing installed base like a network law.] I have seen this with adoptions of newer PHP (it was easier to get PHP-3 -> PHP-4 than PHP-4 -> PHP-5.) Perl.. (it was easy to get perl3 -> perl4. I was finding perl4 systems into the 2000's so that perl5 code had to be written to make sure it could run on perl4).. python (jumps from python 2.n -> 2.n+1 was slower the larger the number of systems already running python).

  7. ES6 was renamed to ES2015.

  8. Superb write-up. Now, I want to dive into Perl 6, head-first. I want to unlearn everything, and learn the Perl 6 way. That said, is there yet a book that guides us, in a reasonably exhaustive manner, into Perl 6? And, is there a reasonable IDE? I cannot wait for the new world, because the old world was amazing.

    1. There aren't really a lot of good books out for Perl 6 right now. Most are years out of date, and include what turned out to be wrong information. Which is not to say that they weren't important at the time.

      The best place for learning Perl 6 right now is, which is growing daily and therefore not exhaustive. There are also the Synopses, more complete but intended for compiler writers. Still, I was able to learn Perl 6 from the Synopses, so they're a good option. Finally, if you know Perl 5, consider learning Moose. Many of the concepts here were backported from Perl 6; the Perl 6 equivalents are similar but have simplified syntax and actually run faster.

      The only IDE we really have the editor of champions, vim. There's pretty good Perl 6 support with vim-perl. (While I'm on the topic of syntax support, Pygments supports Perl 6 so Github can already correctly highlight code.) I suspect that's not what you're asking for in an IDE, but a lot of the toolchain is not yet there.

      And I think it's important that you don't unlearn everything. Perl 6 is a syncretic language. It has learned from Perl 5 (and therefore sed, awk, C, & shell); it has also taken from Haskell, JS, Ruby, and Java. The idea is thoroughly postmodern -- take the good parts from everything. So when you code Perl 6, try to use the best model (functional, object oriented, whatever), rather than relying on the Perl 6 way. We do like function names with hyphens in them, though.