Three Tales of Second System Syndrome

In the last decade, three major scripting languages embarked on project to produce a major revision to each language: Perl 6, Python 3, and PHP 6. Despite surface similarities, such as the problem of Unicode support, each language ended up on a radically different track.

With the Perl 6.0.0 release officially coming this year, it's a good time to reflect on how we got to this point, and to start thinking about what comes after the release.

PHP 6

So -- and I can't believe I'm writing this -- let's see if we can learn something from PHP. Andi Gutmans, who is now the CEO of Zend Technologies, gave an interview back in February 2008. In it, he said,

So we are anticipating a long rollout cycle for PHP 6, and we did not want to take the same route that the Perl project did, with project contributors still working on Perl 6 I think six years later. People make fun of Microsoft, but take a look at Perl 6. . . .

To which Andy Lester of PerlBuzz replied:

Sure, PHP 6 may have a shorter release cycle than Perl 6 has, but at the end of it all, we'll have Perl 6, and you'll still have PHP.

Just sayin'.
xoxo,
Andy

So how did those predictions work out? Well, after a little over six years of development, we discovered that we were never going to see a PHP 6 at all. Having seen how long Perl 6 had taken, and how long PHP 6 was taking, the number 6 is associated with failure. So they cancelled PHP 6 and voted to change the name to PHP 7. Problem solved! No, really, this is some of the actual reasoning given by people on the 6 to 7 RFC. (Someone should tell the ES6 folks before the curse strikes our browsers!)

But the main intent of the renumbering was to justify a much reduced scope of new features for the next major version of PHP. PHP 7 is slated to add:

• "Huge Performance Improvements" to the Zend engine (HHVM already pretty speedy)
• JIT to the Zend engine (but already available in HHVM)
• Abstract Syntax Tree (AST) generation
• Asynchronous IO and functions
• Standalone Multi-threading Web Server (HHVM)
  • Which is cool if you want a language to provide its own server, I guess

EDIT: People both here and on Hacker News have pointed out that this is the above feature list was from a bad source, and that much of PHP 6 was incorporated into 5.3.  See the better summary of PHP 7 features, including generator improvements, and new operators like ??.  However, much of the same analysis still applies -- the end result was very few backwards incompatible changes, not the major revision promised with major Unicode improvements.

Perl 6

Meanwhile Perl 6, which has taken 15 years to get to the 6.0.0 release slated for this Christmas.  I'm sure that there were some embarrassing quotes about when it's going to be done, but that was so long ago, I'll just link to this post forecasting that Perl 6 will be ready for production in 2027.

As it now stands, Perl 6 comes with this set of new features:

• A real type system (not just type hints as in PHP) everywhere
  • The ability to continue ignoring types most short scripting code
  • The ability to use static type checks to catch errors
  • Native types (C strings, unsigned ints, etc.) unlock new performance potential
  • Meta-object programming is available
• A sane native function calling interface
• Rakudo Perl 6 runs on multiple virtual machines (JVM, MoarVM), with more backends planned (Javascript)
  • Can compile P6 to bytecode or an AST
  • Takes advantage of VM's JIT
• Complete syntax refactoring
  • Fully backwards compatible with the use of Inline::Perl5
  • Consistent syntax throughout
• Native Unicode handling, with NFG (grapheme) strings by default
• Hygienic macros
  • As I said above, AST
• Regexes have evolved to Grammars, a first class language
  • PCRE are no longer Perl-compatible (except in Perl 5 mode), but are instead much easier to read
• Easy to use concurrency
  • Some operators will autothread
  • Junction types for set operations
• Module versioning to ensure that even if a module completely changes its API, your code will not break. If you declare a version, of course.

Honestly, there are a whole lot more of these features. This even excludes things that have already made back into the Perl 5 core, like subroutine signatures and smartmatching. And these are all things that are working today.

The eerie thing is that Andy's flippant prediction came true. At the end of it, we have Perl 6, and they still have the same old PHP. Let me repeat that: we have Perl 6. It works, it will get a major release this year, and it is going to come with many more features than originally promised.

Still, Perl 6 has had its share of doubters. Some people proposed, actually seriously, that Perl 5 should leapfrog ahead to Perl 7 with the next version, and Perl 6 can go on calling itself that if it wants. Right. While this idea was rejected by the general Perl community, PHP actually skipped a version a year later. I guess it's another example of PHP stealing the worst ideas from Perl.

Python 3

The Python group, on the other hand, has tried to stay mostly on the side of sanity. Python 3 introduced a much smaller set of breaking changes, in order to keep updates rolling out. It was introduced, well, six years ago in early 2009.

New features of Python 3 included:

• Sane Unicode handling
  • A breaking change that allowed all of the other breaking changes to happen
• Various name changes for style consistency
• Automatically loading C modules when available.
• Refactor of exceptions
• Support for ancient OSes dropped
• Old functions removed, along with generally bad APIs
• Statement form print removed in favor of function print(), ostensibly to make a consistent API but really just to mess with people.

So how's that working out? The latest version of python preinstalled on my fully updated MacBook is 2.7.6. At least Ubuntu gives me 3.4.0 — Apple is well known to be crap at updating OSS. But you'd think someone at Apple would have cared in six years would have cared enough to throw python3 in the XCode monster download; after all, Python does not have the kiss of death known as the GPLv3 license.

The flip side of availability is developer adoption; this story isn't much better. If you look at statistics from a last year and this month, Python 3 adoption rates are abysmal. Hell, even 23% of people inside the Python community still think Python 3 was a mistake. Despite obvious improvements, it's still considered a tough sell.

Second Deployment Syndrome

So the takeaway from all of this is that Second System Syndrome is a real problem, but not the only problem. Successfully executing major revisions to a language is difficult, but getting widespread adoption is just as difficult with breaking changes. Second Deployment Syndrome can be just as hard to face as making the new system in the first place.

So we have three software communities that took radically different approaches to building a second system. PHP is a complete zoo of awful design, begging to be tamed. Yet the PHP community effectively voted to give up, and only offer incremental change that doesn't address PHP 6's number one issue of Unicode support. The Python folks, bless their hearts, made a smaller set of achievable changes, implemented it in 3 years, and shipped the damn thing. And despite the truly useful improvements, only a few people came.

Perl decided to stick to its vision of "break all the things once", and it's taken 15 long years. That's almost as long as the HTML 5 spec. Over this time, the design has continued to evolve, incorporating more modern needs like easily multithreaded code that would have otherwise been missed. Although the complaint of "no final spec" is common, it has been learned the hard way that the spec is the very last thing that should be finalized.

It's easy to naively say that 15 years is a ridiculous amount of development time, but it's obvious from looking at second systems for the other scripting languages, Perl 6 was never going to complete such a major transition in less than a decade. What's still unclear is whether this transition is going to work out for Perl.

Nearly everyone who tries Perl 6 from a Perl 5 background likes it immensely, which is usually followed up by a "can this not be so slow?" Optimization is still getting there, just not prematurely. In general, reception has been a net positive. And unlike the breaking changes introduced in the other languages, Inline::Perl5 allows multiple versions of Perl to coexist in the same program.

Will this be enough? It's too early to tell. Perl 5 is going to last another 5 years at the minimum, if not forever, munging text output by a shell script written by a programmer from generations ago. Perl 6 will have an uphill battle with Perl 5 for ubiquity, legacy code, and language familiarity.

Adoption rate is the next big challenge facing Perl 6. There is a very real possibility that six years from now, Perl 5 will still be the dominant form of an ever shrinking faction of Perl users. After all, Python might be in the same boat right now. Perl needs to reverse an already existing downward trend, at least partially brought on by how frakking long Perl 6 took in the first place.

The best advice I can see for ensuring Perl 6's success is for Perl developers to start writing code in Perl 6. I mean now; it's definitely stable enough. Every module available within a year of release is going to be a major argument for people to try the new version. Getting Shit Done can win a lot of arguments.

After that, it's going to be a tough slog. Is it deployed enough places to distribute code in? Is there enough code written in it to deploy to more places? Package managers like apt and Homebrew are going to help with bootstrapping the user base, but to win Perl 6 going to have to get that killer app.

So for now, it's a giant gamble. In poker terms, Python 3 called, PHP 6 folded, and Perl 6 went all-in. It just might be possible that Perl 6's crazy long development process can produce the best-adopted second system around, if people decide that the overwhelming improvements are worth the hassle of upgrading.

I'll let you know how that went in six years.

Camelia at Age 13: Perl 6 on the JVM Debuts

Perl 6 is now thirteen years old.  And she's very much a teenager in attitude, self-confident yet still growing up.  This contrasts with Javascript, which emerged from Brendan Eich's head, fully-formed like Athena -- but that only shared the Zeus-sized headaches with everyone until JQuery came along.

But Camelia, as she is fondly referred to by fans of Perl 6, is growing up fast.  Both too fast, and not fast enough.  To some of the community, the prospect of major changes to the language is scary.  Perl 6 is trying all of these crazy new things -- invariant sigils, metaoperators, grammars. She's even doing subroutine signatures, because "all of her friends are doing it".

They can't stay little children forever, you know.

And teenagers are liable to do surprising things.  So it was, that this week we announced Rakudo Perl 6 now runs on the Java Virtual Machine (JVM).  It's not perfect yet, but 62% of the files in the spectest pass as of yesterday.  Given the rate things are progressing, I'm sure it's already at a higher pass percent.

And yet, I'm sure there is no small number of you whose first thought about Perl on the JVM was "Heresy!".

There are certainly good reasons to support this point of view.  Startup times are horrible at this early stage, still O(seconds), and much of that is JVM's overhead.  It has well known security issues.  And of course the major implementation is controlled by $corporation, who just wants to make money off it.  And why would we want to abandon open-source VMs?

Still, there are plenty of good reasons for the port.  One is that the JVM is ubiquitous, and many devices have a copy pre-installed.  Most of Java's stability issues have been dealt with, and it serves Perl's competitors well enough through Jython and JRuby.  And it is well-documented, with bazillions of libraries (more than fooilions, anyway).  So we can finally address the longstanding desires of the community in things like sockets and threading, because we can tell the difference between our mistakes and those of the VM.

Instead of thinking of Perl 5 as a "sister language", I like to think of it as Camelia's father instead.  A father that might be kind of upset that she brought "that kind of language" into our house.  But she has a mind of her own, and I'm afraid that this won't be the only boyfriend she'll be bringing home.  There is a GSoC grant to build a Javascript backend to Rakudo.  And Niecza Perl 6 already uses the .NET Runtime.

However, Perl 6 is not abandoning open-source and C language implementations, either.  The announcement of MoarVM shows that Perl 6 developers plan to develop a lightweight VM specifically for NQP and Rakudo.  C functions will be directly callable within Perl itself, with the NativeCall interface.

Now, if Parrot fly off on its own course, that's Parrot's call.  You know how these teenage relationships go -- this could end up in a blow-up on the quad, or just as easily turn into hot makeup coding.  What, you didn't think I was going to say something that would violate the CoC, did you?

But Perl 6 is not done growing yet.  Camelia, like other teenagers, cares about fashion and wishes she had better threads.  And, once we get tuits, this is pretty high priority.  Because any modern language needs to live in the multi-core reality.  This is something that we can still design around,  that may not have recieved the same care ten years ago.  Many threading features are already baked into the language, like hyper operators and async blocks.

So I view the debut of the JVM port as Rakudo's real début, as with a debutante.  A treceañera, if you will.  I guess, given that she's 13, maybe it's a Bar Mitzvah -- except that she's not a boy, she's a butterfly.  But this is a chance acknowledge Perl 6's presence in the language scene.  Of course, these coming-of-age ceremonies don't mean the teenager is truly grown up yet. 

But grow up she will, and faster than some of you might like.  Perl 6 is rebellious, and changes things that those in her father's Perl 5 community don't understand.  But if you talk to the pumpkings, they sincerly hope that Camelia doesn't turn out exactly like her father.

After all, do we want keep the ithreads model?  Do we want to modules that dig into the compiler internals like XS does?  Perl 5 isn't perfect, we are just accustomed to its particular idiosyncrasies.

But for all that Perl 6 is different, she still loves her father.  We still have sigils, classes, @_ in subs (if you still want it), P5-style regexes, modules, and TIMTOWTDI.  It's still Perl.  Moreover, there are at least two efforts to run Perl 5 code in Perl 6 -- the v5 grammar/compiler in Perl 6, and access to libperl from MoarVM.  So the sky isn't falling on compatibility.

Nor is the other extreme true: Perl 6 development is in fact moving forward at a fast pace.  We know that Perl 6 is late.  Very late.  So late, in fact, that it's her father that's going to turn into a Pumpkin.  But when Perl 6 finally comes of age -- sooner than you think -- it will be something that the Perl community will be proud of.

And I apologize in advance for anything Camelia does in her bratty teenage years.

Thanking the Perl Community for an Awesome YAPC

My first time at YAPC::NA was too incredible for words. That said, because this is a blog, I'm going to have to put it in words anyway.

So, about a month ago, I didn't even know that I was even going to YAPC.  I was just talking to raiph++ in an IRC back channel, when he asked me if I was going to the conference.  I said that it would be fun, but I didn't really have money to go.  Being unemployed means lots of free time for hacking, but not so much free money for going places.

Well, raiph told diakopter++, who asked if I could be willing to go, if he found funds.  I responded, "Of course, if you think it's possible."  I soon went to sleep, and twelve hours later, I had a plane ticket to Austin in my inbox courtesy of The Perl Foundation.   So just like Peter Rabbitson's case, the Perl community eagerly gave me a chance to attend.  So thank you to all 440 attendees, and all of the sponsors for your own personal contribution to my attendance.  Even though I'm new, you all gave a me a chance to participate in the community, and for that I am grateful.

And what a community it is.  I've long known that Perl programmers were a little strange.  Naturally, I fit right in.

The conference itself had quite a fun and informative series of talks.  More often than not, I had two or more that I wanted to attend at the same time.  For the most part, I stuck to the Perl 6 "track", where most of my work has been so far.  After all, it's not often that so many of the European contingent make a trip to our humble continent, so I was eager to spend time with them.

No one warned me that jnthn++ has a tendency to spring wild new features on us at YAPCs.  Reversible grammars, seriously?!  I'm still trying to wrap my head around that one.  The announcement of MoarVM was equally exciting, as it offers us a chance to start fresh with everything we learned about Perl 6 and virtual machines in the last five years.

So I have to say diakopter++ once again.  Besides introducing Moar in his talk, Matthew Wilson was constantly busy behind the scenes, making sure that everything ran smoothly the entire conference.  I think the man must be buying tuits on the black market.

YAPC also helped immensely in my hunt for a job.  The job fair brought me several contacts, and the talks helped me learn which skills I'll really need to learn in those jobs.  Person-to-person contact offers so much more in truly understanding the state of the language, and of what projects are of the greatest use right now.

Truly, Perl's community is it's greatest strength. 

It's the community that keeps Perl vital.  After seeing YAPC for myself, the whole "Perl is Dead" meme seems entirely baseless.  Conference attendance was up 19% over last year, which was the previous record high.  Perl feels like a growing language, with lots of experiments in how to revitalize the now 25-year-old syntax with p2, moe, and Perl 6.

The community keeps Perl relevant.  While it may not be the sole alternative to bash scripts like it once was, it is used for enterprise and homebrew projects alike, from the stock exchange to the surface of Mars.  Projects like DBIx, Moose, and Dancer provide modern frameworks to acomplish more with less work.

The community keeps Perl open.  No one seemed to be afraid to say what they felt, on CGI.pm or anything else, but everyone remained civil.  Hallway++ is a great social hack to get everyone to feel comfortable talking to each other.  So when I found myself sitting across from TimToady, instead of being intimidated as a newbie, I had a great conversation with him about supervolcanoes and nonverbal Japanese language.

And the community really wants all of the projects to succeed.  I spent a lot of time at non-profit and political events in the past, where we were all theoretically working for a common cause.  And yet scheming, conflict, and political maneuvering were inevitable.  But in Perl, where we actually have multiple implementations and modules competing for mindshare and tuits, people cheer for everything to succeed.  No one fights each other or rages against $language_of_the_week stealing our users, for the real enemy is the lack of tuits.

I overheard this at dinner last night, from a fellow first-time attendee:

"I'm just happy that the two of you liked my work." -- vanstyn

Although he was talking about DBIx, I think that captures the spirit of conference as a whole.  All of us here -- from the n00bs to the pumpkings -- want to share our work and make something useful for others.  It's not an organization where we wait for pronouncements from on high, but one where users create endless variations and share them.  Not an organization so much as a family.

During TimToady's epistle/speech to the community, he said something like:

"We have faith, hope, and love, but the most awesome of these is love." -- Larry Wall

A line like this might seem a bit hokey out of context, but it was actually moving when I heard it.  We have faith that we can use Perl to solve our problems.  We have hope that Perl 5 and 6 will continue to get better. And we love Perl, unconditionally, despite all of her flaws.  And as Wil Wheaton says about us geeks, we just want to love our special thing the best we can, and go the extra mile to share it with others.

I just want to say that I love the Perl community right back.  You went out of your way to include me and all the other newcomers.  You all gave me all a chance to learn, play, and code with you -- and to be part of your community -- and I am so glad you did.