Heh. Same here, though it’s only been 25 years for me (wow, “only” 25 years?).

The lesson here is that old habits die hard—and that different segments of the developer community have vastly different expectations and preferences when it comes to tools. Case in point: Despite all the advancements in tools over the past decade (and despite the fact that I ran marketing for NetBeans at Sun for one of those years, and that I’m a big proponent of an IDE centric “connected developer” approach for Sun), my “IDE” of choice is still a combination of command line tools and Emacs.

Indeed, as I’ve learned over the years, this presents an interesting challenge for platform vendors. While grassroots momentum in the early adopter community is key to building a successful platform these days (think Linux, MySQL, and Amazon Web Services, among others), and early adopters will put up with a remarkable amount of pain to target these platforms (Linux, MySQL, and Amazon Web Services developers have largely relied, at least in the beginning and in many cases still today, on command line tools), what I call “integrated development experience” is key to taking these platforms to the mass market—and indeed can be an immensely powerful competitive weapon.

Think about how Microsoft was able to successfully leverage its Windows/Visual Studio/MSDN triumvirate to bootstrap .NET in the early 2000s, and how it’s leveraging .NET/Visual Studio/MSDN to bootstrap Silverlight today using largely the same techniques. Or think about how Apple is entrenching itself in the mobile developer community by providing dead simple drag and drop tooling for the iPhone, along with a (mostly) seamless publication and monetization vehicle via the App Store. What would the iPhone platform look like without this dead simple tooling? Let’s just say there’s not a whole lot of demand in the early adopter community for digital flatulence applications.

Of course, the real question for the 21st century platform vendor—one that understands that massive adoption in the early adopter community is key to success—is when to transition from a wild west, bubble gum and bailing wire platform strategy to a more neatly packaged integrated development experience. I’ve long argued that making this transition is key to taking your platform to the next level. When to do it is the real question. Too early, and you damage the limitless possibilities that make your platform so appealing to early adopters. Too late, and you expose yourself to competition from vendors that do a better job at capitalizing on the broader opportunity (and that’s best case—worst case, your platform fades away because it is never able to grow beyond the early adopters).

No one ever said this business was simple..

Everything I know about computing I learned on those Sun workstations, as did so many other early Linux developers; I even had my own for a while, after I joined the University of Arizona computer science department in 1997. But within a year, the Suns were starting to disappear, replaced by Pentiums running Red Hat Linux. More and more people coming through university computer science programs were cutting their teeth on Linux, much as I had on Sun. Pretty soon, Sun was increasingly seen by this new generation as the vendor who didn’t “get it”, and Sun’s rivals did a masterful job running with that and painting the company literally built on open standards as “closed”. To those of us who knew better, it was a sad thing to watch.

The last several years have been hard for Sun, but the corner has been turned. As an outsider, I’ve watched as Sun has successfully embraced x86, pioneered energy efficiency as an essential computing feature, open sourced its software portfolio to maximize the network effects, championed transparency in corporate communications, and so many other great things. Now, I’m going to be a part of it.

And, so, I’m excited to announce that, as of today, I’m joining Sun to head up operating system platform strategy. I’m not saying much about what I’ll be doing yet, but you can probably guess from my background and earlier writings that I’ll be advocating that Solaris needs to close the usability gap with Linux to be competitive; that while as I believe Solaris needs to change in some ways, I also believe deeply in the importance of backward compatibility; and that even with Solaris front and center, I’m pretty strongly of the opinion that Linux needs to play a clearer role in the platform strategy.

It is with regrets that I leave the Linux Foundation, but if you haven’t figured out already, Sun is a company I’ve always loved, and being a part of it was an opportunity I simply could not pass up. I think the world of the people at the LF, particularly my former FSG colleagues with whom I worked so closely over the past year and a half: Jim Zemlin, Amanda McPherson, Jeff Licquia, and Dan Kohn. And I still very much believe in the core LF mission, to prevent the fragmentation of the Linux platform. Indeed, I’m remaining in my role as chair of the LSB—and Sun, of course, is a member of the Linux Foundation.

Anyway. Watch this space. This is going to be fun!

Mary Jo Foley: “Is it Intuit’s fault that its older products don’t work on Vista? Or Microsoft’s fault for changing the operating system in a way that it breaks them?”

George Ou: “As it turns out, it isn’t QuickBooks 2006 itself that has the problem with Vista but all the Intuit and third party add-on software that communicates with QuickBooks 2006 that’s the problem. More to the point, it’s the intercommunication between all those applications and the fact that they’re using forbidden techniques that have been banned since 2001 with Windows XP certification requirements that’s the issue. Intuit admitted to me that they declined to seek Windows XP certification for all these years and they’re just now making the necessary modifications for QuickBooks 2007. The reason this is relevant is because most software that is certified for Windows XP will automatically be compatible with Windows Vista.”

David Berlind: “So, while Microsoft’s response doesn’t point the finger directly at Intuit as the culprit in this situation, it makes it clear that its certification programs — programs that Intuit has apparently eschewed since 2001 — are the centerpiece of its efforts to guarantee backwards compatibility.”

Interesting discussion. Whose fault is it? It doesn’t really matter, as the end result is the same—Microsoft potentially loses customers (those who won’t upgrade to Vista because QuickBooks will break), and Intuit potentially loses customers (those who won’t upgrade to QuickBooks 2007 but rather will move to an on demand solution like NetSuite—customers tend to dislike forced upgrades).

David Berlind’s observation is interesting though, namely that Microsoft has been using its certification programs to limit the scope of the backward compatibility problem to a well defined subset around which it can presumably do thorough compatibility testing. Note that Red Hat too only guarantees backward compatibility for a subset of its platform, and that the LSB doesn’t cover the entire Linux platform either, just the subset that’s import for application portability.

What are the lessons here, from my point of view? First of all, and not surprisingly, backward compatibility matters, and it matters (or should matter) to both sides. Second, to achieve that backward compatibility, both sides have to work at it, and because a typical platform has so many moving parts, limiting the scope is key to understanding the “contract” between OS and application. Finally, some formal way of stating “I am following the contract”, such as a certification program, can make compatibility a lot easier to articulate to mutual customers.

Related: On the importance of backward compatibility, More on the importance of backward compatibility

Why is backward compatibility important? Here’s a great example, via Joel Spolsky (note: from 2004):

Raymond Chen is a developer on the Windows team at Microsoft. He’s been there since 1992, and his weblog The Old New Thing is chock-full of detailed technical stories about why certain things are the way they are in Windows, even silly things, which turn out to have very good reasons.

The most impressive things to read on Raymond’s weblog are the stories of the incredible efforts the Windows team has made over the years to support backwards compatibility: “Look at the scenario from the customer’s standpoint. You bought programs X, Y and Z. You then upgraded to Windows XP. Your computer now crashes randomly, and program Z doesn’t work at all. You’re going to tell your friends, ‘Don’t upgrade to Windows XP. It crashes randomly, and it’s not compatible with program Z.’ Are you going to debug your system to determine that program X is causing the crashes, and that program Z doesn’t work because it is using undocumented window messages? Of course not. You’re going to return the Windows XP box for a refund. (You bought programs X, Y, and Z some months ago. The 30-day return policy no longer applies to them. The only thing you can return is Windows XP.)”

I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. The testers on the Windows team were going through various popular applications, testing them to make sure they worked OK, but SimCity kept crashing. They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it.

This was not an unusual case. The Windows testing team is huge and one of their most important responsibilities is guaranteeing that everyone can safely upgrade their operating system, no matter what applications they have installed, and those applications will continue to run, even if those applications do bad things or use undocumented functions or rely on buggy behavior that happens to be buggy in Windows n but is no longer buggy in Windows n+1…

A lot of developers and engineers don’t agree with this way of working. If the application did something bad, or relied on some undocumented behavior, they think, it should just break when the OS gets upgraded. The developers of the Macintosh OS at Apple have always been in this camp. It’s why so few applications from the early days of the Macintosh still work…

To contrast, I’ve got DOS applications that I wrote in 1983 for the very original IBM PC that still run flawlessly, thanks to the Raymond Chen Camp at Microsoft.

I can almost feel the revulsion among my readership right about now. However, next time you’re in Best Buy or CompUSA, look at the shelf of Windows applications, then compare it to the shelf of Mac applications, and perhaps you’ll better understand why it’s important.

Beyond the results speaking for themselves, I’ll argue that it takes a better engineer to move a platform forward while at the same time making sure things don’t break. It’s pretty easy to wash your hands of something and declare it to be someone else’s problem.

More Steve Jobs: “These are devices that need to work, and you can’t do that if you load any software on them. That doesn’t mean there’s not going to be software to buy that you can load on them coming from us. It doesn’t mean we have to write it all, but it means it has to be more of a controlled environment.”

It’s widely understood that Microsoft beat Apple in the PC game because Apple was a tightly integrated, closed platform, whereas Microsoft was an open ecosystem where anyone could play (albeit at their own peril).

Clearly, the strategy has not adapted well to digital media players. Why? Because people are tired of products that don’t work. Anyone who has ever had to glue a PC back together for a family member knows what I mean.

Does “open” have to mean “poorly integrated” though? Clearly, I don’t think so, given how much I’ve invested in Linux over the years and how much I’m investing in the LSB now. And the job the distributions do in the Linux world is nothing short of amazing, if you think about it—taking thousands of moving parts written by thousands of different people and crafting them into what appears, for the most part, to be an orchestrated whole.

Personal experience bears this out too—it’s not that the entire Windows Media ecosystem produces bad products (iTunes pales in comparison to Rhapsody), it’s that the integration points are extremely fragile (e.g., I tried Rhapsody To Go with a Plays For Sure device, which was an awful experience). In other words, just like a chain, an open ecosystem is only as strong as its weakest link, because it’s the links where things break.

Sure, there are bad products. I’ve owned no less than two MP3 players in the last few years, and they were pretty terrible.

It’s the very, very simple things too. For example, I listen to a lot of podcasts. There’s (gasp!) a Podcasts menu on the iPod, whereas on every other device I’ve owned, podcasts are interspersed with the music, making them difficult to find and, adding insult to injury, mixing gobbledygook into my otherwise well organized collection. Furthermore (gasp!), the iPod keeps track of which podcasts I listen to, and in concert with iTunes, automatically deletes the ones I’m done with. Them’s lasers!

Ah, iTunes. Despite the fact that anyone in the world can write a podcatcher (it’s just RSS with enclosures, fer godsakes), the only one I’ve ever used that works worth a damn is iTunes. Everything else is just “integrated” with bailing wire—create a Windows Media playlist, drop it in this directory, etc.

The situation on Linux is a bit better, but guess what—Rhythmbox, Banshee, etc. didn’t work with either of the open media players I owned, even though the integration point for one of them was the file system. No, the players on Linux support the iPod best because that’s what everyone has, and if I’m going to pay extra for the iPod, aren’t I primarily buying the integrated experience? (On a related note, I’ve never understood why people pay top dollar for Macs and install Linux on them. But I digress..)

Once you taste the fruit of the integrated experience, it’s easy to get sucked in all the way. I eventually tired of all the wasted time and money trying to use an open media player and broke down and bought an iPod. Not too long after, I canceled my Rhapsody subscription. Why? Because I couldn’t use it with my iPod. My overall music experience is diminished (I’m a big believer in the celestial jukebox), but I want one music platform, not two. And the iPod is the one with critical mass. Anything else is swimming upstream.

Are open platforms doomed then? I think that’s, as they say, throwing the baby out with the bath water. The key is to have open interfaces, and the key to having open interfaces that work is cooperation between the vendors that implement them. Perhaps surprisingly, the iPod is a good example of how it can be done. The biggest reason for my iPod purchase: I can hook it directly into my car stereo instead of using an FM transmitter. Apple isn’t getting into the car business—it’s decided to cooperate with other companies to make this work. And you don’t hear too much about people calling their technophile relatives to come glue their cars back together. (And talk about the importance of not crashing..)

The lesson here, it seems to me, is that an ecosystem is never the sum of its parts—it’s either a whole lot more or a whole lot less. Furthermore, it’s the responsibility of the vendors (and community) that make up that ecosystem to ensure the result comes up on the right side of the equation in the final analysis.

After reading through the comments to part 1, let me first point out that our goal is to create a vibrant third party software ecosystem around Linux—you know, like the one Microsoft has built around Windows. No, it’s not about imitating Microsoft. It’s about being competitive. A platform is only as good as the applications that run on it.

Bottom line: Many third parties have built their businesses around proprietary software, and we can’t just ignore them. And “ecosystem” implies decentralized, which I argued in part 1 was a key tenet of open source development anyway, i.e., this should be playing to one of our core strengths. So, if your “solution” is to tell ISVs (independent software vendors) to give us their source code so the distributions can include it because that’s just how we do things, you can safely skip the rest of the post below. You’re simply not going to agree that any of this is a problem.

Ok. Assuming our goal is to create a vibrant third party software ecosystem (and everyone still reading agrees that’s a good goal, right?), we have the following challenges.

First of all, the distribution vendors are hugely invested in their existing package systems, and for the most part, those package systems work extremely well. As I said in part 1, “if [an application] is in your distro of choice, you’re only an apt-get or a yum install away from running it.” (I’m saying it again here because some of the commenters apparently didn’t see that. The tricky bit is “in your distro of choice”, which by definition is not the case with third party software.)

Furthermore, a variety of highly sophisticated systems management solutions are built above those package systems, such as Red Hat Network and Novell ZENworks. This is a pretty important consideration, because these days, the “software management problem” largely involves managing software across the entire infrastructure, not just a single system.

What problem then? This: ISVs have thus far been reluctant to use the native package systems. Why? THEY’RE hugely invested in “package systems” for other platforms, and every Linux specific thing they have to support costs money. In a world where decisions are ruled by cost vs. benefit, this is a pretty important consideration too.

How do ISVs handle this today? For the most part, they ignore the package systems on Linux and do their own thing. Trouble is, while doing their own thing gives ISVs the flexibility to work cross platform, it ultimately makes their products integrate poorly in the broader systems management context because the package systems know nothing about them.

What is needed is a way to bridge the gap between what the distros provide and what the ISVs want. But how?

First off, we have to understand what ISVs want. The answer is simple: ISVs want to treat Linux as a single platform, which means they want to offer a single package for Linux, much as they do for Windows. So, if one commenter is right that “[t]he problem is that people (including software distributors) believe there’s such [a] thing as ‘Linux’ as a target platform” and that “[i]f you’re distributing software for ‘Linux’ then it won’t be simple to install it, ever”, well, then Linux is destined to suffer the fate of UNIX. I’m not ready to give up so easily.

Several commenters suggested that what we need is a brand new package system. That’s a non-starter—for one thing, the distros aren’t going to be too keen on replacing something they’re hugely invested in, and if ISVs aren’t going for RPM today, why would they go for something different tomorrow?

No, to find a way forward, we need an evolutionary step from where we are today. From there, perhaps we can do more, but even the first steps can be quite valuable in their own right.

To help find those first steps, we put some of the leading minds in Linux packaging in the same room (including the maintainers of RPM at Red Hat and Novell and the authors/maintainers of APT, yum, alien, and klik) along with ISVs large and small, server and desktop.

The discussion pretty quickly converged on constructing a single API that could be implemented across the various package systems, because APIs make for nice evolutionary steps and can, done right, mask underlying implementation differences.

Question is, what do ISVs need in such an API? Limiting the scope is key here, because providing an API that spans all the functionality of, say, RPM and dpkg is overwhelming to the point of being unworkable, not to mention more work to implement, which in turn makes it less likely to get into the distros so that ISVs can count on it being there, the whole point of this exercise in the first place.

Fortunately, the ISVs don’t really need much. At the most basic level, an installer just needs to be able to query the system to see if it’s LSB compliant, and if it is, what version of the LSB it’s compliant with; and it needs to be able to “register” with the package system, so the package system knows about it, including what files it has installed. And that’s really about it.

Importantly, because we assume an environment that’s LSB compliant, we don’t have to worry about dependencies, because everything is covered by the single LSB dependency, and dependency management is 95% of the package systems right there. We still need minimal dependency support—components can extend the LSB, and applications can depend on those other components being installed—but we’re talking on the order of a handful of components, not the tens of thousands of components typical package systems have to deal with.

We only had a day, so we obviously don’t have a complete solution yet. For one thing, we barely touched on the issue of uninstallation (should we allow applications to register GUI uninstallers?), and the issue of how applications go about changing the system configuration still needs discussion (in a lot of cases, the assumption of LSB compliance means the installer is going to be well behaved, but there’s undoubtedly corner cases to be explored). And it’s going to take time for the API to be implemented and put into widespread enough use that ISVs can depend on it.

However, everyone in the room did come to consensus pretty quickly that this was an important problem, and that providing a simple API that provides the minimum necessary functionality was the way to go. Perhaps most exciting, the very people whose support are needed to put the API into widespread use were in that room, and active participants in the discussion too.

Everyone is certainly motivated: The distros get more applications, which makes the shared platform more attractive; and ISVs get lower cost, which tilts the cost vs. benefit equation in their favor, making Linux versions more economically attractive and potentially opening new markets.

Because this is just a start, the FSG is launching a Packaging workgroup to continue the discussion we started at the Packaging Summit. If you’re interested in this topic, I’d encourage you to join the mailing list and get involved. There’s still plenty to be done.

Update: One more thing: In addition to making it easier for ISVs to better support Linux by simply extending their existing installation scripts, the API approach really comes into its own when you imagine it implemented in things like Autopackage and InstallAnywhere. Let the market decide!

One of the quotes I’ve had queued up for a long time is this one from Jon Udell:

I have a confession to make. Sometimes, when I’m trying out an unfamiliar open source component, I cheat. Even if the software I’m working on will deploy to Linux, I’ll sometimes develop it on Windows first. Why? Because on Windows, an open source component is likely to come with an installer that just works.

He’s right. Unless an application is included with your Linux distribution of choice, installing that application on Linux is a nightmare compared to Windows.

Here’s an example. To install Sun’s Java Studio Creator on Windows, I just click on the .exe on Sun’s web site, which downloads the file and places it on my desktop. I double click the .exe (after, of course, checking it for viruses) and am up and running in a few minutes.

In contrast, on Linux, I click on the .bin, which downloads the file and.. up pops a text editor showing me a /bin/sh script.

Nice. Fortunately, I know what that is, so I save the script to a file on my desktop. I double click the file, and.. up pops a dialog box telling me the file isn’t executable.

Nice. Fortunately, I know what that means, so I drop into a shell and run chmod +x ./creator-2_1-linux-ml.bin. I double click the file again, and there’s a nice graphical installer now.

Finally, it looks like everything is going my way. Halfway through, though, the installation fails, telling me I need to install the RPMs for compat-libstdc++ and compat-libstdc++-dev.

Nice. Assuming I even know what an “RPM” is, I then realize: I’m running Debian, and Debian doesn’t use RPM. Maybe I know about alien, but even if I do, where do I go about getting the compat-libstdc++ and compat-libstdc++-dev RPMs?

At this point, I’ll probably hit Google—that is, if I haven’t already thrown up my hands in disgust and gone back to Windows. After a bit of Googling, I find this page, which tells me on Debian what I really need is libstdc++2.10-glibc2.2 and libstdc++2.10-dev. Of course! I should have known that. (Note: I’m being sarcastic.)

After installing those two packages, I restart the installer (which, thankfully, knows how to deal with the fact that it’s already half installed, but that won’t always be true). The installation finishes this time, and the installer kindly offers to start the program for me. However, after poking around a bit and exiting back to the desktop, I don’t see a menu entry or a desktop icon, so I’m not sure how I’m going to find it again (and I hope I don’t have to explain why cd’ing to ~/sun/Creator2_1/bin is not an answer).

Anyone who has ever installed software on Linux is familiar with this song and dance. If it’s in your distro of choice, you’re only an apt-get or a yum install away from running it. But if not, you’d better know what you’re doing, have a lot of patience, and understand how to construct effective Google search terms. (And, no, moving everything into the distribution is not a very good option. Remember that one of the key tenets of open source is decentralization, so if the only solution is to centralize everything, there’s something fundamentally wrong with this picture.)

Oh, well. At least I didn’t have to check for viruses!

Fortunately, some of the problems I experienced are bugs. The above was done on a pre-release Debian etch system over the summer, so it’s likely the problems have been fixed. I repeated the experiment on an Ubuntu edgy system, and it didn’t open the text editor, nor did it complain about an incompatible C++ environment. However, there were still no menu entries or desktop icons, and there was an additional problem too in that when I double clicked the file, it opened it in CrossOver Office, which I also have installed. Regardless, even if it works better on some distros than others, there’s still no usable solution until ISVs and end users alike can depend on things consistently working regardless of the distro being used.

Again, fortunately, we have solutions to parts of the problem already. The LSB abstracts away the differences between the runtime environments of the various distros, so the Java Studio Creator installer could have simply said “you must install the LSB environment” rather than trying to deal with the hundreds of little variations in both the environments and the package namespaces that provide them (e.g., compat-libstdc++ vs. libstdc++2.10-glibc2.2). Better yet, on distros that provide the LSB environment in their default configuration, the installer doesn’t have to do anything. And Project Portland promises to give us a consistent interface for creating menu entries, desktop icons and such things.

However, far too few applications take advantage of the LSB today (though that’s changing), and Project Portland isn’t in any of the distros yet (though we’re looking at bundling its primary deliverable, xdg-utils, with the LSB 3.2 SDK to work around that). Finally, even though the LSB provides ISVs with a consistent way to create an LSB compliant executable, there’s no consistent way to deliver an LSB compliant application that’s easy to install and that integrates well with the distribution’s package system. Yes, the LSB includes RPM today, but for a variety of reasons, ISVs don’t want to use RPM, and as already mentioned, not all distributions support RPM natively.

Fortunately, once again, this isn’t just a rant. The LSB tackled these very issues at the LSB face to face and Packaging Summit last week in Berlin, Germany, and we think we have a way forward that’s acceptable to all involved: Linux distribution vendors who already have well established package systems and systems management tools built around them; ISVs who need to support multiple platforms and so don’t want to support the Linux specific RPM format or who otherwise want more control over the installation experience; and end users who want to use the software management facilities their distributions provide, whether that’s RPM or something higher level like APT and yum. More in part 2…

This is a big deal in and of itself, of course, because it will make applications based on ODF, AJAX, and other emerging open standards more accessible to visually impaired users. But it’s also an important milestone for the FSG, so I thought I should say a few words about that here from my own little pulpit.

The first clue is that iAccessible2 is an accessibility API for Windows. Windows? Yes, the FSG is primarily a Linux organization (our main project is, of course, the Linux Standard Base), so why on earth are we now involved in building APIs for Windows?

First of all, we’ll be moving this project forward as an extension to our existing accessibility projects—which, given the way the FSG is structured, flow into the LSB as they are adopted by the Linux distributions (who, as I’ve already mentioned here, are active participants in the LSB project, so we’re not just hoping the distros adopt our work, we’re proactively trying to drive consensus on these key issues). So, this definitely impacts our Linux work by bringing accessibility features to Linux that not only equal those of Windows but exceed them (iAccessible2 itself came about because Microsoft Active Accessibility has notable shortcomings).

But it’s more than that. Another way to look at this is that the LSB APIs will be taking on a distinctly more cross platform flavor, particularly in the “around the edges” cases such as this. Again, that begs the question: Why is cross platform support important for a Linux organization? That one’s simple. APIs are for application developers. Application developers target multiple platforms. The less Linux specific work an application developer has to do to support Linux, the more cost effective a Linux version will be, the more likely it is to get done. It’s all about more Linux apps.

There’s another dimension to it as well. There are a certain class of features that the vast majority of developers don’t think much about because that vast majority doesn’t need them. There’s no itch to scratch, as it were. Accessibility is a great example of such a feature: I have no idea what it’s like to be blind, nor do most people. If we can provide a set of open, cross platform APIs that allow application developers to easily add such features to their products, more standards based products will support them.

As Massachusetts and ODF brings into sharp focus, these “around the edges” cases developers don’t think much about can turn out to be very important indeed.