In The Cathedral and the Bazaar, Eric Raymond analyzes exactly how it's possible for free software to compete against professionally developed software. I remember how reading it made me completely re-evaluate how cooperative relationships work with humans. It's made me think differently about my relationships with people, as an employee, a supervisor, and as a human being.
Ostensibly, it's about the growth of the Open Source software movement and, more importantly, how it has been effective at developing world-class software. Open Source is a software development method wherein almost all the software is free of charge and in which the source code is publicly available, so anyone can rewrite it. Furthermore, the vast majority of the programming is done on a volunteer basis, for no money.
This sounds like a recipe for terrible software. But, much of the internet runs on just such technologies. In fact, this weblog is almost entirely dependent on at least three such technologies: the Linux operating system, the Perl programming language, and the Apache web server software. All of it open source, highly powerful, and reliable as a stone. Developed entirely by by volunteers working mostly for free. How is that possible?
The essential idea is that software development can follow two models: the traditional European Cathedral, which represents a very hierarchical, centered, elaborately planned way for getting things done. The Bazaar, on the other hand, is a somewhat spontaneous, very loosely ordered and informal way of selling goods and services. Stuff gets done mostly by cooperative relationships between people who have mutual goals.
I won't try to explain much further; if you doubt it can work, I merely point again to this weblog, to the continuing challenge Linux represents to Microsoft, and to the fact that about half the internet these days is running on this kind of software.
What I've noticed is that many of the dynamics Raymond talks about are also reflected in the weblog community. Most particularly, the spontaneous peer-review process, which tends to keep webloggers honest. References can be checked at the speed of light, and webloggers who refuse to acknowledge their errors also gain a reputation for being unreliable or dishonest. Collaborative journalistic projects can also uncover things that the mainstream media, with its full-time paid reporters, often miss, or fact-check them on items the reporters themselves only vaguely understood when they wrote their stories.
Weblogs have made me increasingly impatient with traditional media. Whenever I see an assertion I question, I want to see a link to the evidence. If I don't have it, I trust the statement less. I suspect I'm not the only blog-reader and writer who feels this way.
My buddy Lysander recently pointed me to a similar "open source style" phenomenon: the Wiki. It's a funny name for an exciting concept: reference works developed by free collaboration among mostly-voluntary contributors.
I think Lysander makes a mistake in suggesting that Wikis would ever replace weblogs. That, to me, is like saying that compact discs will replace radios. Nevertheless, the concept is very exciting. I suspect that it will represent the same sort of challenge to traditional reference book publishers that weblogs represent to mainstream journalists, and that Open Source software represents to traditional software publishers.
Which is not to say that any of the new technologies will replace the old. But they do represent exciting new phenomena that are going to change how the old school folks do their jobs.
I am a full-time software engineer and I hold affiliate faculty positions at two universities where I teach programming and software engineering part-time. In the introductory software engineering course that I teach to graduate students we spend a considerable amount of time comparing open-source methodologies to closed-source methodologies. One of the things that I force my students to do is to evaluate both the strengths of open-source development as well as its weaknesses. I agree with you 100% that the blogging movement shows significant parallels to the open-source movement. I see this in its strengths and also in its weaknesses. You have pointed out some of the strengths, so I would like to expand on your statements by exploring some of the weaknesses. This is not to say that I necessarily favor closed source software over open source software (OSS) any more than I necessarily favor traditional media over Weblogs. I’m simply taking the discourse in that direction.
One of the things that you mention is the spontaneous peer-review process of both open-source software and blogging. Many consider this to be a strength. Few explore it well enough to recognize the inherent weakness that it can also represent. At my job I write software that must exhibit a high level of security. There are many people out there who feel that the peer-review process leads to more secure software. This is logically fallacious. There is very secure OSS but there but not all OSS software is secure. The problem is that not every programmer involved in the OSS peer-review process has any real knowledge of how to write secure software. What good is a peer review if your peers are just as ignorant as you are? As evidence, look at some of the recent bugs in sendmail and OpenSSL that have been there, literally, for years to only now be discovered. The same is true of bloggers. Yes, the open nature of blogging allows for instant peer review, but how frequently is the peer review done by an actual “peer" of the blogger who is every bit as ignorant as the blogger him/herself? I have seen you, Dean, addresses on your site bloggers who are your detractors but that hasn’t changed their behavior in the slightest. Much of the blogosphere is what my late father would have referred to as “preaching to the choir."
In traditional media, however, the review process is generally institutional. Stories are often checked by fact checkers prior to publication. This is similar to the code review processes that are in place in most software development shops. The quality of the product ends up being directly proportional to the quality of the review process. With some products, such as those by Microsoft and (in one notable case) the New York Times, there are problems. When these problems occur, however, there is accountability both by the system and within the system. The institutional quality processes can then be adapted by the institution to address the flaws (Microsoft’s Trustworthy Computing and the NYT’s removal of Blair – we’ll see how they both work) in hopes of retaining customers. Subsequently, the product being produced by the institution has ample opportunity to be successful.
Another area where both the OSS movement and the blogosphere suffer from a similar weakness is in the success ratio. It is very easy to look at successful OSS projects like Linux, Apache, and Perl and tout the successes the same way that you can look at some of the successful Weblogs and come to similar conclusions. How many OSS projects fail, however? Take a visit to FreshMeat or SourceForge someday and see how many projects never get out of the conception phase. There is never any risk involved in starting an OSS project just like there is never any risk in starting a Weblog. There are many, many Weblogs out there where the blogger posts day after to day and never gets a single comment. They do this because there is no risk. Traditional media and closed source software development don’t exhibit this quality. If a magazine, for example, stays in print then you can be assured that it has value to a number of people significant enough to keep it in print. The same can be said for any of the other traditional news services. When I go to the bookstore or the software store I have to look through far less chaff to get to the wheat than I do when perusing OSS and Weblogs.
The comparisons do go on, but I think that I’ve taken more space on your Weblog than a commenter has a right to, so I won’t go further. I think that you have drawn a valuable comparison and one that I hope that others continue to explore. My final view is that both Weblogs/OSS and traditional media/packaged software have their place. I’m not going to start getting my information exclusively from Weblogs any more than I have stopped purchasing packaged software. Like you, however, the vast number of “free" (“free" as in speech not “free" as in beer) has caused me to start expecting more from the traditional suppliers.
One of the more exiting wikis out there is Wikipedia. It’s an honest to the gods encyclopedia created entirely by anyone at all. It’s proved to be a rather useful tool...
Well, I must say Jerry that, whether "logically fallacious" or not, most analysts have concluded that Linux and FreeBSD are, in fact, more secure than Windows. This tends to fly in the face of the notion that commercial products will be more secure. Factually speaking, we already know that's not true.
Indeed, I worked for a while for a Linux company, and did some research on computer insurance. There are insurers now who offer lower premiums to companies using Linux for server products because it's been proven more reliable and more secure.
Having also worked as a project manager for a commercial software developer, I can tell you point blank: years and years go by without known bugs being fixed in commercial software. Years and years go by before the bugs are even acknowledged, at times.
Yes, it's true, open source projects fail quite frequently. Then again, I've been involved in, and know of many other, commercial software ventures that also failed.
So, while I do not disagree with you--I think there is definitely a place for closed-source commercial software--I'm not at all convinced that the problems you're identifying for Open Source are the actual problems. I think that, more broadly, the inherent limitation is that you have to get enough people to want to work on something before it will go anywhere. Necessary work may never get done otherwise.
Also, the motivation to fix bugs is only there when there's sufficient interest in the product to get it done.
I think that's the real weakness.
Otherwise, though, on the whole we do agree.
Jerry, I generally agree with the substance of your critiques of open source. What I don't see is how these are "weaknesses". Maybe they're overhyped "strengths".
For example:
There are many people out there who feel that the peer-review process leads to more secure software. This is logically fallacious.
Leaving aside the fact that it isn't logically fallacious (the statement "peer review leads to higher security" is a meaningful statement, whether wrong or not), the fact that peer review does not grant some benefit does not, in itself, constitute a weakness. You do compare the internal code review to peer review, and it's possible to say that "formal code review is better than informal peer review", but that's a statement that requires evidence. And given that closed source has an even worse record for security than open source does, I'd say that's a theory that has yet to be proved.
Probably it's best to say that there is no magic bullet in security, whether in formal review or informal.
As for the lack of risk inherent in open source: I utterly fail to see the down side. Are you saying that we should discourage people from thinking about software? It's not like anyone's being forced to use some of that software. The good stuff tends to stick around, which is the important part.
OSS tends to be best for infrastructure, things like DNS, email, Webservers, language development. In desktop applications, the accountability and financial motivation to be responsive to the bugs that users care about most, can be seen as a real benefit.
I think this is part of why OSS hasn't done too great on applications. When you've got 1000 users all screaming/emailing "I paid $34.95 for this thing, and this feature is broken!" you'll get on it NOW. When it's OSS and it's a feature that isn't the pride and joy of some developer, the bug can languish for far too long.
Wiki's are indeed good for things like encyclopedia's and documentation/topical sites, but I don't see them as filling the same niche as weblogs. Wiki's generally allow anyone to alter/post to a page, which takes away the "this is MY soapbox!" beauty of blogs, and opens them to increased damage from trolls; the trolling isn't even buried in the comments anymore!!
Dean, Jeff:
You misunderstand my point about security and the peer review process. You are both correct that many open source software products have proven to be secure. I said this in my initial comments. I am also not denying the existence of commercial software products that aren't secure. Again, I made this point in my initial comments. What I am saying is that peer review by itself isn't enough. If you look at how products such as Apache, OpenSSL, the Linux kernel, etc. are produced you will see that there is some level of formal process involved. Do you think that Linus simply lets anyone insert code into the official kernal? Of course not. There is a formal process that must be followed. At the end of the day, any code that Linus hasn't personally approved doesn't get in. Peer review is an integral part of these processes but it only one part of a larger, formal process. The simple fact that one or more peers reviews your code isn't enough. If it was, then Windows would be the most secure operating system in the world because Microsoft has a HUGH quality assurance staff. They have a miserable track record for security because security was never an integral part of their process. I agree with you that peer review in a process is a strength of that process. The reason that I mentioned it as a potential weakness is because there are numerous OSS zealots out there who believe that peer review by itself makes one product intrinsically more secure than another. This simply isn't true. It takes peer review plus a whole lot more.
To carry the analogy back to blogging, which I believe was the initial point of the discussion, just because I have someone proofread my works before I post them does not mean that there will be no errors. If no one bother to check the facts or if the person/people doing the proofreading are ignorant of spelling/grammar/the topic/etc. then errors will occur. In order to be error-free there must be some level of formality. Take this blog, for instance. You yourself, Dean, will not simply allow anyone to post. They must submit their posts to you for review. You specifically have asked for no sports stories unless they have sort of social relevance. This is a formal requirement that you will check prior to posting. The other day you hosted a discussion on religion in which you placed restrictions on 1) who could post and 2) what could be said. This was a formal process. You did not leave the discussion entirely up to peer review because you felt that this additional level of formal requirement and review was essential to maintaining quality.
Jeff:
You asked: "Are you saying that we should discourage people from thinking about software?" Of course not. All I'm saying is that there are a lot of blogs out there that no one reads and that have little or no real relveance simply because there is no risk involved. It's like when the KKK schedules a rally. In order to have truly free speech we must allow every idiot with an opinion speak, even when their beliefs are morally abhorent. The strength of free speach is that everyone can be heard. The weakness is that some of what is said is reprehensible. OSS is similar. The strength is that consortiums of volunteers can create great products like Apache. The weakness is that people can also create worms and viruses.
Wikis are pretty good tools for managing software development. The company I work for uses it in an XP (Extreme Programming) development process. As far as being useful for much else, especially if open to the public, I have serious doubts.
Well Jerry (Jerry #1, not Kindall) you make a good point. I think the larger point I was trying to make was that "security" really isn't your best example. Like I say, overall I'm in agreement with most of what you say.
In fact I have an odd theory I'm brewing: that at some point, all intellectual property will become public domain whether we mean it to or not, and that the success of Linux is just a reflection of that. Perhaps that will inspire another article on the subject.
Anyway, I'm mostly saying you're right, man. ;-)
You're right, Dean, I probably should have used a better example than security. It's interesting to me that when you mention security, many people go directly for Microsoft and then draw negative conclusions regarding closed-source software. Oracle is every bit as closed source as Microsoft yet they market their products as "unbreakable" and no one questions it. The flip side is that, quite often, even the slightest criticism of anything open source can send some people into a defensive frenzy (just read Slashdot [http://slashdot.org] once and you'll see what I'm talking about.)
Going back to the discussion of parallels between blogging and OSS, many corporations are now incorporating OSS principles into their own internal processes. You mentioned in your original post that you now demand more from your traditional media outlets now as a result of the rise of blogging. I'm curious, in what ways to you see traditional media changing so as to combat the threat posed by blogging? What other changes do you think we will see in the near future?
In order to be error-free there must be some level of formality.
I'm not sure that's even true. Formality might be a good idea, but can responsiveness be an effective substitute? It's something to think about.
There's also the question of whether a lower level of formality, combined with peer review, isn't better at ensuring quality than a higher level of formality alone. If this turns out to be the case (which I think it is), then open processes such as open source and blogging have an advantage over closed processes such as closed source and mainstream journalism.
Oracle is every bit as closed source as Microsoft yet they market their products as "unbreakable" and no one questions it.
That's not quite true; I remember seeing a flurry of security advisories on Oracle soon after that campaign started, initiated by hackers who resented the "unbreakable" claim and decided to try and disprove it. I don't think you hear much today because it's old.
In fact I have an odd theory I'm brewing: that at some point, all intellectual property will become public domain whether we mean it to or not...
Are you being ironic on purpose here? None of the versions of "intellectual property" are immune from the public domain by law. Copyrights and patents expire, and trade secrets and trademarks can be invalidated.
Of course, the implication of the words "intellectual property" is the notion that one can own ideas forever, a notion Disney is trying to perpetrate by indefinite extension of the terms of copyright. I can't think of many ideas that are more dangerous to the idea of a free society as the idea of "intellectual property" is, if taken to its logical conclusion.
I actually do agree that some level of formal review is necessary. I find that most of the most successful Open Source projects have one. They just don't like to acknowledge it.
Linus is the best example for Linux. If you look at Mozilla, there are acknowledged project leaders, and someone always makes the final call on whether something will be included or not.
Indeed, I'd go so far as to say that unless there are some strong leaders in an Open Source project, who the other volunteers are willing to follow, it's not likely to succeed. Unless it's a fairly minor project, like tweaking something that's already largely finished, or a project that's small in scope to begin with.
Once again, I note that Raymond pretty much documents this--how it works, how it happens. Call it informal formality if you like.
The question with Open Source to me is not whether there will be formal review. There will be, in most successful projects. The real question is: will a project draw enough interest, and enough strong leaders, to be a success?
Closed source works better with projects that bore the crap out of most developers, to be blunt. "Barbie's Super Funhouse!" is never going to be an Open Source project. (Well, never say "never," but you know what I mean.)
As for intellectual property: well Jeff, I do believe in intellectual property, and I think the lack of it would be as dangerous as too much of it. I would agree, however, that we've gone much too far with it, and Disney's one of the worst offenders--all the more ironic because so much of their work is based on things in the public domain (Cinderella, Snow White, Beauty & The Best, the Jungle Book, etc.).
BUT, at least in computer software: I should really write an essay on this, but I notice that Open Source initiatives seem to spring up to replace certain things that were innovations a couple of decades ago, but are really just commodities now. Operating System kernals, in my view, SHOULD be public domain--there's just no reason that anyone should be "protected" from the "innovation" of such an obvious idea. GUIs are going that way with KDE and GNOME. Checkbook programs, email programs--these are not "innovative" products anymore.
The useful arts and sciences were supposed to be protected in such a way that inventors and creators could get reasonable compensation for a decade or two for their innovations--then they become public domain. Companies like Microsoft are trying to "protect" fairly non-innovative products, and trying to do it, in effect, forever. Yet the world is finding a way, slowly, to work around that.
That's kind of what I mean.
As for how traditional media will have to answer to blogs: frankly, they are already doing it. They're also showing an irrational fear of it. I should write an essay on the subject, but, basically, I believe the big institutions are responding. Slowly, but they're responding.
A few are also whistling past the graveyard, but I'll leave that observation for another day, too. ;-)
Software patents are the main thing slowing down tech innovation these days in my opinion. It is increasingly hard to develope anything without stepping on someone's patent(s). This drives out all but the largest corporate players, who can afford to cross-license, and who also are not going to be as innovative.