So given all this, why is the article so flawed? And in its basic premise...
Early on it becomes fairly clear the article is written with an agenda:
How, if at all, should governments use [open source software (OSS)]? One important theoretical insight starts from the observation that [OSS is] ...imperfect [and] has distinct areas of advantage and disadvantage (von Engelhardt 2008). This implies that large modern economies will usually require a mix of both [OSS and closed source software (CSS)].
The article goes on:
[Engelhardt and Maurer] point out that the existence of CSS code increases OSS output and vice versa. To see why, consider an all-OSS world in which each company offers consumers exactly the same shared code as every other company. By definition no company can then compete by writing more OSS code than its rivals. This lack of competition suppresses code production for the same reason that cartels suppress output.
Quotes from " Open vs closed source software: The quest for balance" linked above, emphasis added
So, as it is a very good place to start, I'll start at the beginning; with the definition of the economic concept referred to, a cartel.
A cartel in economic theory is generally seen to occur at a particular point in a range of market types. This range stretches from perfect competition to monopoly. A monopoly market is the condition which the game of the same name defines as victory, that is the absence of competition. Perfect competition at the other end of the scale is a market where all parties know all things about the goods sold in the market (known as perfect knowledge) and it is easy to set up in business. As is clear in the terminology used, perfect competition is seen to be good and monopolies bad.
Economists see a sliding scale between monopoly and perfect competition, and degrees along the way. It is generally accepted that a near or effective monopoly is as bad as a monopoly; a near monopoly can be seen to exist in a market where a single company controls more than two thirds of that market. Below a monopoly in economic badness lies an oligopoly, where a small number of large companies control the majority of a market. It is at this point in the scale that cartels are seen to form. A cartel is where a number of firms in the oligopoly get together and conspire to fix pricing, using their power to inhibit competition, to create an effective monopoly.
One of the consequences of suppressing competition in this way is that as there is less need to compete, not only are prices maintained artificially high but the members of the cartel have no need to try and compete in other ways, through improved production methods, higher quality output, higher rates of output etc.. This is the cartel effect the bad, bad pure open source world will have. No incentive to compete, so reduced output (where output is defined as code).
It is important at this point to note that the basic models and concepts of mainstream economics are very old, and that these models essentially assume a physical product, known as a "good", is being produced from raw materials and being sold into the market of whatever type. It is also useful to understand that there are theoretic possible consequences of a cartel, and the one used in the article does not take into account the search for profits i.e. that while operating a pricing cartel, companies will still seek to improve their individual profit position and thus continue to evolve in areas centred on reduced production costs and increased output.
Phew. Right, so reasons why monopolies, oligopolies and cartels are bad while competition is good: one basic concept is that while an accounting profit is considered ok an economic profit is considered bad. So for your company to return a profit in its accounts, no matter how large, is absolutely fine; but for the resources used in your company to produce more than a very small (marginal) amount of return than they would if used in their next best option, e.g. if your staff worked for another company which would make a lot less money than yours? This is bad. A second and pertinent concept is that because you have little incentive to make your products better or in a cheaper way, resources are not being utilised in the most efficient manner i.e. not contributing their maximum value to the economy.
So back to the premise: that is that if all software firms were selling the same open source codebase, output of code would fall and this would be "a bad thing". Ok, first off. Software companies that sell software don't do open source. You don't sell software if you're in the open source market, you sell services. Moreover you do so in something approaching a perfectly competitive market.
So why would I give you money to use your Linux when I can get this other Linux, based on the same codebase for free? Oh, you'll ensure patches are timely and applied in a tested manner before being provided to me? You'll provide a ridiculously good SLA if I have problems with my Linux desktops and servers? You have many of the best coders and therefore will be able to fix any issues I encounter in very short timeframes? In which case, your differentiations, albeit minor, are important to me, the buyer in this near perfect market, so I'll give you my money even though there is a free version available.
Then to the second and defined negative aspect: that code production would fall and this would be a bad thing. I will with great willpower refrain from swearing at this point and try to calmly point out the fallacy of applying pure manufacturing thinking to software code. Quantity is utterly irrelevant in software code. Quality is everything. This is then back to the conversation outlined above. The quality differentiator is what I will pay for.
You can make an office productivity tool a little faster, a little more standards-compliant, a little less prone to arbitrarily adding and removing spaces? And you can do it for me now, although other people will get the benefit later? Why, that sounds good to me. I suppose others will be paying to get a shinier interface and some new functionality, which I will gain from later? That's nice. Why, that sounds suspiciously like a win-win from that game theory stuff doesn't it...
So back to the economics: no, a completely open source software world would not act as a cartel, with a negative consequence of reduced code output. Sure the amount of code written would be less, but not in a negative way; it would be a result of the increased efficiency of the perfectly competitive market software would become.
So open source software would be, more or less, a perfectly competitive market? Well, sort of.
You see, actually there's another underlying premise in that article that is flawed, a premise the article doesn't acknowledge. Quite simply you can't treat software as if it is a normal good - it doesn't exhibit any of the features of one. But I'll leave that for another day...