Thoughts for startups (no. 7)

"He never created a finished product. Finished products are for decadent minds. His was an evolving mechanism..."

Isaac Asimov, 1964

Related to the over-hyped start-up concept of the Minimum Viable Product (MVP), on which more later, but also related to the Japanese manufacturing concept of kaizen, or continuous improvement, which in turn draws on martial art philosophies (or the power of practice).

Your tech product isn't finished, and isn't going to be. Ever. You might stop developing it and people might carry on using it, and preferably paying for it; but it's not finished. There will always be some improvement, however slight, that can be made to some aspect of it. But more importantly you won't really start to understand what your product does and can do until you make it. Great that you came up with an idea, tweaked it around a bit before realising it. But until you actually have it in front of you, there will be features, UI elements, loads of better stuff to do with it that you couldn't imagine before; and that's nothing to what happens when you put it in front of someone else.

Another lesson to draw is that those changes should be evolutionary as much as possible, at least from the end user perspective. So it's great that you can get a 10% improvement in UI responsiveness by redeveloping the entire back end - but make that change without disrupting the end user. Thought of 6 great ways you could improve the UI? Make them one at a time. Because each is like a product in its own right; seeing the change in use opens up a whole set of possibilities about how the product could be improved.

So make your product work and then improve it; don't be a decadent mind.

Stopping PDF encryption hurting your productivity

Here's how to get around copy and paste issues with text from a protected PDF. No programming talent required (which is good, is not a talent I have...). Sharing this knowledge may put me in contravention of some anti-consumer law somewhere, I don't know, it's a weird world.

Scenario: my partner was working on updating a report, reusing and updating material from an earlier report produced by a colleague. This earlier report was only available as a PDF and it turned out this PDF had protection enabled to prevent copy and paste. Now, I could either let her manually type out the material (amounting to quite a few pages of text) or I could find a way around it.

Simple it turns out to be...


In summary: convert the PDF to an image and feed it into an OCR engine. With care on the PDF to image conversion, and presuming a decent font/typography in the original PDF, the resulting OCR output should be near 100%.

There's a bunch of ways this can be achieved, what follows is the process I used (which should be more or less applicable anywhere, but was done with Linux on my partners XPS15). You should be able to more or less cut and paste the following commands to achieve a similar outcome on your own machine, once you've got the right software installed (tesseract and ghostscript).

First convert the PDF to an image (the TIFF format was used here as the PDF was multipage):

gs -o /path/to/outputfile.tif -sDEVICE=tiffgray -r720x720 -g6120x7920 -sCompression=lzw /path/to/inputfile.pdf

Then run OCR software to convert the image to txt:



tesseract /path/to/outputfile.tif /path/to/ocroutputfile

Done. One text file, simply and cleanly formatted, containing all the text from the protected PDF. All DRM can be defeated this easily - if you can see it, it can be copied.

NB If in another scenario if you needed the final output to be more nearly a copy of the original PDF you'd need to involve OCR layout functionality - but much of that's doable with Tesseract by changing the output format to hOCR. And then, if you really wanted to, you could use hocr2pdf to recreate the PDF...

Thoughts for startups (no. 1)

Thomas Paine, British-born philosopher, French revolutionary and US founding father wrote any number of useful sayings, but I think this one is particularly apt for anyone trying to start a business that's on the leading edge:

"Perhaps the sentiments contained in the following pages, are not YET sufficiently fashionable to procure them general favour; a long habit of not thinking a thing WRONG, gives it a superficial appearance of being RIGHT, and raises at first a formidable outcry in defense of custom. But the tumult soon subsides. Time makes more converts than reason." 

Commonsense, 1776

So for our startup, TwoTen, getting people to see that "family protection" when it comes to Internet filtering makes no sense, because age-appropriate is utterly not the same as "not adult". Not so much a formidable outcry, but definitely some denial - though the same people would object to film certificates being changed to simply "adult" and or not.

Thanks to Alex Kablanian for prompting this post.

For fsck's sake add swap...

If you're like most people undertaking an attempt to repair a really large volume (in this case 36TB), you run  fsck (or specifically e2fsck) without a second thought. Some days later (oh yes) you realise it's eating all of the RAM and making big inroads into swap. A quick check via your favourite search engine gets you the information that:

(i) it's going to eat more and when it runs out of RAM and swap fsck will crash and you'll have to start over, and;

(ii) you should have had more swap or used a scratch disk by creating an e2fsck conf file and configuring one as necessary.

Oh for the gift of foresight...

But an  option I ran into elsewhere is to simply add more swap on the fly. As this worked neatly and without any fuss, I thought I'd document it here for others with a similar issue.

As root or using sudo, do:

> fallocate -l 8g /mnt/fsck.swap

> mkswap /mnt/fsck.swap

> swapon /mnt/fsck.swap

where what you call it is up to you, where you put it is up to you (ideally on a volume not doing much else) and the size is dictated by the available space and set with the "-l 8g" line i.e. I made an 8GB swap file in the above example.

Now I've doubled swap there's a chance this fsck could finish. Sometime in the not too distant future :)

Copyright, content industries and "that's not fair"

In the sense of "and I'm going to tell my mummy government". Robert Heinlein penned the following quotation, as part of his first ever published story:

"There has grown up in the minds of certain groups in this country the notion that because a man or corporation has made a profit out of the public for a number of years, the government and the courts are charged with the duty of guaranteeing such profit in the future, even in the face of changing circumstances and contrary to public interest. This strange doctrine is not supported by statute or common law. Neither individuals nor corporations have any right to come into court and ask that the clock of history be stopped, or turned back."

Robert A Heinlein, Life-Line, 1939

The copyright reform battle is enshrined in this thinking. On the one side we have people reasonably pointing out that the move to digital from analogue completely changes how content can and is being reproduced, reducing some formerly expensive processes to zero cost; on the other we have the rights holders shouting "no fair" and demanding the law is made tighter, rights extended longer, and that everyone else must be made to expend their resources and all unreasonable efforts to protect their lucrative business model.

But their business model, historically lucrative and rather young, is now outdated. Setting a book for a PDF can be carried out using cost-free software in hours and reproduction is free; people are creating popular music in their bedrooms and distributing it at no cost, and with the rise of Aframe (and presumably soon a Samwers clone ;) ) film production costs are following the huge drop in distribution and reproduction that digital and the Internet had already brought.

BT and the insanity of over-large companies

Here's a thing. Rural broadband. It's bad. It's bad partly because it's a long way along copper lines to the houses from the exchange (although I know people next door to their rural exchange who get suspiciously low speeds). This major element of the problem could be quickly and easily resolved by moving to fibre to the cabinet (FTTC). The cabinet is typically only a few hundred yards from the houses it serves, and fibre optic connections can carry signals over great distances with minimal loss of performance.

But BT, who rural residents are dependent on for infrastructure, don't see rural exchanges as economic to upgrade. The same BT, but in a different department, are tearing their hair out over the cost of maintaining copper infrastructure for these rural exchanges, as copper theft becomes increasingly common and rural locations are relatively low risk targets. Replacing a stolen length of copper serving an entire village is expensive in so many ways; the cable is expensive, the civil engineering that can be required is expensive, the telephony engineers to reconnect all the connections is expensive, the customer support dealing with complaints for the week or so the village is disconnected is expensive; and then there's the societal cost where elderly residents are cut off from the outside world for that long, potentially putting lives at risk.

Is it really so expensive to move rural exchanges to FTTC? That much more expensive than the costs of the copper cable thefts? More expensive than lives? Is it really uneconomic? No, probably not. But the upgrade department budget will be so very, very distant from the repair department budget; there's almost certainly no one with oversight of the two cost centres at a detail level necessary to see the opportunity. So rural residents get hit from both sides, having the lowest level of service while paying the same amount for their connection, while BT pay out for patching symptoms of a problem they could just rid themselves of.

Big companies don't have to be run in this short-sighted, disconnected way. But so many are...

Installing Linux on Dell E5420

As is customary, to help others following along the path, here's some instructions for installing Linux (Kubuntu 11.04 but I'd guess any modern distro would be a similar experience) on a 2011 Dell E5420. In case the particular specification has an impact on the steps needed I've appended it to the bottom of this post.

These steps assume you have a prepared bootable install medium. I use a USB key, but CD/DVD would be similar.

Step 1 (optional): Ensure laptop is connected to mains and network (best to use wired).

Step 2: Press F12 during BIOS spash screen to reach boot menu, and select your boot media.

Step 3: When install option comes up, select it. Accept all defaults and complete user information as requested/desired.

Step 4: When instructed, reboot laptop. If using USB and an SSD this is in less than 5 minutes.

Step 5: Everything works. Enjoy :)

Sadly this traditional post is basically pointless, because it was that easy. All but one of my nieces could have done it (the youngest is only a month old, give her a chance...). In less time than it takes to install a Windows service pack you can take a blank laptop and transform it into complete working environment with all the core applications (office suite, etc.) already installed. No technical expertise required.

PS Rather nice, solidly built, nice to use laptop, in case you're interested. 3 mouse buttons on the Trackstick too. Battery lasted 7 hours out of the box, including all the install work :)

Spec (yes, no Microsoft Windows was supplied nor paid for):

Latitude E5420
Processor : One Intel Core i3-2310M (2.10GHz, 3MB cache, Dual Core)
Display : 14.0in UltraSharp HD+ (1600X900) AntiGlare LED-backlit with Premium Panel Guarantee
Palmrest : Dual Point Touchpad and Trackstick
Camera : Integrated Full HD Camera with Microphone
Memory : 2GB (1x2GB) 1333MHz DDR3 Dual Channel
Hard Drive : 128GB Mobility Solid State Drive
Optical Drive : DVD +/-RW Drive
Battery : Primary 9-cell 87W/HR, 3 Year Warranty
Wireless : Dell 375 Bluetooth Card
Wireless Card : EMEA Intel Centrino Advanced-N 6205 (802.11 a/b/g/n) Half Mini Card
Keyboard : Internal UK/Irish Qwerty Backlit Dual Pointing Keyboard (antimicrobial)
OS Media : Not Included

When not finding something is an error

Or to be exact is a #VALUE! - a passing tip for those using spreadsheet functions that return an error code instead of a usable outcome.

I was using the FIND function to look for a character in a text string. Specifically I was trying to find "/" so that I could rip domain parts from a huge list of URLs. My problem came when some of the URLs were just domains and contained no trailing "/" - at this point my formula

=FIND("/",A1)

returned a #VALUE! code instead of a 0, where FIND's outcome is normally a numeric reference indicating the location of string. A bug, I thought, so let's get it fixed. As I was using LibreOffice Calc and their release cycle is more or less monthly, I figured I could live with the issue for now and get the bug fixed for next month.

A conscientious software user always, I duly posted the bug and a day later I get an email indicating someone was processing it. I open the email and... the bug has been closed, marked NOTABUG. The associated comment said "This is not a bug. It is the expected behavior in ODF 1.2". As the newly completed OpenDocument Format 1.2 had some of the world's top experts developing the formula specification, this seemed, well, odd.

For those who, like me, couldn't understand why 0 isn't a logical outcome for "not found" in the FIND function, I asked someone I know who has some expertise in this sort of thing (Rob Weir of IBM, a spreadsheet developer since 1990ish and the chair of the ODF Technical Committee). Turns out it's that damned legacy thing again...

Back in the early days of spreadsheet development, 20 or so years ago, it was believed that a user wouldn't understand 0 as the outcome of a test that finds nothing, potentially causing subtle errors to cascade through the spreadsheet. So spreadsheet developers adopted the practice of outputting an error as the null result for this type of function. As it is now a well-established practice it was decided ODF should stick to it in the formula specification.

As my FIND function was nested in a larger formula, a error output just caused the whole formula to fail, breaking my automated process. Luckily it turns out there is a workaround using another function called ISERR (ain't it great knowing really clever people?). So if like me you need FIND to return an integer or otherwise testable answer rather than an error, wrap it in a ISERR:

 =IF(ISERR(FIND("my string",<the cell to search>)),0,FIND("my string",<the cell to search))

Sorted. Or rather found...

The public good of software

Or public goods, to be more precise. Every economics student can promptly skip three paragraphs.

The subject of economics likes to classify and simplify things; some of these classifications and simplifications are useful, some not. Several in classical economics are downright dangerous, but I digress. One classification that more or less all economists would agree on is the concept of the public good. A public good is one that is collectively paid for because it makes no sense to pay for it any other way; the majority of services provided by the public sector are, or should be, public goods per se.

A classical example of a public good is streetlights. Streetlights have been around since 800AD or so; the first streetlights as we recognise them today were gas powered and switched on in Pall Mall in 1807; the first streets to be lit by electric light were in 1881 (Cleveland, Ohio and Newcastle-upon-Tyne appear to share this honour). All these streetlights share one distinctive feature; it is effectively impossible to have streetlights and charge individually for benefiting from them.

Early streetlights were either installed for public benefit or as a marketing ploy; modern day streetlights are more or less universally funded from local taxation. No one has ever managed to charge directly the user of the streetlight; creating and managing a use-based charging model would be several orders of magnitude more expensive than the provision of the light itself. Light being tricky stuff would provide much of the complication. How much light is enough before you get charged? So how close to the light can you get for free? Do you get charged different amounts depending on the how dark it would be if the light was turned off? How foggy it is? Do people who get the light spilling into their houses get charged? Anyhow, it's far too complex a question and to my knowledge it has never even been attempted. Residents in the town/region pay, everyone who passes by at night benefits.

Recently I argued against open source software being a cartel market. I closed that post by pointing out that software could not actually be seen as a normal economic good; can software be best seen as a public good? Interesting question...

The formal definition of a public good is that it is non-rivalrous and non-excludable. A good is seen to be excludable if you can easily and cheaply prevent those who have not paid for the good from consuming it, while a rivalrous good is one where your consumption of it prevents another from having it.

So here I am with my computer, using LibreOffice to write this post. If software is a normal good then my using it should be preventing you using it. Which clearly it does not as even if you don't yet have it you could just follow the download link to get it, I could give you a copy of the one I'm using and, on a thin client setup, we could all be using the same installed copy, so software (all software) is non-rivalrous; one tick in the box. Interestingly the common example of a non-rivalrous good is broadcast media - yet software is utterly non-rivalrous while broadcast media has a number of restrictions (no simultaneous use of wavelengths, requirement of near line of sight to transmitter, limited range). An alternative definition of a non-rivalrous good is one that could be provided at a very low additional (marginal) cost for each additional customer. Software clearly satisfies this definition and then some as the marginal cost for each additional customer/user is effectively zero.

So is software excludable? Well, companies whose business model is the production of software for sale have historically gone to great lengths to prevent people from using software without paying for it; and yet even with significant public monies being spent in helping prevent this illicit use, people continue to make use of all sorts of software they are not supposed to, entirely free. Arguably even the reasonably effective methods that have been used to prevent software being used illicitly, like dongles, are so expensive relative to the cost of the copy of the software that they would automatically fail the excludability test.

So software is non-rivalrous and non-excludable. A pure public good by normal economic definition. Yet it is not how many people currently view it, and much software is sold at high prices to individuals as if it was a normal good. Actually much software is sold prices so high relative to the cost of the copy of the software that it's priced at what economics would view as monopoly prices.

Why isn't software provided as a public good? Why isn't it something we collectively provide? Surely this would be to the greatest benefit of all, economically speaking, and be far more in keeping with the nature of software. Oddly, it is not uncommon for companies who produce software for sale to spend more money on selling, marketing and protecting the sales of the software than they do on making the software better. Economically speaking this makes no sense - it is the attempt to create artificial exclusions that costs so much money, rather than improvements to the product.

So software could be, should be a public good. Come to think of it, wouldn't it be even better, something more than a public good, if software was not only provided collectively but in a way that made it easy for others to improve? Like if the underlying programming code was available and people could fix stuff and add new features and then make the resulting improved, added value version available to everyone else allowing the virtuous cycle to start again. Wouldn't that be a truly amazing thing? Ever more added value for all, the economics are just brilliant; just need a name for the approach and then surely this "available code" software will spread like wildfire...

A Tablet Conundrum

CES came and went without any significant/useful/timely Microsoft announcements in the tablet sphere. If 2011 is finally to be the year of the tablet, and ownership of tablets starts to become as common as ownership of smartphones, what can Microsoft do about their current uncompetitive position?

Some have asked why has Microsoft not reacted more quickly to the new tablet form factor. After all they have considerable experience making software for tablets and touchscreen devices. I encountered a modified version of XP on tablets in 2003 (and have owned an XP-licenced IBM X41T for over 5 years) and touchscreen handhelds running Microsoft operating systems had been around for some years before that. Even the acquisition of the Surface product should have brought in relevant expertise.

We can point the finger of blame at Microsoft's incumbent status, or at their arguably over-bureaucratic HR and management systems; but there is another far more problematic issue: the business model currently required to succeed in this new marketplace.

Consider the leading operating systems in the new tablet marketplace, Apple iOS and Android. Consider also the bit-players of desktop Linux derivatives like Ubuntu, as well as the established business solution provider RIM with its PlayBook.

So what is the challenge for Microsoft from these tablet operating systems? Simply that those listed above have one thing in common - they are not in themselves sold. All are made by companies with different business models. Apple design and sell consumer hardware based on a premium brand and iOS is a by-product of that business. Android is made by multiple companies, the largest contributor, Google, gets its revenue from advertising and most of the other contributors design and/or make and sell hardware, both components and end-user products. The heavy lifting in Linux is done by companies who sell services or hardware, with significant input from academics and volunteers, and RIM sell a high value end-to-end service that is embedded in those companies that have adopted it.

Microsoft makes much of its money by selling software licences in bulk; specifically most of its profits are generated by software sales, with a large proportion of that being OS licences in massive quantities. But in the current tablet market there seems little opportunity for charging for an OS. Apple's pricing certainly has a premium, but that is accounted for by the brand; the Android tablet market is highly competitive on both price and design innovation, as will be the desktop Linux tablet market, while RIM will sell to RIM users with, maybe, a slight premium based on their unique proposition.

So there's a conundrum - whether or not to rush to invest large amounts of money in developing a tablet OS to sell in a market that does not charge for the OS. So if a decent entry level tablet stabilises at a 2011 retail price of USD350 (based on component and assembly costs and normal channel markups) what can Microsoft really charge manufacturers for the OS?

Depending on local sales taxes, a charge of USD50 for the operating system would put a 15-20% premium onto the tablet price, which can only be sustained if the market can be led to believe the product is premium. Given Microsoft's lack of recent success in mobile devices in general, to achieve this requires some considerable thought and work; they need to ensure that their tablet operating system is launched with minimum bugs, a decent upgrade process and roadmap and enough innovative elements to gain mindshare and thus justify a premium on the cost of the tablet. Of course they will also need a thriving application marketplace where most apps are free or nearly so.

That gets Microsoft to a requirement for more time. They need time to come up with innovations, time to optimise an operating system for a new hardware platform, time to have a strategy that allows a premium in mass market sales and time to generate market numbers sufficient to sustain the Microsoft operating cost model. The length of time required is likely to be at least 18 months, so nothing can be forthcoming until 2012 or later.

There is every sign that the advances in hardware that allowed the new tablet designs are also allowing tablets to have sufficient processing power to do most everything that a desktop PC can, but with 12 hours battery life. Then there is the increasing move online via wireless broadband, 3G or wifi - it is more and more common to have documents edited and published via the web, more common to have corporate and social interaction and collaboration via the Internet; even thin client and VDI developments lend themselves to a move to tablets.

Watch the tablet become ubiquitous, encouraging ever more people to move corporate and personal IT/IS functionality online; if Microsoft do not have a significant presence in the tablet market as this happens this hits their revenue stream from operating systems as the tablets eat at desktop and laptop sales, and hits it again as more people move functionality online, diminishing interest in high-priced desktop productivity tools.

This then is the real conundrum: do Microsoft rush to bring out an uncompetitive (features, user experience, extendability) operating system into an already competitive marketplace and risk failure because of the revenue requirement, the presence of incumbents and their alternative business' models, or take time to bring out a competitive OS while their main revenue streams are under attack, the players in the new market become ever stronger and ever more established; and while hardware prices reduce, increasing the premium their business model requires. And then still have a significant risk that the new OS is unable to get sufficient (or any) market share to replace their lost revenues...

I think Lorenzo Dow said it best a couple of centuries ago:

"And you will be damned if you do - And you will be damned if you don't"

Talk Talk Text Text...

So, a few weeks ago my aunt reports to me her email is not working. As this was something I originally set up for her a few months ago I promptly arranged to drive down to London and have a look.

A quick diagnosis followed; not so much not email as not Internet. The TalkTalk branded ADSL router was not syncing up to the TalkTalk service. I delved deep into my technical archives and checked everything I could, tested the router on different phone sockets and with different microfilters. Nothing.

So I call TalkTalk for her. I quickly get bounced to 2nd line support when the answer to the question "what version of Windows do you have" got the response "Linux". The 2nd line guy was good, in that he got I was technical enough to know it was the router, promptly ran some line tests and said he would escalate it. Escalation would take 3 days to get something and up to 5. I gave them my mobile number to call if they needed any more information.

Ok.

So, next day I get a text "Our engineers have tested your line and have been unable to find a fault. If a fault still persists please reply with the words NOT FIXED". I ignore it, presuming it is a mistake as the guy on the phone had said his tests found a fault.

Two more days pass. I call my aunt just to check if anyone has been in contact with her or if the Internet is now working again. Nothing. She assures me she will email the instant she gets her email back. So I call TalkTalk, to be told that the fix response time is up to 6 days...

So, next day I get a text "Our engineers have tested your line and have been unable to find a fault. If a fault still persists please reply with the words NOT FIXED". I duly respond with "NOT FIXED".

Some more time passes. Eventually, some 11 days after the report I get a happy email "Two TalkTalk men came and fixed it". Excellent. That's good. Slow, sillily slow, but good.

Two days ago, another 11 days after the problem was resolved, I get a text "Our Engineer reports that your fault is fully resolved. Please reply using the words RESOLVED or ONGOING, Thank you". Ignoring the extended delay and unnecessary capitalisations in the message, I dutifully respond "RESOLVED".

Then this morning, another text message "Following further investigation on your fault. Please reply FIXED or NOT FIXED". Mildly confused, but still willing, I respond "FIXED".

To which came the prompt response "We did not recognise the response you sent back. Please refer to our previous text for the correct response to send. Thank you". Well, the only response left to me is "NOT FIXED"...