Su Tech Ennui

Wednesday, February 6, 2008

Formatting documents for the Amazon Kindle

Today's Blog post is going to be a little different. Instead of preparing an article in advance, I'm going to treat this entry like a Wiki page and update it continuously as I make new discoveries…

(For the moment, I've still to add some links to example pages and a couple of screen shots.)

We were lucky enough to get our hands on a Kindle quite early, but as I mentioned in another post, we're paying the Early Adopter Tax because the facilities in the Kindle are quite limited. (Bottom line if you're interested in getting one, as an e-book reader its merely adequate, but as an always-on internet device with no usage charges, it's a winner. Although I qualify that latter statement by adding that you'ld need to get about 2 years of usage out of it to make it financially viable against Sprint service on a Palm Treo 650… With the Internet access on the Kindle being flagged as 'experimental' it's not clear that it will remain free indefinitely, if they get cold feet over people actually using it for internet and not just for downloading books…)

First, some basic facts. The screen is 600 x 800 pixels at a physical size of 3.6in x 4.8 inches (no hardware border — the pixels go right up to the edge), giving a resolution of 166.66dpi. It only supports 2-bit grey scale (white, light grey, dark grey and black) even though the browser tells the web server that it supports 32bit color! Images are transcoded on the fly by Amazon's proxy, which all traffic goes through (meaning that they could later shut down free internet access if it starts costing them too much). There's no Javascript, so no way of asking the browser what the screen resolution is.

Now, it has to be said that text on the Kindle looks OK, but with only one font supported in the Browser, it's not going to allow much creativity in page design, especially if you want to offer free eBooks via HTML. The only supported font is Caecilia, which is a slab-serif face. You might be aware of a font called Rockwell — it's in that style. The font comes with only the ISO Latin1 character set glyphs, with bold and/or italic supported as would be expected.

There are a limited number of sizes supported, although interestingly there appears to be a bug in the HTML rendering, because HTML defines 7 font size names for use in CSS: xxsmall, xsmall, small, medium, large, xlarge and xxlarge. These are supposed to map to the font sizes 1 through 7 (as in <font size=7>). However it seems that xsmall and small are the same font size but with marginally different letter spacing, and that all the following sizes are 1 less than they should be. This means that xxlarge is really size 6, with size 7 not being accessible using CSS — only with <span ...>. This means in effect that 8 sizes of type are supported.

So... the first discovery about the Kindle is that ebooks look a lot nicer than web pages. For example, ebooks are usually justified and often have hyphenation enabled, but the web browser doesn't support justification and of course HTML doesn't have any automatic hyphenation.

What's worse is that the HTML browser is very limited by default, although there's an option that the user can manually turn on to enable a few extra features. Justification, however, is not one of them.

So, at this point, I started examining what was possible under the 'advanced' version of the browser. One critical feature that was enabled was the use of <div> to position text at absolute coordinates. With this turned on, it's possible to simulate full justification by microspacing between words.

By the way, if you don't need to do justi­fi­cation, there's an easy hack to do hyph­en­ation that doesn't req­uire pre-form­atting the text: just pre-process your HTML to in­sert the soft-hy­phen char­acters at hy­phen­ation points in every word. (Hope­fully this para­graph will con­tain at least one hy­phen that came from in­sert­ing a prof­lig­acy of soft hy­phen char­acters and long­ish words...)

One downside of pre-formatting text nicely inET X is that it's not scalable. If the Kindle didn't allow you to change the font size, that wouldn't be a problem, but it does. There's a hidden menu where you can change the default font size. So if you do change the default size, any documents created using absolute positioning will fall apart. So documents will need an intro page to detect if the font size has been changed. It's on my to-do list. (There's no javascript and no way of detecting it except by asking the user; similarly you can't find out the screen size in pixels — you just have to know it, and hope that later Kindles don't change it)

I've been looking at usingET X to do this justification, which requires writing a dvi to html driver that positions text absolutely. Which meant that I had to find font metric files for the Caecilia fonts that HTML uses. I believe they're actually on the Kindle, but I don't have a cable to download them so I poked around the net until I found something close enough although not quite the exact same font. Then I spent an evening of trial and error tweaking theET X font sizes until the spacing exactly matched the text in HTML.

What remains to be done is to build aET X base file that uses the characters available in Kindle HTML only.

Incidentally the subset of ISO Latin1 that HTML supports is smaller than the subset that's supported in ebooks, even though the ebook format is basically just HTML too.

Since this doesn't work in the basic browser with the extensions not turned on, we need a way to detect the browser style so that we can advise the user to change mode. Since the browser doesn't support Javascript, this was quite a challenge. I eventually solved it by designing a page that uses divs to overlay some extra text only when the browser is in advanced mode. The extra text contains the link that takes you to the nicely formatted document; otherwise you're taken to a page with instructions on how to enable advanced mode. (By the way that link will only work on a Kindle. I need to add even more HTML hi-jinks to provide a version that will work on a regular web browser as well. It'll make sense when I include the screenshot later this week...)

Here's a screenshot of the intro page to a test book in 'dumb' mode

[insert scan here]

(by the way I do know that the Kindle can save screenshots to SD memory — I just put it under the scanner for fun, since you can't scan traditional displays!)

and here is the same page in 'advanced' mode

[insert scan here]

And finally, here's some tech info that will come in useful for anyone working on this device:

[insert table of pixel and font sizes]

Also to add: page showing HTML features that work in browser (distinct from table of entity references).

Mention the lack of left and right single and double quotes, and ways to hack around that; also em-dashes, and the various tricks around that (superscripted underlines, or rules, or overlapping minuses).

Point out the trick used to typeset the wordET X here, and show how it falls apart when you use view/text size/larger to increase the font size — same problem as on the Kindle. So you have to do the font size switching *outside* of the browser, at the server, and always leave the browser in the default font size.

Also point out that justification in the Kindle (and most web browsers) is poor, and thatET X does a far better job, although in regular browsers you can combine the justification and the soft hyphenation trick to get quite a readable page.

(And add a comment about the appalling physical layout and the accidental pressing of all the side buttons — leaving one awkward area where it can be held safely, which rapidly leads to dirty fingerprints in that area)

Monday, January 21, 2008

Good customer service experience with Netflix

The overall theme of this blog is that there's really no excuse for crap software and the needlessly wasted hours we spend working around it. So I felt a blog post coming on this morning when I discovered that Netflix "Watch Now" wasn't working on my laptop, and I had spent several hours researching the problem on the net, and of course on the Netflix web site.

Getting nowhere, I finally gave up and tried the last resort of all techies - calling the support line (something you never expect to work...) and of course, what should happen but an hour and a quarter later I hung up the phone, tired of waiting for someone to pick up.

Well, while researching the problem, I had accidentally come across the blog of Netflix's CIO, and I'd noticed that a couple of people had posted requests there for help with netflix problems and appeared to have received satisfactory help. So somewhat apologetically I did the same because I'd pretty much spent the day on this problem (my day off work - MLK) and was getting somewhat frustrated.

I was much surprised and delighted when later that afternoon Netflix called me and a very competant young man, Matthew, worked out exactly what my problem was and pointed me at the solution.

Good work, and credit due to Netflix.

Now, for anyone else who bought the Walmart "Black Friday" Acer portable (model 5315-2153) who finds that they get a blank screen when watching Netflix Watch Instantly (and for the benefit of other Googlers looking for a solution, 'black screen' when watching 'Netflix Watch Now'), here's the cause and the solution:

At some point you've updated your Intel 965 Chipset driver, and picked up the October 2007 one. This breaks Watch instantly and apparently some other things such as Second Life!

A later driver (10th Jan 2008) can be found as follows:

Start with Googleing for upgrade intel 965 driver acer aspire 5315 which will take you to an Intel support page.

Select the drop-downs: graphics/laptop graphics controllers/965 express chipset family/advanced windows users

This takes you to a form for email requests - but we're only interested in the link to "download" which is above the form.

Select "Windows vista home basic 32 bit", and download "Intel graphics
media accelerator driver for windows vista 32 (exe)
"

Running this exe (as Administrator) installs smoothly, and on reboot, Netflix Watch Instantly is now working just fine.

Kudos to netflix for knowing the problem; minus several thousand brownie points to Intel (and Acer) for releasing a driver update that breaks stuff.


G
PS Matthew explained that the delays in answering the support line were uncharacteristic and due to a huge influx of new folks all trying Watch Instantly for the first time.

Friday, December 7, 2007

Get on the train!

I'm beta-testing a new Google service at the moment which isn't all that well known yet: Grand Central. It's a web-based voicemail system and like many of the VOIP services on the net, it lets you select a phone number from almost anywhere in the US - you're not limited to your local calling area.

When I picked my number, I started with a few words that might make good phone numbers when spelled out, and worked out the area code that corresponds to the first 3 digits - there aren't that many area codes out there that make sense as words so I was lucky when I found that "TOA" mapped to a valid one. Then they gave me a choice of a couple of dozen numbers, so I narrowed that list down to ones which started with an exchange code beginning with '5' so that my number would then be dialable as "TOAL nnnnnn". From there it was trivial to feed the handful of proferred numbers into a "phone to word" lookup program, and find which of them made a sensible word! Bingo! - a personalised phone number for free! :-)

The GC service sports some pretty nice features; the only downside is that it looks like it'll be free only during the beta test - that wasn't obvious until *after* I'd signed up. So there's a serious risk I might get hooked on this stuff (or have too many people using my new number to be willing to cancel it after the beta) and so end up paying for yet another phone line.

Still, it is nice. Really nice. Sign up at their web site - I put my name on the waiting list around 11pm one evening and had an invitation to join in my mailbox by midnight. Once you're in, it's like the early days of GMail where you get 10 invitations to pass on to friends, and they in turn get 10 more, ad infinitum.

G

Tuesday, December 4, 2007

Why Linux isn't ready for the desktop

My Linux at work (OpenSuSE 10.3) stopped working for no apparent reason the other day - well, not entirely, but it wouldn't start X-Windows and gave me the totally content-free error message "The display server has been shut down about 6 times in the last 90 seconds. It is likely that something bad is going on."

This was a major problem because I was running another server on that system under vmware, and when I tried to start up the virtual machine from the command-line instead of the usual GUI, that failed too. (I suspect it never had worked, I just hadn't tried it before now.)

Fortunately I have some experience of unix and after much net research I finally found the solution:

touch /var/cache/fontconfig/stamp
/usr/bin/fc-cache -f

but can you imagine what would happen if a typical home user hit a problem like that? It would be "Honey, where's the WinXP install disk?"...

As much as I love unix, it just isn't ready for prime time on the desktop. Which is why I do all my own work from the command-line still...

G

Monday, November 19, 2007

like a phoenix, risen from the ashes...

This post is dedicated to anyone Googling for "overheating problems for HP Pavilion dv4000"...

For a couple of weeks now, my HP portable has been getting very hot underneath at the back, at which point it unceremoniously powers down. I searched for other people with the same problem and found many.

Well, I'm delighted to report that the problem is trivial to fix, and cheaply too!

The idiots at HP designed the portable so that the intake to the cooling fan is on the underside of the machine, so if you put it down on a bed, or a particularly plush carpet, it blocks. The fan also gets blocked if - like me - you've had it for over a year, and have never once blown out the dust bunnies!

So that's the fix... just grab a can of compressed air (or your home vacuum) and blow through the inlet and outlet cooling vents. The outlet is at the rear left as you face the keyboard, and the inlet is on the underside about three inches in from the back.

Now my machine is nice and cool again, and I saved myself the cost of a new portable :-) (Although truth be told, I did actually buy myself one of the Walmart pre-black-friday specials for about $350 last week, thinking I was going to be needing a backup soon...)

By the way the Walmart machine is quite nice, though there are odd things about it that you have to do a fair bit of net.homework to master. Also it's not a good choice if you want a Linux portable. If you do, put vmware on it and get your linux fix that way for now.

Circuit City was selling a similar machine this morning for $270. At this rate we'll be hitting One Laptop Per Child prices with regular portables soon :-)


G

Sunday, November 4, 2007

Quick hack #17 in a series of 42: inlining LaTeX "\newcommand" macros

A poster on the TeXhax mailing list asked how he could pre-process his LaTeX file to remove his own macros (created using \newcommand) because his typesetter for some reason does not allow user-defined macros. It turns out that TeX/LaTeX doesn't have the ability to output the de-macroed source, so I hacked up a macro processor in C that's compatible with LaTeX's syntax and leaves anything it doesn't recognise untouched.

I was quite surprised how hard it was to get right. It took me about 3 hours to write rather than the 30 minutes I expected, and here's why: in previous code where I've needed to do macro expansion, I've done it as a filter on reading the source stream, and as a result, if a macro definition included a call to another macro, it was trivially expanded on the fly before being fed into the new definition. This implementation falls out in the wash with no explicit effort, and for the sort of things I've needed it for in the past (such as an alias mechanism for a command-line shell, or for a cpp-like source-code preprocessor) it's actually what you want, because aliases chain correctly when redefined (eg alias ls = ls -l - you don't need to worry if ls is already an alias or not).

However for TeX's mechanism, the generated text is then reprocessed for recursive macro expansion. This is useful in TeX because it lets you use a macro to generate other macros, but it does complicate the business of writing the macro expansion code.

The solution I used was one that I had first come across in 1977 in an implementation of the POP2 language, called "Wonderpop". Wonderpop treated its input stream as a list like any other list. The last element of the list was a function call to fetch data from the input file (or console), and if that was all that was left in the list, then it was just like normal I/O ... but if you joined some other elements in front of the list, the next time you read from it, you'ld get those elements before you'ld get the remaining data in the source file. In C terms it was something like ungetc(), but far more powerful because the list could be manipulated. As well as pushing text back on the head of the input stream, you could pull out the Nth item from the input stream, and if that item had not been read yet, the next N items from the input stream would be fetched and converted to list cells, and the object at the end of the list would be the lazy evaluation function which would read the rest of the stream only when needed.

In fact, I believe you could append items to the end of these lists (though I hadn't ever done that myself) - in which case, once the function to fetch new data had exhausted the real file, it would return items from the appended list elements.

Wonderpop made great use of this modifiable input stream facility, by allowing the user to modify PROGLIST, which was a stream containing the source of your program itself. By using syntax-directed macros, you could add new keywords to the language which were implemented by manipulating your source code as a list. For example, if the base language only supported LOOP/IF ... BREAK/ENDIF, you could build your own implementation of WHILE or FOR on top of it. This made Wonderpop into a user-extensible language for very little added complexity in the compiler. Very cool.

So a simple version of the POP2 model was what I used for this TeX-like preprocessor: it has a large buffer in front of the actual stream object, to which you can push data back and have it be re-read before you get to the following data from stdin. And of course to avoid multiple buffer copies (shunting up data after a read if you didn't want your array to grow indefinitely), it uses a cyclic buffer. One advantage of a cyclic buffer that I hadn't thought of before I started implementing the code was that you could push back items to either end of the buffer, so they would be read either instantly or after the current expansion was finished. Turned out to be useful and necessary.

Being a quick hack so that the guy could get his document printed, I haven't made it ultra robust yet, but with the exception of a couple of deliberate shortcuts that I already know about, I think the code turned out pretty nicely. Here it is: http://www.gtoal.com/src/newcommand/newcommand.c.html

G

Saturday, November 3, 2007

Lesser artists borrow; great artists steal.


Remind you of anyone?

I was reading this post by Adam Finley claiming that a 1948 Startling Comics cover was the inspiration for Bender in the wonderful Futurama cartoon, and it brought to mind a children's toy my wife keeps in her jewelry box... she picked this up in a packet of Fritos somewhere around 1968 give or take a year. About the same time that Matt Groening would be in his formative years, given that he's about the same age as us. I think it's much more likely that the youthful Groening had a penchant for potato chips than a collection of 1940's comics!

The robot's carrying a spanner, fer gossake! How much more Bender-like can you get? Don't you think it's a stronger resemblance than the more he-man-ly robot from Startling #49..?

I can easily imagine that the image stuck in Groening's mind without him explicitly remembering that he'd seen it elsewhere.

It's a cool toy. Fifty bucks and it's yours, Matt, or I'll trade you for a signed Bender drawing ;-)

G