Friday, April 11, 2008
Semi-literate programming
Don Knuth had another problem. He wanted to write a large typesetting package, but wasn't happy with the tools available. So he wrote WEB, a tool that basically presents your source code in a more readable way - suitable indeed for publishing as a book. His argument was (I hope I'm not mischaracterising him) that we read programs more than we write them. So his Web system made reading programs really easy, though in my humble opinion they were still damned ugly to write.
Meanwhile, on the other side of the pond, Inmos was building transputers, and they created a parallel programming language called OCCAM which, among other things, was innovative in its layout style - it made the depth of indentation significant to the compiler rather than just the reader. (A trick that has been copied by Python in more recent years)
This feature allowed them to add a tweak to their text editor, which was the ability to fold a section of code so that it wasn't all displayed at once; but if you went into a fold (I was going to say 'clicked on it', but this was before the days of rabid clicking) it would expand in situ.
Many years later, and we have the World Wide Web (as opposed to Knuth's WEB) and we're still struggling to publish software in a way that's easy to read online. Knuth's stuff is great for books, but not so hot for web pages.
And we have various tools that take source code and display them as HTML, though not all are of equal quality. Some of them seem to take the approach that if you can distinguish a syntactic feature, you should do so, in as bright a colour as possible!)
And finally we get to the meat of today's post. Borrowing a little from Knuth, and a little from Inmos, I've modified my C to HTML filter so that it adds folds to the HTML view of a program. Although it's technically possible to do so automatically as Inmos did with OCCAM, it's not desirable because doing so would fold at syntactic units rather than logical groups. So I have taken the old school approach of inserting markup into the source file for the benefit of the folding code.
I've tried to so so fairly unobtrusively, with markers that don't grate on the sensibilities too much when you're editing the raw text; and they're hidden completely in the HTML view. After I wrote this I discovered that there is a sort of a standard for folding markup ("{{{" and "}}}") but rather than go back and change my code to the standard, I'm going to stick with my first idea for now. I may make that change later if I want to use a folding editor on my sources, but for now I'ld rather have the aesthetics of my own scheme which I think is less intrusive.
Here's my current programming project: it's about 2000 lines of C, but displays as a single page. My rule of thumb in selecting what to mark is to fold about a screen's worth at a time - pretty much the same rule we used to use when writing procedures with a regular text editor, except now our screens are significantly larger.
I have to say I like the way it has come out. It doesn't have all of Knuth's tricks, such as the ability to reorder program fragments, but I think the one thing it does, it does well, and gives most of the benefit that came from Knuth's WEB - the ability to read the overall structure and a textual description of an algorithm, and only push down into the code if you really need to.
This was an easy hack to add to a C to HTML filter, and I've already done the same trick for PL/SQL. Duplicating it for other languages should be quite easy too.
Wednesday, February 6, 2008
Formatting documents for the Amazon Kindle
(For the moment, I've still to add some links to example pages and a couple of screen shots.)
We were lucky enough to get our hands on a Kindle quite early, but as I mentioned in another post, we're paying the Early Adopter Tax because the facilities in the Kindle are quite limited. (Bottom line if you're interested in getting one, as an e-book reader its merely adequate, but as an always-on internet device with no usage charges, it's a winner. Although I qualify that latter statement by adding that you'ld need to get about 2 years of usage out of it to make it financially viable against Sprint service on a Palm Treo 650… With the Internet access on the Kindle being flagged as 'experimental' it's not clear that it will remain free indefinitely, if they get cold feet over people actually using it for internet and not just for downloading books…)
First, some basic facts. The screen is 600 x 800 pixels at a physical size of 3.6in x 4.8 inches (no hardware border — the pixels go right up to the edge), giving a resolution of 166.66dpi. It only supports 2-bit grey scale (white, light grey, dark grey and black) even though the browser tells the web server that it supports 32bit color! Images are transcoded on the fly by Amazon's proxy, which all traffic goes through (meaning that they could later shut down free internet access if it starts costing them too much). There's no Javascript, so no way of asking the browser what the screen resolution is.
Now, it has to be said that text on the Kindle looks OK, but with only one font supported in the Browser, it's not going to allow much creativity in page design, especially if you want to offer free eBooks via HTML. The only supported font is Caecilia, which is a slab-serif face. You might be aware of a font called Rockwell — it's in that style. The font comes with only the ISO Latin1 character set glyphs, with bold and/or italic supported as would be expected.
There are a limited number of sizes supported, although interestingly there appears to be a bug in the HTML rendering, because HTML defines 7 font size names for use in CSS: xxsmall, xsmall, small, medium, large, xlarge and xxlarge. These are supposed to map to the font sizes 1 through 7 (as in <font size=7>). However it seems that xsmall and small are the same font size but with marginally different letter spacing, and that all the following sizes are 1 less than they should be. This means that xxlarge is really size 6, with size 7 not being accessible using CSS — only with <span ...>. This means in effect that 8 sizes of type are supported.
So... the first discovery about the Kindle is that ebooks look a lot nicer than web pages. For example, ebooks are usually justified and often have hyphenation enabled, but the web browser doesn't support justification and of course HTML doesn't have any automatic hyphenation.
What's worse is that the HTML browser is very limited by default, although there's an option that the user can manually turn on to enable a few extra features. Justification, however, is not one of them.
So, at this point, I started examining what was possible under the 'advanced' version of the browser. One critical feature that was enabled was the use of <div> to position text at absolute coordinates. With this turned on, it's possible to simulate full justification by microspacing between words.By the way, if you don't need to do justification, there's an easy hack to do hyphenation that doesn't require pre-formatting the text: just pre-process your HTML to insert the soft-hyphen characters at hyphenation points in every word. (Hopefully this paragraph will contain at least one hyphen that came from inserting a profligacy of soft hyphen characters and longish words...)
One downside of pre-formatting text nicely inET X is that it's not scalable. If the Kindle didn't allow you to change the font size, that wouldn't be a problem, but it does. There's a hidden menu where you can change the default font size. So if you do change the default size, any documents created using absolute positioning will fall apart. So documents will need an intro page to detect if the font size has been changed. It's on my to-do list. (There's no javascript and no way of detecting it except by asking the user; similarly you can't find out the screen size in pixels — you just have to know it, and hope that later Kindles don't change it)I've been looking at usingET X to do this justification, which requires writing a dvi to html driver that positions text absolutely. Which meant that I had to find font metric files for the Caecilia fonts that HTML uses. I believe they're actually on the Kindle, but I don't have a cable to download them so I poked around the net until I found something close enough although not quite the exact same font. Then I spent an evening of trial and error tweaking theET X font sizes until the spacing exactly matched the text in HTML.
What remains to be done is to build aET X base file that uses the characters available in Kindle HTML only.
Incidentally the subset of ISO Latin1 that HTML supports is smaller than the subset that's supported in ebooks, even though the ebook format is basically just HTML too.
Since this doesn't work in the basic browser with the extensions not turned on, we need a way to detect the browser style so that we can advise the user to change mode. Since the browser doesn't support Javascript, this was quite a challenge. I eventually solved it by designing a page that uses divs to overlay some extra text only when the browser is in advanced mode. The extra text contains the link that takes you to the nicely formatted document; otherwise you're taken to a page with instructions on how to enable advanced mode. (By the way that link will only work on a Kindle. I need to add even more HTML hi-jinks to provide a version that will work on a regular web browser as well. It'll make sense when I include the screenshot later this week...)
Here's a screenshot of the intro page to a test book in 'dumb' mode
[insert scan here]
(by the way I do know that the Kindle can save screenshots to SD memory — I just put it under the scanner for fun, since you can't scan traditional displays!)
and here is the same page in 'advanced' mode
[insert scan here]
And finally, here's some tech info that will come in useful for anyone working on this device:
[insert table of pixel and font sizes]
Also to add: page showing HTML features that work in browser (distinct from table of entity references).
Mention the lack of left and right single and double quotes, and ways to hack around that; also em-dashes, and the various tricks around that (superscripted underlines, or rules, or overlapping minuses).
Point out the trick used to typeset the wordET X here, and show how it falls apart when you use view/text size/larger to increase the font size — same problem as on the Kindle. So you have to do the font size switching *outside* of the browser, at the server, and always leave the browser in the default font size.
Also point out that justification in the Kindle (and most web browsers) is poor, and thatET X does a far better job, although in regular browsers you can combine the justification and the soft hyphenation trick to get quite a readable page.
(And add a comment about the appalling physical layout and the accidental pressing of all the side buttons — leaving one awkward area where it can be held safely, which rapidly leads to dirty fingerprints in that area)
Monday, January 21, 2008
Good customer service experience with Netflix
Getting nowhere, I finally gave up and tried the last resort of all techies - calling the support line (something you never expect to work...) and of course, what should happen but an hour and a quarter later I hung up the phone, tired of waiting for someone to pick up.
Well, while researching the problem, I had accidentally come across the blog of Netflix's CIO, and I'd noticed that a couple of people had posted requests there for help with netflix problems and appeared to have received satisfactory help. So somewhat apologetically I did the same because I'd pretty much spent the day on this problem (my day off work - MLK) and was getting somewhat frustrated.
I was much surprised and delighted when later that afternoon Netflix called me and a very competant young man, Matthew, worked out exactly what my problem was and pointed me at the solution.
Good work, and credit due to Netflix.
Now, for anyone else who bought the Walmart "Black Friday" Acer portable (model 5315-2153) who finds that they get a blank screen when watching Netflix Watch Instantly (and for the benefit of other Googlers looking for a solution, 'black screen' when watching 'Netflix Watch Now'), here's the cause and the solution:
At some point you've updated your Intel 965 Chipset driver, and picked up the October 2007 one. This breaks Watch instantly and apparently some other things such as Second Life!
A later driver (10th Jan 2008) can be found as follows:
Start with Googleing for upgrade intel 965 driver acer aspire 5315 which will take you to an Intel support page.
Select the drop-downs: graphics/laptop graphics controllers/965 express chipset family/advanced windows users
This takes you to a form for email requests - but we're only interested in the link to "download" which is above the form.
Select "Windows vista home basic 32 bit", and download "Intel graphics
media accelerator driver for windows vista 32 (exe)"
Running this exe (as Administrator) installs smoothly, and on reboot, Netflix Watch Instantly is now working just fine.
Kudos to netflix for knowing the problem; minus several thousand brownie points to Intel (and Acer) for releasing a driver update that breaks stuff.
G
PS Matthew explained that the delays in answering the support line were uncharacteristic and due to a huge influx of new folks all trying Watch Instantly for the first time.
Friday, December 7, 2007
Get on the train!
When I picked my number, I started with a few words that might make good phone numbers when spelled out, and worked out the area code that corresponds to the first 3 digits - there aren't that many area codes out there that make sense as words so I was lucky when I found that "TOA" mapped to a valid one. Then they gave me a choice of a couple of dozen numbers, so I narrowed that list down to ones which started with an exchange code beginning with '5' so that my number would then be dialable as "TOAL nnnnnn". From there it was trivial to feed the handful of proferred numbers into a "phone to word" lookup program, and find which of them made a sensible word! Bingo! - a personalised phone number for free! :-)
The GC service sports some pretty nice features; the only downside is that it looks like it'll be free only during the beta test - that wasn't obvious until *after* I'd signed up. So there's a serious risk I might get hooked on this stuff (or have too many people using my new number to be willing to cancel it after the beta) and so end up paying for yet another phone line.
Still, it is nice. Really nice. Sign up at their web site - I put my name on the waiting list around 11pm one evening and had an invitation to join in my mailbox by midnight. Once you're in, it's like the early days of GMail where you get 10 invitations to pass on to friends, and they in turn get 10 more, ad infinitum.
G
Tuesday, December 4, 2007
Why Linux isn't ready for the desktop
This was a major problem because I was running another server on that system under vmware, and when I tried to start up the virtual machine from the command-line instead of the usual GUI, that failed too. (I suspect it never had worked, I just hadn't tried it before now.)
Fortunately I have some experience of unix and after much net research I finally found the solution:
touch /var/cache/fontconfig/stamp
/usr/bin/fc-cache -f
but can you imagine what would happen if a typical home user hit a problem like that? It would be "Honey, where's the WinXP install disk?"...
As much as I love unix, it just isn't ready for prime time on the desktop. Which is why I do all my own work from the command-line still...
G
Monday, November 19, 2007
like a phoenix, risen from the ashes...
For a couple of weeks now, my HP portable has been getting very hot underneath at the back, at which point it unceremoniously powers down. I searched for other people with the same problem and found many.
Well, I'm delighted to report that the problem is trivial to fix, and cheaply too!
The idiots at HP designed the portable so that the intake to the cooling fan is on the underside of the machine, so if you put it down on a bed, or a particularly plush carpet, it blocks. The fan also gets blocked if - like me - you've had it for over a year, and have never once blown out the dust bunnies!
So that's the fix... just grab a can of compressed air (or your home vacuum) and blow through the inlet and outlet cooling vents. The outlet is at the rear left as you face the keyboard, and the inlet is on the underside about three inches in from the back.
Now my machine is nice and cool again, and I saved myself the cost of a new portable :-) (Although truth be told, I did actually buy myself one of the Walmart pre-black-friday specials for about $350 last week, thinking I was going to be needing a backup soon...)
By the way the Walmart machine is quite nice, though there are odd things about it that you have to do a fair bit of net.homework to master. Also it's not a good choice if you want a Linux portable. If you do, put vmware on it and get your linux fix that way for now.
Circuit City was selling a similar machine this morning for $270. At this rate we'll be hitting One Laptop Per Child prices with regular portables soon :-)
G
Sunday, November 4, 2007
Quick hack #17 in a series of 42: inlining LaTeX "\newcommand" macros
I was quite surprised how hard it was to get right. It took me about 3 hours to write rather than the 30 minutes I expected, and here's why: in previous code where I've needed to do macro expansion, I've done it as a filter on reading the source stream, and as a result, if a macro definition included a call to another macro, it was trivially expanded on the fly before being fed into the new definition. This implementation falls out in the wash with no explicit effort, and for the sort of things I've needed it for in the past (such as an alias mechanism for a command-line shell, or for a cpp-like source-code preprocessor) it's actually what you want, because aliases chain correctly when redefined (eg alias ls = ls -l - you don't need to worry if ls is already an alias or not).
However for TeX's mechanism, the generated text is then reprocessed for recursive macro expansion. This is useful in TeX because it lets you use a macro to generate other macros, but it does complicate the business of writing the macro expansion code.
The solution I used was one that I had first come across in 1977 in an implementation of the POP2 language, called "Wonderpop". Wonderpop treated its input stream as a list like any other list. The last element of the list was a function call to fetch data from the input file (or console), and if that was all that was left in the list, then it was just like normal I/O ... but if you joined some other elements in front of the list, the next time you read from it, you'ld get those elements before you'ld get the remaining data in the source file. In C terms it was something like ungetc(), but far more powerful because the list could be manipulated. As well as pushing text back on the head of the input stream, you could pull out the Nth item from the input stream, and if that item had not been read yet, the next N items from the input stream would be fetched and converted to list cells, and the object at the end of the list would be the lazy evaluation function which would read the rest of the stream only when needed.
In fact, I believe you could append items to the end of these lists (though I hadn't ever done that myself) - in which case, once the function to fetch new data had exhausted the real file, it would return items from the appended list elements.
Wonderpop made great use of this modifiable input stream facility, by allowing the user to modify PROGLIST, which was a stream containing the source of your program itself. By using syntax-directed macros, you could add new keywords to the language which were implemented by manipulating your source code as a list. For example, if the base language only supported LOOP/IF ... BREAK/ENDIF, you could build your own implementation of WHILE or FOR on top of it. This made Wonderpop into a user-extensible language for very little added complexity in the compiler. Very cool.
So a simple version of the POP2 model was what I used for this TeX-like preprocessor: it has a large buffer in front of the actual stream object, to which you can push data back and have it be re-read before you get to the following data from stdin. And of course to avoid multiple buffer copies (shunting up data after a read if you didn't want your array to grow indefinitely), it uses a cyclic buffer. One advantage of a cyclic buffer that I hadn't thought of before I started implementing the code was that you could push back items to either end of the buffer, so they would be read either instantly or after the current expansion was finished. Turned out to be useful and necessary.
Being a quick hack so that the guy could get his document printed, I haven't made it ultra robust yet, but with the exception of a couple of deliberate shortcuts that I already know about, I think the code turned out pretty nicely. Here it is: http://www.gtoal.com/src/newcommand/newcommand.c.html
G