A Little Less Failure This Time?

This week started out much like the last.  More adventures in failure.  The guys on Zipit IRC seemed to be struggling with an improperly rotated links -g running on DirectFB in openwrt.  I pointed out Ray Haque's secret sauce for that problem, but failed to get it done.  So I got the bright idea to try and build it myself, with SDL and all my favorite patches from the IZ2S world.  Sound familiar?  Yeah, and it started out exactly the same way.  One failure after another. 

To start with, I had an old failed rockbox for openwrt that I never quite got working right.  It runs, but horribly, and eventually I lost interest and moved on.  Shoulda took that as a warning.  Stay away.  But no, I fired up the old P3 with my openwrt setup on it and updated all the git archives following Anarsoul's instructions.  At least I thought that's what I was doing.  Apparently the browser still had a cached version of the instructions from last February when I was working on the rockbox build.  And the cached instructions contained a typo.  A dash character in one of the feed names prevented me from building the qi repositories with the links package that I wanted.  For several hours!  The guys on IRC were quite helpful, and we eventually worked out the problem only to discover the instructions on the internet were already fixed, and probably had been for months.

See.  There's the typo, right there.  Can you spot the difference?

Oh well, moving on.  So I started the P3 building the goodies and watched for a few minutes before... guess what?  Hard disk failure!  Aargh!  Stupid old 80GB external drive developed a bad superblock.  I sorta remembered this drive gave me trouble before so I decided to abandon it and get a newer one rather than try to fix it.  Someday maybe I'll try to recover the bad rockbox build with one of the backup superblocks.

The next day I nabbed a 500GB drive from one of the kids and started over.  I refetched all the openwrt archives, followed the correct instructions to setup the feeds and started the build again.  Then I went to sleep, exhausted but hopeful.  The build failed a few times on supertux, the kbd package, and libmpg123 so I had a few restarts to do the next day.  But eventually it built everything else I'd selected.  Gotta see what's up with kbd and libmpg123 since I may want them someday.  But for now I'm gonna build with IGNORE_ERRORS=m to keep it going after failures.

Anyhow, the links package compiled so I just needed to apply my patch from IZ2S and rebuild.  Or so I thought.  I tested the patch in the directory with the links sources and it applied cleanly.  Cool.  However when openwrt compiles, it wants to wipe out the directory with the package sources, start fresh, and apply it's own patches.  But it took a while to figure out how to remake my patch the way openwrt wants it.  You need to make it run with patch -p0 so the destination directory name in the patch must match exactly with the directory name in the package tarball used by openwrt.  At least I think that's the case.  It's hard to see exactly what the byzantine openwrt build system is doing, even with the V=99 command line option.

I got some more hints from the IRC guys on how to reproduce the openwrt build environment so I could manually step through the patching and configuring and try and fix the broken parts.  I don't think I ever completely reproduced the environment but got close enough to figure out some of the trouble spots.  Apparently for openwrt you need to make a makefile for each package that runs configure to create a makefile for the package.  Hmmm.  I examined the existing meta-makefile and attempted to convert the tricks used to configure directfb into tricks that would configure links for SDL.  It didn't exactly work.  The openwrt build process was successfully applying my patch, but failing during the configure stage because it couldn't find SDL.  I tried looking at some other packages like nupdf and sdlwidgets for SDL config hints, but to no avail.  As many things do, the answer crystalized in my sleep, and when I woke up I found the sdl-config program I needed in the openwrt-zipit/staging_dir/target-arm_v5te_uClibc-0.9.33_eabi/usr/bin directory.  I fixed up the meta-makefile with that path.  Now the configure stage managed to find SDL and moved on... only to fail writing the new makefile.  Syntax error in the patched configure script!  Did I already say arrgh!?

Now, the original SDL patch from the internet required me to rerun autoconf to create a shiny new configure script (and Makefile.in) with SDL options.  I did that for my IZ2S build, but I have no idea how to do that in the openwrt environment.  Also, different versions of autoconf tend to generate a completely different output files.  And configure is over 600MB!  So I'd attempted to make a small patch for the configure script that would paste in only the changes involving SDL and skip all the gratuitous changes caused by the different version of autoconf. That got me all the way to the syntax error.  I took another look at my patch, and the modified configure script, but that's not work meant for humans.  Hand editing 600MB of butt ugly machine generated script file is not my cup of tea.  Even the mighty emacs with it's color syntax highlighting and electric parenthesis matching wasn't helping me deal with the horrendous tab formatting.  So instead I gave up and changed the openwrt meta-makefile to replace the whole 600MB configure script with the one I'd regenerated earlier with a different autoconf on a different PC.  Oh well.  It's kludgy, but it works.   

Not Fail!  Yeah!

The new openwrt Makefile, patches, some zipit friendly config files, and key docs are on github.

I put up the links-sdl_2.3pre1_pxa.ipk package for openwrt zipits, with config files and a kbd cheatsheet.

Here's my wrt-env.sh script (modified from slugs).

Update:   Anarsoul added PKG_FIXUP:=autoreconf to the Makefile to auto-magically regenerate the correct configure script as part of the openwrt build process.  And projectgus pulled it all into the master github project, so you should be able to get the links package from the usual place.  Cool!

Meanwhile, here's a qemacs executable for openwrt.  I still need to learn how make it into a real openwrt-zipit package...

Putting Links on a Diet

I got a bit ahead of myself (skipping a few stops on the roadmap) and compiled a bunjalloo executable for IZ2S.  It doesn't do anything but segfault right now, probably hitting one of those pesky uclibc bugs.  I'll have to shake that out one of these days, but I should probably stick to the original plan before attempting it.

However, I did discover something very interesting.  The size of the executable.  It came out to right around 250K upxed.  That's really, really small compared to the 2+MB upxed links executables that I've been building.  This got me thinking again about how to put links on a diet.  I know OpenSSL is significantly heavier than MatrixSSL.  But the built in font contributes most of the links bloat.  Bunjalloo gets by with only one size (small) font containing only ASCII characters as far as I know.  On the tiny zipit screen that's probably good enough.  If you think about it, the text mode of links does exactly that.

Now, I already knew the enormous font_include.c file was just a bunch of png files shoveled into C code.  But after some digging around I finally realized the graphics subdirectory contains the actual png glyph files and source to the generate_font program which created font_include.c.  Jackpot!  I pruned the glyph files down to just the basic Latin15 codes and some extra punctuation glyphs and then ran generate-font.  Next I rebuilt links with the new slimmer font_include.c and voila!  The upxed executable is almost megabyte lighter.

I tested it on some ebooks and a few websites.  So far It seems to have all the glyphs I typically run into, except for some unusual arrow characters on slashdot that weren't in the original font anyhow.  It's definitely a step up from retawq on the jffs.

I'm now positive I can get a upxed build of links (with graphics) slimmed down to less than a megabyte.  Some of the options I might try are:

•     Only one font (no monospace text)
•     No bold characters
•     ASCII only (no Latin15)
•     MatrixSSL instead of OpenSSL
•     No SSL.

Here's my first attempt, weighing in at about 1.6MB:  links-sdl-latinonly.zip

That should free up a whole bunch of space on the rockbox/links iz2jffs distro variant...

For the gmenu2x/gmu iz2jffs I save 200K right off the bat by using the existing jpeg, png, and libz shared libs required by gmenu2x instead of the static libs.  I can get another 150K by retiring retawq.  And I think removing the bold characters buys 200-300K.  If I rebuild gmenu2x without libfreetype (using the code from slug's github?) I think it all might just fit.  For now I just went only ASCII only, no bold, and no SSL for a 750K quick and dirty retawq replacement on the gmenu2x/gmu iz2jffs system.

links-iz2jffs-gmenu2x.tar.bz2

I'll try to put SSL and such back in after I see how the new gmenu2x build goes...

The Return of Zpub

While comparing the screenshots of glinks and bunjalloo, I was reminded of my old zpub post.  At the time I hadn't really dug into the links code very much, so I just sorta settled for a so-so reading experience.  I'd thought the epub text in glinks was somewhat small and ugly compared to netsurf.  But since netsurf requires the "mouse" I was a bit aprehensive about using it for an ereader.  I also remembered that dronz had tweaked his glinks config for ereading, so I decided to do a little experimentation with zpub and glinks.

The first change I made was to bump up the text size to something closer to the fonts used in a typical paperback book.  I'm pretty happy with an html-user-font-size setting of 18, but 16 is also not too shabby.  With the larger font, it makes sense to reduce the html-margin setting to 1 character to waste a minimal amount of space on the sides.  To bad you can't set it to 1 or 2 pixels.

While I was working on the margins I decided to see what I could do about that big fat ugly scroll bar.  The default G_SCROLL_BAR_WIDTH setting in setup.h is 12 pixels, which is extremely wasteful on the zipit.  Simply reducing it by 4 pixels gives me another full line of text with a font size of 18.

The other thing that really bugs me about glinks is the ugly default battleship gray background color.  You can change the colors of the menus, status bars, and even the scroll bar, but the html page background color must be set with in the document with a <body bgcolor="#FFFFDD"> tag or else you get the hard coded default_bg_g color from the default.c file.  I may try to code up an html-default-bgcolor command line and config file option to set this at runtime, but for now I just compiled in a yellow tinged off white color #FFFFDD that reminds me an old paperback book.

Another thing I noticed was the occasional odd characters in the text. Turns out the books I fetched from Project Gutenberg are utf8 formatted. I wouldn't be surprised if all epub documents are utf8.  Gotta review the specs someday.  Anyhow, I simply added the html-assume-codepage utf8 setting to the zpub script and now all is right with the text.  Check it out.

Remember how it used to look?  The new look is sooo much more warm and inviting, and actually displays even better on the zipit itself because the hardware rotated screen orientation washes out some of the yellow and possibly better matches my glinks subpixel anti-aliasing orientation setting.  Compare to this.

Gotta see if I can add that links command line option, then get the goodies posted and resume work on the bunjalloo code.

Update:  That turned out to be pretty easy, just a few ifdefs in the default.c file.  But now I've got a few more ideas.  I read about a third of the "Little Fuzzy" novel last night, and when I put down the zipit I realized the links bookmark system is probably inadequate for ereading.  I'd really like to save the scroll bar position so I can resume where I left off.  I'm gonna see if I can use the M and R keys to Memorize and Recall a scroll bar bookmark.  I might also look into defining the L and D keys in links to Lighten and Darken the LCD backlight for an adjustable day or night reading experience.  Either that or I'll have to setup the keymap so I can switch to another virtual terminal to adjust the lights and maybe check the battery level.

Update 2:  I finally finished reading my first full e-novel on the zipit and I can honestly say the experience was actually better than a paperback book in one way because I was able to pocket the zipit and take it with me.  I did find and fix a bug in zpub during my test read that briefly prevented me from viewing the 2nd half of the novel.  The M and R key scrollmark patch was fairly simple, and really made it easy to put down the zipit and pick it up again later to continue reading.  I decided against patching dimmer keys into links because that should be done at a system level so all apps can benefit.  I just need to work out which key combos make sense for all apps.

Here's the updated IZ2S zpub and links executables:  zpub2-iz2s.zip

Here's the modified links sources:  links-sdl-iz2s-zpub.tar.bz2
Here's a comprehensive patch for links-2.3 on the zipit:  links-2.3-zipit.patch

And here's a patch with just the ereading modifications.  http://pastebin.com/weJbzHWT

A jffs version will come later.  I need to make a zpub-lite that works without bash, and bc.  And I'll either have to provide a mini unzip executable, or build zpub-lite as an executable so I can use zlib. Maybe another mcb...

Moving Forwards

I finally posted the source code for the links browser with all my changes in the Friday the 13th blog entry.  Phew!  What a relief it is to get that done.

In the meantime, I've made some real progress on bunjalloo.  I read through all of the display code and consulted with some online docs about the NDS "main" and "sub" screens, their features, and operating modes.  Once I had it all marked up with comments, and thought I had a fair understanding of what was going on, I started making some changes to the code.

First I replaced all the occurrences of hard coded NDS screen sizes with SCREEN_WIDTH and SCREEN_HEIGHT.  Then I redefined those to match the Zipit screen size of 320x240, and tweaked the size of an opengl texture to accommodate the larger width.  This got me a double sized zipit screen display like this.

But I can only display the bottom "main" screen on the zipit so next I tweaked the SDLHandler.cpp code to use a single screen sized SetVideoMode() call, and tweaked the mouse button code to match the new screen size.  That gave me something like this, which should work on the zipit.

I really like the way bunjalloo can rotate the bottom toolbar around the screen to the right side as seen above.  It can also hide the toolbar so it doesn't obscure anything at all.  The scroll bar looks nice while using a minimal amount of screen real estate.  For comparison here's glinks displaying the same web site.  It's clearly more sophisticated than bunjalloo, but did they really have to make all the widgets so hideous?  Just look at that scroll bar.  It's the orthopedic shoe of scroll bar widgets.

I also double checked http:\\m.gmail.com to make sure I could still retrieve my gmail on the reduced zipit sized screen.  It's a bit slower than I'd like, but seems to work.

Unfortunately, in it's current form the bunjalloo code is still wasting considerable time and effort updating the video buffers for the "sub" screen even though it's no longer displayed.  This still needs work.  However, the (shared lib) bunjalloo executable is currently about 700K on x86 ubuntu (250K upxed).  That's significantly smaller than glinks, so I might just find a use for this when I'm all done.  Here's a roadmap of what I'm planning to do with it if all goes well.

• Compile without the WAF (python) build system.  Done
• Remove hardcoded NDS screen size numbers.  Done
• Change the SDL screen size to fit zipit screen.  Done
• Replace all opengl code (and unwanted libs) with one SDL Blit.  Done
• Remove unused SDL_mixer audio code from SDLHandler.cpp  Done
• Eliminate drawing to the emulated NDS upper "sub" screen.
• Remove (ifdef?) "sub" screen datastructs, leaving only "main" screen code.
• Fix SDL kbd code to use unicode field, and make it more responsive.
• Add support for ENTER, BS, ESC keys from real kbd on kbd gui.
• Add link navigation via tab, shift-tab, and enter keys instead of z2mouse.
• Consider changing arrow keys (and others) to match glinks keybindings.
• Create real makefiles in all source code directories.
• Build on Zipit.
• Optionally link bunjallo cache dir to /tmp/bunjalloo/cache for jffs.
• Try to build it without libstdc++ (to reduce exe size).
• Try other build options to reduce exe size for a better fit on the zipit jffs.

I've put the code onto my github site so you can see what's changed from the original sources.

Had another thought.  If I repackage my gmenu2x/gmu jffs distro to supplement retawq with bunjalloo I might get a better fit with a shared libstdc++ since I'll have 2 apps using it.  Something to think about...

The Long Way Around the Top

I'm not exactly satisfied with the ebook contents page produced by zpub, so I thought maybe I could make zpub build the html using the same algorithm as the original script.  But that uses the bc utility to assemble it's javascript and xml contents, and bc isn't included with IZ2S, so I figured I'd build it and see if it helped. 

At around the same time, I somehow got to wandering around on the nanonote wiki where I happened upon a shiny picture of htop running on the nanonote.  Ooh, shiny... Had to have it!  So I compiled it and tweaked the screen and keys a bit to better fit the Z2.  I love the way it lets you scroll to the right to see the entire command line.  Way better than the usual run-of-the-mill top utility!

Also around the same time I got to thinking about how I still really wanted a C compiler for IZ2S.  What's wrong with me?  That's just stupid.  But sometimes the only way to get these crazy ideas out of my head is to just go ahead and do it.  So I did.  Well... Ok, it's not really a compiler per se, but the PicoC interpreter is probably a better fit for IZ2S anyhow.  Since I had some previous trouble with floating point math support on the zipit I built two versions: one without floating point support, and one with it.

Unfortunately, before I got to test it all that much, I stumbled upon an SDL patch for the links browser.  What?  How did I miss that before?  With this patch I should be able to make a version of links for IZ2S that works properly, with or without the mouse, and without requiring the messed up DirectFB libs.  If it works, I can finally toss out the broken IZ2S DirectFB libs and remake them correctly from scratch whenever I get back to working on fltk, xdialog, vnc, or any of my backburner projects that need a normal working version of DirectFB.

Anyhow, I tweaked the links-sdl patch a bit for the IZ2S screen size, and for the current version of links.  I also added link traversal via TAB and BackTAB, and horizontal scrolling with the <> keys to make it more workable with zipit keyboard and the fake "mouse".  Initial testing feels good.  Gotta get it finished soon so I can patch bc back into the zpub script and see if it helps before something else pops into my head...

See?  The "mouse" actually works.

Look in the Zipit Z2 Goodie Bag for links to all the stuff.

Layers upon Layers.

One of my goals for the zipit is to get gtk running on Directfb so I can have all sorts of goodies built outta gtk; like fbreader, some OpenStreetmap stuff, and even xdialog.  The Directfb folks have a recipe with about 100 steps to do this, but I'm not so good at following directions, so I haven't got it done yet.  I did manage to compile all the various bits and pieces, supposedly in the right order too.  But all the gtk programs I've tried (even the tiniest of demos) crash when I try to run 'em.  Prolly something's not configured right in one of the many layers leading up to gtk.  So I decided to start testing things from the bottom layer up.  

The first new layer in the gtk food-chain is Directfb.  Now iz2s comes with a Directfb of sorts.  It provides the g for the graphical links browser.  But it has a few warts, sorta like the iz2s alsa libs.  Once again the libs are compiled to look for things in a weird, nonexistent path.  The iz2s glinks works around this by explicitly linking with the fbdev and keyboard modules in their unusual location instead of allowing the main Directfb lib to load them on demand from the normal places.  That's fine for glinks, but it interferes with my plans to build more stuff on directfb.  Now apparently each consecutive version of iz2s has a different build of glinks.  And fortunately it turns out one of 'em was compiled with the normal lib setup.  But it had issues with the keyboard.  Oh well.  Also, I discovered the hack used to rotate the glinks display is really just a window rotate, and not a screen rotate, so the mouse does not reorient itself as well to match the display.  Once again, fine for glinks running mouseless, but not so good for me and my nefarious plans.

Then it hit me.  I've already spent a considerable amount of  time patching up SDL to solve various screen, keyboard, and sound issues.  And Directfb can be built to run layered on top of SDL in addition to running directly on fbdev.  Maybe I can fix both layers with one set of patches.  So I rebuilt Directfb with both backends enabled and tried it.  The SDL backend fixed the keyboard issue, the mouse rotation problem, and even the blinking cursor bug.  Whoo Hoo!  It doesn't support double buffer mode, but neither does the the fbdev backend, unless you run it without the 90 degree CCW rotation required to match the screen orientation with the keyboard.

I'll probably have to patch SDL yet again to add the rotated double buffering.  The SDL screen rotation code uses what they call a "shadow" buffer to hold the pre-rotated pixels.  So it's already double buffered.  The only difference is the timing of the updates.  The shadow implementation sneaks in the updates when you're not looking, whereas regular double buffering does it on page flips.  I honestly don't know how the SDL folks missed that.

Meanwhile I think I might just try to build fbvnc so I got something useful to show for all this work, not just a bunch of demos and an irrelevant mouse pointer for glinks.  And maybe I'll even do a little sidetracking and build FLTK_Directfb and then murga-lua, just in case I never make all the way up past gtk to the xdialog layer.  Otherwise, next up is the glib layer.  I gotta find me an interesting program that'll put glib through its paces without requiring a full up gtk install...