Putting Links on a Diet

I got a bit ahead of myself (skipping a few stops on the roadmap) and compiled a bunjalloo executable for IZ2S.  It doesn't do anything but segfault right now, probably hitting one of those pesky uclibc bugs.  I'll have to shake that out one of these days, but I should probably stick to the original plan before attempting it.

However, I did discover something very interesting.  The size of the executable.  It came out to right around 250K upxed.  That's really, really small compared to the 2+MB upxed links executables that I've been building.  This got me thinking again about how to put links on a diet.  I know OpenSSL is significantly heavier than MatrixSSL.  But the built in font contributes most of the links bloat.  Bunjalloo gets by with only one size (small) font containing only ASCII characters as far as I know.  On the tiny zipit screen that's probably good enough.  If you think about it, the text mode of links does exactly that.

Now, I already knew the enormous font_include.c file was just a bunch of png files shoveled into C code.  But after some digging around I finally realized the graphics subdirectory contains the actual png glyph files and source to the generate_font program which created font_include.c.  Jackpot!  I pruned the glyph files down to just the basic Latin15 codes and some extra punctuation glyphs and then ran generate-font.  Next I rebuilt links with the new slimmer font_include.c and voila!  The upxed executable is almost megabyte lighter.

I tested it on some ebooks and a few websites.  So far It seems to have all the glyphs I typically run into, except for some unusual arrow characters on slashdot that weren't in the original font anyhow.  It's definitely a step up from retawq on the jffs.

I'm now positive I can get a upxed build of links (with graphics) slimmed down to less than a megabyte.  Some of the options I might try are:

•     Only one font (no monospace text)
•     No bold characters
•     ASCII only (no Latin15)
•     MatrixSSL instead of OpenSSL
•     No SSL.

Here's my first attempt, weighing in at about 1.6MB:  links-sdl-latinonly.zip

That should free up a whole bunch of space on the rockbox/links iz2jffs distro variant...

For the gmenu2x/gmu iz2jffs I save 200K right off the bat by using the existing jpeg, png, and libz shared libs required by gmenu2x instead of the static libs.  I can get another 150K by retiring retawq.  And I think removing the bold characters buys 200-300K.  If I rebuild gmenu2x without libfreetype (using the code from slug's github?) I think it all might just fit.  For now I just went only ASCII only, no bold, and no SSL for a 750K quick and dirty retawq replacement on the gmenu2x/gmu iz2jffs system.

links-iz2jffs-gmenu2x.tar.bz2

I'll try to put SSL and such back in after I see how the new gmenu2x build goes...

The Return of Zpub

While comparing the screenshots of glinks and bunjalloo, I was reminded of my old zpub post.  At the time I hadn't really dug into the links code very much, so I just sorta settled for a so-so reading experience.  I'd thought the epub text in glinks was somewhat small and ugly compared to netsurf.  But since netsurf requires the "mouse" I was a bit aprehensive about using it for an ereader.  I also remembered that dronz had tweaked his glinks config for ereading, so I decided to do a little experimentation with zpub and glinks.

The first change I made was to bump up the text size to something closer to the fonts used in a typical paperback book.  I'm pretty happy with an html-user-font-size setting of 18, but 16 is also not too shabby.  With the larger font, it makes sense to reduce the html-margin setting to 1 character to waste a minimal amount of space on the sides.  To bad you can't set it to 1 or 2 pixels.

While I was working on the margins I decided to see what I could do about that big fat ugly scroll bar.  The default G_SCROLL_BAR_WIDTH setting in setup.h is 12 pixels, which is extremely wasteful on the zipit.  Simply reducing it by 4 pixels gives me another full line of text with a font size of 18.

The other thing that really bugs me about glinks is the ugly default battleship gray background color.  You can change the colors of the menus, status bars, and even the scroll bar, but the html page background color must be set with in the document with a <body bgcolor="#FFFFDD"> tag or else you get the hard coded default_bg_g color from the default.c file.  I may try to code up an html-default-bgcolor command line and config file option to set this at runtime, but for now I just compiled in a yellow tinged off white color #FFFFDD that reminds me an old paperback book.

Another thing I noticed was the occasional odd characters in the text. Turns out the books I fetched from Project Gutenberg are utf8 formatted. I wouldn't be surprised if all epub documents are utf8.  Gotta review the specs someday.  Anyhow, I simply added the html-assume-codepage utf8 setting to the zpub script and now all is right with the text.  Check it out.

Remember how it used to look?  The new look is sooo much more warm and inviting, and actually displays even better on the zipit itself because the hardware rotated screen orientation washes out some of the yellow and possibly better matches my glinks subpixel anti-aliasing orientation setting.  Compare to this.

Gotta see if I can add that links command line option, then get the goodies posted and resume work on the bunjalloo code.

Update:  That turned out to be pretty easy, just a few ifdefs in the default.c file.  But now I've got a few more ideas.  I read about a third of the "Little Fuzzy" novel last night, and when I put down the zipit I realized the links bookmark system is probably inadequate for ereading.  I'd really like to save the scroll bar position so I can resume where I left off.  I'm gonna see if I can use the M and R keys to Memorize and Recall a scroll bar bookmark.  I might also look into defining the L and D keys in links to Lighten and Darken the LCD backlight for an adjustable day or night reading experience.  Either that or I'll have to setup the keymap so I can switch to another virtual terminal to adjust the lights and maybe check the battery level.

Update 2:  I finally finished reading my first full e-novel on the zipit and I can honestly say the experience was actually better than a paperback book in one way because I was able to pocket the zipit and take it with me.  I did find and fix a bug in zpub during my test read that briefly prevented me from viewing the 2nd half of the novel.  The M and R key scrollmark patch was fairly simple, and really made it easy to put down the zipit and pick it up again later to continue reading.  I decided against patching dimmer keys into links because that should be done at a system level so all apps can benefit.  I just need to work out which key combos make sense for all apps.

Here's the updated IZ2S zpub and links executables:  zpub2-iz2s.zip

Here's the modified links sources:  links-sdl-iz2s-zpub.tar.bz2
Here's a comprehensive patch for links-2.3 on the zipit:  links-2.3-zipit.patch

And here's a patch with just the ereading modifications.  http://pastebin.com/weJbzHWT

A jffs version will come later.  I need to make a zpub-lite that works without bash, and bc.  And I'll either have to provide a mini unzip executable, or build zpub-lite as an executable so I can use zlib. Maybe another mcb...

Moving Forwards

I finally posted the source code for the links browser with all my changes in the Friday the 13th blog entry.  Phew!  What a relief it is to get that done.

In the meantime, I've made some real progress on bunjalloo.  I read through all of the display code and consulted with some online docs about the NDS "main" and "sub" screens, their features, and operating modes.  Once I had it all marked up with comments, and thought I had a fair understanding of what was going on, I started making some changes to the code.

First I replaced all the occurrences of hard coded NDS screen sizes with SCREEN_WIDTH and SCREEN_HEIGHT.  Then I redefined those to match the Zipit screen size of 320x240, and tweaked the size of an opengl texture to accommodate the larger width.  This got me a double sized zipit screen display like this.

But I can only display the bottom "main" screen on the zipit so next I tweaked the SDLHandler.cpp code to use a single screen sized SetVideoMode() call, and tweaked the mouse button code to match the new screen size.  That gave me something like this, which should work on the zipit.

I really like the way bunjalloo can rotate the bottom toolbar around the screen to the right side as seen above.  It can also hide the toolbar so it doesn't obscure anything at all.  The scroll bar looks nice while using a minimal amount of screen real estate.  For comparison here's glinks displaying the same web site.  It's clearly more sophisticated than bunjalloo, but did they really have to make all the widgets so hideous?  Just look at that scroll bar.  It's the orthopedic shoe of scroll bar widgets.

I also double checked http:\\m.gmail.com to make sure I could still retrieve my gmail on the reduced zipit sized screen.  It's a bit slower than I'd like, but seems to work.

Unfortunately, in it's current form the bunjalloo code is still wasting considerable time and effort updating the video buffers for the "sub" screen even though it's no longer displayed.  This still needs work.  However, the (shared lib) bunjalloo executable is currently about 700K on x86 ubuntu (250K upxed).  That's significantly smaller than glinks, so I might just find a use for this when I'm all done.  Here's a roadmap of what I'm planning to do with it if all goes well.

• Compile without the WAF (python) build system.  Done
• Remove hardcoded NDS screen size numbers.  Done
• Change the SDL screen size to fit zipit screen.  Done
• Replace all opengl code (and unwanted libs) with one SDL Blit.  Done
• Remove unused SDL_mixer audio code from SDLHandler.cpp  Done
• Eliminate drawing to the emulated NDS upper "sub" screen.
• Remove (ifdef?) "sub" screen datastructs, leaving only "main" screen code.
• Fix SDL kbd code to use unicode field, and make it more responsive.
• Add support for ENTER, BS, ESC keys from real kbd on kbd gui.
• Add link navigation via tab, shift-tab, and enter keys instead of z2mouse.
• Consider changing arrow keys (and others) to match glinks keybindings.
• Create real makefiles in all source code directories.
• Build on Zipit.
• Optionally link bunjallo cache dir to /tmp/bunjalloo/cache for jffs.
• Try to build it without libstdc++ (to reduce exe size).
• Try other build options to reduce exe size for a better fit on the zipit jffs.

I've put the code onto my github site so you can see what's changed from the original sources.

Had another thought.  If I repackage my gmenu2x/gmu jffs distro to supplement retawq with bunjalloo I might get a better fit with a shared libstdc++ since I'll have 2 apps using it.  Something to think about...

I Don't Know What I Was Thinking.

Boy, that'll teach me to make promises on Friday the 13th, just before taxes are due.  The taxes got done, but not much else.  And now I'm really having trouble remembering where I stashed all the zipit goodies I promised to make available.  I swear I'll get them posted, real soon now...

In the meantime, whilst avoiding packaging up the old goodies, I somehow found myself on an old Nintendo DS homebrew site, which reminded me of the spiffy Bunjalloo web browser.  It runs quite well (with graphics even) in the limited 4MB and small screen real estate of the NDS.  Plus there's some notes about building it for linux with SDL, and it uses the tiny matrixssl library for shttp connections.  I've been looking for an excuse to use that one, but I'm far too lazy to graft it onto one of the other zipit browsers.  So off I went, merrily avoiding what I should be doing, and attempting to port bunjalloo instead.  Turns out it uses some (python based?) WAF build system that I've never heard of, and a bunch of stuff from the NDS devkitarm toolchain.  Well, python and WAF were quite unhappy with my scratchbox build environment, so I quickly ditched them and moved to script based compiling.  I had to resort to Windows to use the NDS oriented grit tool to convert some png files to .c and .h source code because I'm too lazy to pull down all it's requirements and build a linux version.

Then, just as I was reaching 100% of the sources compiled I ran up against a wall.  Oops, the linux build requires opengl.  Apparently some 2D opengl calls match up well against the native NDS hardware, so an opengl layer makes sense for portability.  But not really for my purposes.  Opengl is not so great for the zipit, with it's conspicuous lack of floating point hardware.  However on closer inspection, it looked like most of the opengl calls were using the integer versions of the functions.  Hmm, that got me thinking.  Apparently I still have a few brain cells left over from olden times, when DSLinux wasn't quite dead yet.  Just before the DSI came out and flatlined the last remaining DSLinux effort, I managed to sneak in a really lame port of the opengl version of the classic Elite space game.  I dug through the archives and sure enough, there was the integer only tiny-SDL-OpenGL lib I used for the port.  I think it needs a few tweaks to make it suitable for bunjalloo, and then there's the whole business of reconfiguring the screen and keyboard code, but the wall has been breached. 

Hopefully bunjalloo doesn't animate or use 100% of the CPU in a busy loop, because the opengl example programs are not exactly top performers.  Here's the well known gears program grinding out a whopping 2 frames per second on the zipit.

The shiny spinning triangle isn't a whole lot snappier.

Hopefully I can continue to restrain myself and not try to port any of my old opengl based CAD programs to the zipit.

 Although this does look almost useable in 320x240 pixels...

Must, resist, temptation...

For your  amusement, here's a zip with the tinygl sources.  The gears and triangle executables are also included in the examples directory. 
tinygl-iz2s.zip

And here's what bunjalloo currently looks like running in the debugger on ubuntu.

As you can see, I have plenty of work to do on the screen layout before I can even attempt to run it on the zipit.

Update:  I dug through the code, looking for all numbers near 256 and 192 (the NDS screen dimensions), and replaced all the 192s with 120 to use a half zipit screen instead of one of the NDS screens.  It sorta works but the scrollbar looks weird constrained to the bottom half of the display.  So I think I really want to make it use only one 240 pixel high screen instead.

I was able to confirm that I can use bunjalloo to access my gmail account (cool!), however hotmail seems to really want javascript.

Latin Lessons (and Other Things)

Wow, it's been over a month since I did any sort of real blog post.  This must be what writers block feels like.  I've been off tinkering with so many different things that it's difficult to put enough coherent thoughts together for a decent post.  After I put up my previous "Where's the Beef?" post, I went back and edited the dead links in all my old posts to redirect them to the "Where's the Beef?" message.  It's quite a relief to be done with that.  But that's just clean up.  Nothing new.

So why am I so scatterbrained?  Well, I'm currently lugging 5 zipits around, all with different stuff on the internal flash.  In addition, I've got a pocket full of micro/mini SD cards with various versions of IZ2S, Flashstock, openwrt, etc.  And the pocket is really rough on the labels I've made for them.  They keep fading or falling off, so I can no longer remember what's on what card.  At least some of the zipit's are still labelled, and I'm testing some of the official zipit skin stickers to see if they're more resiliant, or easier to remember.

Zipit 1.  I finally loaded Mozzwalds openwrt rescue onto a virgin zipit, as a prelude to loading slug's shiny looking gmenu/gmu rescue image.  But now this particular zipit no longer likes any of my SD cards, so I'm back to being disappointed with uboot again.  I could probably leave it at home in a drawer somewhere, but I keep hoping I'll come up with a brilliant plan to restore the blob bootloader and the stock kernel via the wifi.

Zipit 2.  After I gave up on zipit 1, I decided to update a zipit from Mozzwalds initramfs uboot rescue image to slugs gmenu/gmu image.  The initramfs was functional, but limited due to the high memory use of the filesystem in RAM.  This zipit seems to have less trouble with the SD cards, even though it has the exact same uboot image.  Weird.  Anyhow, aside from my general misgivings about uboot, this one looks quite promising.  However testing revealed a problem where some component of gmu (sdl, alsa, pthreads, nanosleep, or whatever) is causing it to use 70% of the CPU instead of the expected 17% or so.  I did a test build of rockbox for openwrt a while back which also performed poorly.  I wonder if it had the same problem...

Zipit 3.  I've been updating my gmenu/gmu image for the blob bootloader and the stock kernel.  It's not as pretty as slugs, but it's still fun to work on.  I'm currently fiddling with the wifi setup scripts in an effort to add options for multiple profiles.  I've also started poking through slugs gmenu2x updates to see how hard it would be to backport the wifi setup dialogs and other changes.  I've also done some work on running a 2nd tty so I can have gmu in the background and a clock on the screen, but running in another tty.  Both ttyclock and dgclock seem to function with gmu in the background.  However this required me to map the home key as F1 so I could enter the magic ctrl-alt-F1 sequence recognized by SDL programs to escape to tty1.  I created a t2 script to launch a new tty and a cls script to rotate the tty, turn on the cursor, and clear the screen, just in case it gets hosed somehow.  I also added a deallocvt utility to the lcdval mcb so I can clean up abandoned ttys.

Zipit 4.  I've been tinkering with the rockbox/glinks image that dronz whipped up.  I built a smaller partial static glinks for sdl and added latin1 keyboard support so now there's a little bit of room to grow.  Here it is with a totally fictional foreign URL. 

I also built a slightly smaller mpg123 in case I want to grow this jffs into something with internet radio support.  And I've been tinkering with the keymap because I really want to standardize on one sticky latin15 keymap and one non-stick keymap for all my IZ2S derivatives.  I still need to do something about the other Fn keys though.

Zipit 5.  This has my original jffs base image and plenty of free space for testing things while I attempt to shrink them down.  For example, I used this one to test a latin1 build of pspmaps.  I recycled the latin1 kbd routines from glinks and made pspmaps accept latin1 locations.  Then I discovered RFC 3986 which says you now need to convert URLs to utf-8 before percent encoding and assembling your http GET request.  So I finally got my foot in the door a utf-8 app (well half utf-8 anyhow).  My son just went overseas with his classmates so I offered the zipit, just in case he needed to use it for directions.  He took his ipod instead.  C'est la vie...

With the new pspmaps you can travel from Paris to Mâcon, without crossing the Atlantic to wind up in Macon, GA.

I also used htop on zipit 5 to do some baseline performance testing of gmu and mpg123 on IZ2S based systems.  Mpg123 seems to use about 13% to 18% of the CPU to play and gmu about 2% more.  While testing I discovered that I need a .bashrc file in the home directory on the jffs to set the terminfo path if I want to run curses programs like htop in an ssh session.  Gotta add that to the next revision of all the IZ2S based jffs systems.

Finally, I'm currently investigating toybox and it's oneit utility to see if it's more efficient than own-tty or this busybox sh trickery.  Toybox is a tiny busybox alternative with some interesting features for my jffs images.  It's fairly compact and comes with a nice set of utilities that complement the stock busybox without all the overhead of the full busybox replacement from IZ2S.  I could use it to replace nc, killall, tee, and a few other standalone programs that I've built and get quite a bit more for just a few K.  Since toybox comes with a mini bzcat, I figured I might as well look for mini versions of zip and gzip to round out the compression options.  I found some, but more testing is needed.

There will be goodies when I track down where I posted them on the #zipit IRC channel...

links-latin-sdl-iz2s.zip
Here's the source, finally:  lnks-sdl-iz2s-latin-src.tar.bz2
Here's a comprehensive patch for links-2.3 on the zipit:  links-2.3-zipit.patch

pspmaps-utf8latin-iz2s.zip
pspmaps-utf8latin-iz2s-src.zip

mpg123-iz2jffs.zip

Here's toybox, deallocvt, and related scripts for using multiple vts (as shown below).
toybox-iz2jffs.zip

Source for the new lcdval mcb with deallocvt:  ez2sutils-src.zip

For some reason I felt compelled to compile the lynx browser.  lynx-iz2s.zip

Where's the Beef?

The goodies you seek are here. and/or here.  Read on if you want to know why.

But first, a note about the goodies.  Much of it is intended for either iz2s or iz2jffs.  They're basically the same thing except iz2jffs is much smaller and resides on the internal   flash, whereas iz2s is installed on an SD card.  Both are hacked up systems that make use of the stock kernel that comes pre-installed on the zipit.  It's an ancient kernel, but I like it for my purposes.  Now since the entire thing is hacked together there's no real package management system for beginners.  Sorry about that.  You might just have to learn your way around the linux file system if you want to get a lot out of this.

Anyhow, if you want to use lots of this stuff, then your best bet is to  boot from an iz2s SD card.  Otherwise, if you know what you're doing, you can use all of the goodies with an iz2jffs setup, but it takes a little extra work to get it to recognize the libraries and paths used by the stuff.  The addz goodie pack does this for you for some of the older goodies that can't possibly fit on the jffs.  You'd have to do something similar to the addz pack to adapt the newer goodies to work with the jffs.  I may do it someday, but right now I'm booting from the SD card most of the time.

And now for the story of the goodie bag...

It's been said that the internet giveth and the internet taketh away.  And sometimes that's true.  It's also been said that the internet never forgets, but that's another story.  This one is about give and take.  Previously my cable provider of 10 years decided to eliminate their crappy (but free!) web hosting service because absolutely nobody was using it, except me.  So the Zipit Goodie Bag had to pack up and move.  Rkdavis graciously provided a new home for it, but it's fallen into a bit of disrepair lately.  I suspect he's taken in too much raspberry pi vapor, and will be back when the buzz wears off.

In the meantime I've been keeping things patched together by hosting new goodies at datafilehost.com.  But already some of the links have gone dead due to lack of interest.  Imagine that!  My 40MB IZ2S emacs package has gone missing due to lack of downloads.  Don't you people realize the awsome power of a fully operational emacs?  Even my own son has turned against me, recently declaring allegiance to vi instead of emacs.  I've seen this movie before and let me just say, it doesn't end well for the zipit.

But now Dronz has provided the Zipit Goodie Bag with a new summer home in France.  So it's time for some spring cleaning.  I'm planning to prune all the dead links in the blog and redirect them to this post, where I still have the editing power to keep things up to date.  From now on, this post will tell you to where all the goodies actually reside.

Here's the Zipit Goodie Bag Summer Home, including the new jffs goodies.
And here's the old tried and true (but in need of an update) Zipit Goodie Bag.

In the Zone

I decided to use my renewed focus on the jffs to try and see if I could squeeze gmenu2x onto the stock zipit jffs along with gmu. (I've already fielded a few not so subtle hints from my co-worker that the linux command line is perhaps not the easiest way to use the zipit.)  This wasn't gonna be easy though because gmenu2x is fairly bloated with C++, freetype, and lots of png files.

The first thing I did was to cut it back to a minimal installation with just the default skin, only one translation (french), and fewer icons and the smallest background image file I could find.  See above.  I also ran all the images through pngcrush for good measure.  This resulted in a gmenu2x installation that was almost small enough for the jffs.  Unfortunately it didn't quite work when run from the stock busybox shell.  I had to build a tiny killall executable to enable my previous uclibc workaround from the iz2s build.  Then I had to make sure to "exec gmenu2x" from my gmenu wrapper script or it would complain about various tty ailments whenever it finished running an app.  Weirdness.

Next I attempted to link gmenu2x without libstdc++ according to these instructions.   But gmenu2x uses STL strings, dynamic casts, and more from libstdc++ so the best I could do was skip exceptions and static link with libstdc++.a.  The shared lib is about 800k stripped, and would would use about 300K compressed on the jffs, so I managed to squeeze a little bit out of it.  I recently discovered the jffs seems to garbage collect on boot up, which means I may have a reliable way to compare the space usage.  So I'll probably try to install the full lib someday to see exactly how much I saved.  Actually, I suspect I'll hold off on that until I find another C++ program that I just gotta have...

I also tried static linking libfreetype, but that didn't save anything.  A stripped libfreetype is 400K, or about 200K compressed on the jffs, so I bit the bullet and stuck it on there.  This meant I could also toss in the dgclock program for only 20K if could trick it to use the same font as gmenu2x.  So I moved the /mnt/ffs/gmenu2x/skins/Default/font.ttf file to /mnt/share/fonts/font.ttf and replaced it in it's original location with a soft link.  Then I soft linked the mplus-2p-medium.ttf file used by dgclock to font.ttf and voila!  The dgclock works.  The gmenu2x font file is fairly compact but I can barely tell the difference in the dgclock program.  It might have slightly narrower glyphs.  Maybe it'll be good enough for a few more small SDL-ttf programs.

This fit onto the zipit, but just barely, so I swapped out the wpa-supplicant and wpa-passphrase for my cut down versions.  The ones from the V1 firmware (and IZ2S) are fairly large, and the ancient wpa-supplicant was having troubles connecting reliably to WPA2 routers anyhow.  The new smaller version actually seems to work better.  This got me to about 700K free on the jffs.  That's enough room for some more small goodies.

While working with dgclock, I noticed the zipit appeared to be stuck in a different timezone (somewhere in europe?) instead of EST5EDT.  I'm not sure what caused it, but eventually I deleted the /mnt/ffs/TZ file and rebooted.  The timezone setting script popped up.  I entered EST5EDT, and it recovered.  Then I realized I was off by an hour because the 2007 daylight savings rules weren't incorporated into the iz2s uclibc.  Fortunately Dom pointed out the uclib TimeZone fix for the US 2007 daylight savings change in one of the comments here.    Down at the bottom it mentions the magic EST5EDT,M3.2.0,M11.1.0 setting for the /mnt/ffs/TZ file.  EST5EDT,M3.2.0,M11.1.0 is US Eastern Time with Daylight Savings Time starting on the 3rd month (March), 2nd week, day 0 (Sunday) and continuing until the 11th month (November), 1st week, day 0 (Sunday).

Here's a combo jffs image containing the base, gmu and gmenu2x packages to make it an easier install for newbs.  Note:  I said "easier", not "easy".

gmu-gmenu2x-jffs.tgz   Here's the install instructions.

And apparently the Goodie Bag was offline for a while.  Sorry.  I think I'll try and mirror it...