Anyhow, building with legos always makes me think about LDRAW, and so I paid a visit to the ldraw.org forum. There were all sorts of new projects under development there, but one in particular caught my eye. LPub3D is a new version of LPub that appears to incorporate leocad into the right side panel, I assume to assist in positioning the lego models live, as you're preparing the building instructions for publishing. The latest release announcement mentioned some updates to the ldglite code, which is used as the quick and dirty low end rendering option.
What's that? New ldglite code? And there were bug reports too, complaining about the ldglite image quality and inaccuracies compared to LDView. Wow. My ldglite sourceforge repository's been mostly inactive for like 4 years now. I had to get involved in this before people lost interest and moved on. Sorry zipit gps, we'll get back to you later...
So I requested a copy of the ldglite code updates, dug an old pentium 3 Windows laptop out of cold storage (the garbage) and got to work installing ldraw programs and reconnecting with my sourceforge archives. The new code adds support for more ldraw search directories, where I'd fallen far behind in support of the current LDRAW standards. So I merged it into the CVS repository. The new stuff pulls in some C++ code from LPub3D and adds it to the l3 parsing code, so I ended up splitting the l3 code into a C version and a C++ version That way I can still build things with just my C compiler. Eventually I'll try to duplicate the directory search code in C because I want it too.
Then I got to work on the bug reports. I tried to duplicate some of the problems in LPub3D with the car.dat example model that comes free with the part files. Here's what I did.
First open the example car.dat model and go to the last step (step 8):
Then configure the preferred renderer in lpub3d preferences to LDView.
Do an alt-printscreen and paste into an mspaint window.
LDView looks pretty good.
Repeat configured for ldglite rendering and paste into the same mspaint window.
Then toggle back and forth with control-Z and control-Y to see the differences.
Ldglite is clearly the low quality image winner. Darn it.
It looks like the same viewing angle, but the ldglite part images are slightly bigger. The size difference is quite subtle. It could be just different edge line width options, but I suspect it's a little more than that.
Now take a look at Step 3 of the instructions. This shows how subtle differences in the part images can lead to a different layout of the parts inset box.
Here's step 3 using LDView.
And here's step 3 with ldglite. The parts inset box is different.
You might have noticed the lighting is also different. Both LDView and ldglite have some ambient light and a spotlight, but ldglite defaults to placing the spotlight at (x,y,z) = (-1000,1000,1000), probably for compatibility with the original LDRAW.EXE program, or the old LDLite.exe viewer. The default places the spotlight above the viewer and off to the left. Adding -lc0,1000,1000 on the command line makes it more like LDView default lighting which is above and directly behind the viewer. But I didn't see anywhere that I could set this in LPub3D. I thought I remembered a place in the old LPub for extra settings, once upon a time. That'd make it much easier to test things.
Meanwhile, I could see where LPub3D keeps its work in the ldraw models directory. It'd be nice if it had a debug setting to keep a running log there with all of the commands it sends to the external rendering programs. However, the source code patch came with an example ldglite command line, so at least I had that to work with. For rendering, LPpub3D spins the parts (or the model) before calling ldglite with command line settings to view it from really far away with a small view angle in order to simulate an orthographic view with a perspective view matrix. My notes in the ldglite code suggest this is because the POV ray tracer doesn't, or didn't, support orthographic projections. LDView might not either. Examining the render.cpp code in LPub3D appears to confirm this.
Anyhow, I made a model with just one red 4x4 plate to see what differences other than the light source I could spot between LDView and ldglite.
The plate in LDView
The plate in ldglite
The plate in l3p/POV
I pasted the LDView red plate image over the ldglite image and zoomed in.
I think it's pretty clear that ldglite is being told to render on a white background with thicker anti-aliased edge lines. You can't readily tell if it's a transparent png in mspaint, but viewing them here shows both images have transparency enabled for the background. The edge line widths could add up to a pixel or 3 difference in size, but we see more than that, and it's proportional to the shape of the box, so it's a slightly different view. It could simply be that ldglite uses the origin point (0,0,0) in the model as the look at point and LDView looks towards the center of the bounding sphere. I'll have to experiment with this to get a better match in the generated images.
For now, I attempted to fix some of the easy stuff. And this is where burgerspace plays a part. The readme.txt file for ldglite tells how to scale up by 2 and dump an image that can be post processed by another program to downsample with antialiasing for a more pleasing appearance. Once upon a time the ldraw parts tracker did just that. But fresh from my success with burgerspace, I figured I had everything I needed to add a simple average and decimate by 2 output filter right into the ldglite code. So I did, and ran it through LPub3D -- with hard coded ldglite settings for now -- to see how it stacked up. It's still not LDView quality, but I think it's a vast improvement. Perhaps medium-low quality.
Here's step 8 with the 2x AA Filter in ldglite. I also moved the light source.
Here's step 3 in the improved ldglite. The inset is more like LDView now.
Not too shabby. The code is in CVS on the sourceforge, but needs some cleanup. I'm thinking of trying a Gaussian blur filter, possibly with gamma correction, and maybe 3x oversampling and decimation. Why not?
Meanwhile I did some math on the look at point and I don't see how it makes much of a difference if the origin is on the corner of the plate or in the center at the great distance we're viewing from. Here's the setup used by LPub3d. What is that, one light year off in the Z direction?
-ca0.01 -cg0.0,0.0,3116890
The camera viewing (FOV?) angle is a teensy 0.01 (degrees?) so a tiny change to that number might be enough to create the difference in the plate images. Perhaps I'm using floats instead of double precision somewhere, leading to larger noticable errors in the calculations. More investigation is needed. I hope LDView keeps a log of the commands it executes. Then I can experiment directly in ldglite and LDView without even loading LPub3D. That'd make this whole process go more quickly.
Anyhow, after looking at the inconsistent treatment of edge lines that fell evenly on the 2x2 pixel samples versus the ones that straddled the 2x2s, I decided the gaussian blur filter was a must. The quick and dirty 2x2 was good enough for burgerspace, but this is important. Appearance actually counts. So I coded it up and replaced the 2x2 filter.
Here's a 2x scaled image of the car (-S2 -w1). No downsampling, plenty of jaggies.
Here's the same car scaled up by 4 (-s4 -w2), then decimated by 2 with the blur filter.
Overall the quality of the blur filtered image is better, and more consistent around the edge lines than it was with the 2x2 averaging filter. There are still some issues with the overall image quality from ldglite. You can see the blended edge lines on the black surfaces appear darker in the blur filtered image. I'm gonna need to gamma correct before the blur and then reverse the gamma afterwards to make those grey on black edges pop. The mtPaint Handbook suggests I should read "ITU-R recommendation 709" and implement that. Ok, will do.
Here's the same thing but with 8/12 bit lookup table gamma corrected blurring.
Hmm, the gamma correction seems to have had more of an impact on the transparent parts than the edge lines. I'm not sure if I like it. Something seems fishy here. Look at the faded colors of the head and tail lights where they sit over the background color. For a while I thought I might have to gamma correct the alpha channel, but then I rendered one with with a black background and realized the problem. The transparent parts that appear over the background color are blended with the alpha channel and the background color, which makes them too white if the background is white. You can see the same effect in the grey edge lines that border the background. The image must be rendered on a black background for best results with a transparent png. Yeah, that's what I was trying to remember from years ago. You can use either -b0x2000000 to set the background to absolute black, or use -b0 to go with the very dark ldraw color number zero.
I rendered one on a black background, and also moved the lighting to head on.
Anyhow, about those transparent parts... Transparent parts are handled quite differently by ldglite than in LDView or POVray. I imagine the POV ray tracer takes on a hint of the translucent color whenever a ray is cast through a translucent object. So multiple passes through the glass will add more of the glass color. LDView probably (but I'm not positive about this) applies a hint of the translucent color for each surface of of translucent object between the viewer and the background. So you take a hit from both the front and backside of the glass. Ldglite simply applies one stipple mask for any amount of translucent stuff between the viewer and the background. Zoom in on the non-downsampled ldglite images and you can see the stipple pattern. Downsampling blends the stipple pattern into whatever happens to lurk behind it. This makes it pretty, but you still only get one hit of glass color -- from the glass nearest to you, the viewer -- no matter how many translucent things you're looking through. I'm actually ok with this approach even though it's less realistic, because it lets you see better what's behind the glass. Some might even say that makes for easier to follow assembly instructions. And it's sorta retro, like me. See?
I've quantified the difference in scale between LDView and ldglite. Adjusting the camera angle setting from -ca0.01 to -ca0.01021 is enough to make the plate images match up with the LDView plate image. I'm still not sure where the error creeps in, but I have my suspicions...
Size up this filtered ldglite image (with front lighting and -ca0.01021) to the others above.
Unfortunately the suspicions didn't pan out. I currently have no idea what's different, or for that matter, which program is at fault. I was hoping to compare both ldglite and LDView with some l3p/POV renders, but POV runs out of memory and fails on my junky old Windows laptop. 384 Megabytes sure as heck don't go as far as they once did. Maybe there's some lower quality reduced memory settings I could use, but I'll have to read up some POV documentation to figure that out.
In the meantime I double checked the -ca0.01021 ldglite setting with the car model instead of just the plate. The wider camera angle worked the same magic on the car, so at least the error is consistent. Click on these images and compare the sizes yourself. It's a pretty good match.
This car image was produced by LDView in LPub3
This car image was produced by ldglite (with hard coded settings) in LPub3.
I probably should add a note at this point that I've done nothing to the LDView settings since I started testing, simply because I'm more interested in improving the abysmal quality of the ldglite pictures. So I'm certain these LDView images are not even close to representative of what it does with the proper settings selected for LPub.
Update February 28 2016:
Real life stepped in and forced me to back away from this for a bit. So when I got back to it I started up on a different tack. Examining the LPub3D render.cpp code I can see that all of the rendering options appear to use the exact same command line camera angle setting. But the LDView camera distance setup code has a bunch of fudge factors in it. First it multiplies the scale by 0.775 inside the LDView::cameraDistance function, and then whenever it uses this function it multiplies again by 1700/1000. What is that?!? Some sort of aspect ratio adjustment, or is it really two different fudges? The cameraDistance function is only used in render.cpp so there's absolutely no good reason to fudge it in two different spots. It must be some kind of subterfuge because there's also a hideously complex table of ratios in the comments right before the LDView::cameraDistance function that I cannot make heads or tails of. I'd swear it's just put there as a distraction. But if I had to make a bet, I'd say this is where the ldglite and LDView rendering options diverge. Perhaps we need to add tiny nudge to the final fudge factor to make it all good?
I had to know. So I loaded all the programs on a newer Windows laptop that I've got with more memory, but with a busted fan unit. I let it rest on an ice pack fresh from the freezer and did a POV render, just hoping to finish rendering the plate image before the CPU burned up. The laptop fan squealed an agonizing screech of death, but POV did whatever it does without a popup error message. Nothing appeared in the LPub3D window (possibly a setup issue that I don't know about) but I rummaged around in the ldraw/models/LPub3D directories and found the image. Now I can compare the output image sizes of all 3 rendering options and figure out where to go next.
The ray traced plate image from l3p/POV is pretty, but I find all the shadows distracting.
I had to clip the POV output image to compare it to the others. I also added a picture of it up higher in the blog to make it easier to compare plates in the image viewer here, but I think it shows ldglite and l3p/POV are using the same view setup. And once again the LDView rendered plate is smaller.
I pasted the LDView red plate image over the l3p/POV image and zoomed in.
So what's the story with the fudge factors wrapped around the LDView camera distance calculations? I suspect they might just be cruft left over from a work around developed for an earlier version of LDView that perhaps only provided the -FOV command line option instead of the -ca option used by l3p and ldglite. Or maybe the LDView -ca option is only sorta compatible. But the truth is I don't really know, and that scary table of ratios in the LPub3D source code comments makes me nervous.
Oddly enough, when I visited the old LPub4 source repository I saw the comment for revision 1.10 said "Adjusted LDView scale to match LDGLite". Now that sounds pretty good. In fact, it could be just what the doctor ordered. So I took a peek at the code and discovered the gawdawful table of ratios was apparently added in that revision, and so were the fudge factors. But the factors were different from those present in the LPub3D code. When I multiply the old fudge factors together I get a number very close to 1, so they used to cancel each other out. That's weird! Why bother? Makes me think we should maybe just try to remove them in LPub3D.
Meanwhile, I think LDrawModelViewer.cpp is the LDView code I want to examine. There's a few things I don't understand, like why does the getHFov function calculate hfov from fov if we gave it hfov in -ca on the command line? And why do we need all that aspect ratio stuff anyhow if we only care about the horizontal FOV? All that business with the tiled rendering makes it even harder to follow. The complexity of this code makes me feel inadequate and uneducated. But if I had to guess, I'd say maybe -ca doesn't work quite right in LDView when the window width is less than the height. I bet it'd all work just fine if we told it to render on a big square surface instead of a rectangle. An aspect ratio of 1 can't do any harm, and all the fudge factors should simply disappear.
However, I guessed wrong. I removed both fudge factors and with the same width, height, -ca, and -cg command line arguments LDView renders the exact same scene as ldglite and l3p/POV. That should make for an easy fix in LPub3D, just use the same cameraDistance function for everything.
And since we're making things easy, I decided to restore the -s ldglite command line option to it's original purpose. Then I added a new -2x,2g command line option to scale the image and edge lines up by 2 and also decimate by 2 with the Gaussian blur filter for the png output file(s). For offscreen renders it doubles the size of the viewport as well, to ensure you don't lose anything outside the frame. That should make things easy. If you just want to decimate, use -2g alone. If you only want to scale up, use -2x alone. I did it this way because someday maybe I'll add more more scale factors, or filter options, all separated by commas.
Update March 5 2016:
At this point, I believe I've got solutions to all of the reported flaws concerning rendering with ldglite in LPub3D. So I've moved on to assimilating the new ldglite code from the LPub3D patch. It looks like I can simply incorporate the C code from the LDrawIni sourceforge repository with a few tiny modifications to make it compatible with the LPub3D patch. No need to pull in all of QT. That's a relief. I really should've done this 12 years ago when I was actively engaged in the LDrawIni discussion on lugnet. BTW, as far as I know that's the most up to date (only existing) documentation on how to use it, so I could've jumped on it right away instead of waiting until now.
These are the changes to the LDrawIni code that I've spotted so far.
- The filenames are lowercased. Nice. Bumpy case is pinky abuse, like emacs.
- The ldrawini.c file adds the base Unofficial directory to the default list. ok...
- It uses an LDSEARCHDIRS environment variable instead of LDRAWSEARCH.
- The additional LDRAWDIRSEARCH01-99 environment variables are skipped.
For a while I also thought the L3PInput.cpp code was changed to reverse the search order of the unofficial parts and primitives from the order used for official files, and from LDrawIni default order. I couldn't imagine why you'd want to do that, and it turns out some string compares were swapped in the code, but the search order was not actually changed. That's a relief.
I'm still unsure why the new LDSEARCHDIRS environment variable was created. I'd think the original environment variables should've sufficed, especially if LPub3D is already making use of the built in LDrawIni support with LDView and L3P. But after further study, I see there's more to it. It looks like the LPub3D search strategy makes use of the LDrawIni codebase, but takes some shortcuts on the LDrawIni approach. The directories in the new LDSEARCHDIRS environment variable are used to supplement the baseline search directories, and might possibly include their own P, Parts, and Models directories without explicitly spelling it all out like in LDrawIni. That would make LDSEARCHDIRS more of a variant on the LDRAWSEARCH01 through LDRAWSEARCH99 settings.
I guess maybe I'll have to support both strategies? Thats a bit more work, but I think it's still feasible. I'm still checking, but hopefully that's it. Once I'm certain I'll add some modified LDrawIni code to the ldglite project and call it a day.
Update March 20 2016:
Heh, I haven't actually made all that much progress on the LDrawIni stuff. I discovered the existing "documentation" no longer matches the current API in the real LDrawIni code. Darn it! There's plenty of comments in the code itself --it's self documenting -- but no simple cut and paste example code that I can leverage. I got confirmation that LDSEARCHDIRS code from LPub3D is in fact a modified/simplified variant of the original LDrawIni code. So my plan is still good. I think I'm just simply having trouble motivating myself because there are more interesting problems to solve.
It's mostly with the linux build. I got an email telling me that I'd broken the linux build on some of the major distros with namespace collisions in the new stuff. So I chose some different names and did some testing on my main LMDE linux PC. It all compiled, but some of the new stuff didn't quite work as well on linux. So I fixed a few bugs, enabled the Alpha transparency bits, and even tried linking to both the off screen osmesa driver and windowed opengl. That always gave me problems (10 years ago) and so I got pretty excited to see it finally work, just like it always did on Windows. It was pretty fast too. As a result I decided to try and compile an older libOSMesa version 7.6 for the aging puppeee linux my ancient eeepc netbook to see if I could make it work there too. And it did work. I hope it's like that on other linux distros.
I stashed the osmesa config settings that I used for puppeee linux here, just in case.
configure-osmesa-for-puppeee-linux.sh
Now I think I may have finally run out of fun excuses for avoiding the LDrawIni conversion. Oh well, at least now I can experiment with it while lounging around the house with the tiny netbook.
Update March 26 2016:
Progress is slow, perhaps glacial, but things are moving. I've started making test builds with progressively larger chunks of the ldrawini code enabled. And I've already introduced my first hacks to the once pristine ldrawini.c file. I had to lowercase some directory names (p, parts, and ldraw) and expose the previously static DirHasPandPARTS() function to make the basics work without having to install a painfully slow LDrawIniFileCaseCallback() function. I refuse to go there. The ldraw library should unpack in lowercase. Linux users will just have to be responsible for matching mixed or upper case filenames and directories exactly if they choose to use them for any other files or directories on on case sensitive file systems. I also added support for more sensible fallback directories under modern linux distributions. So far things are working, but it's hard to fully test without more comprehensive documentation. I did find some docs on the file format of the ldraw.ini file itself, mixed in with the l3p documentation, So now I've made a sample ldraw.ini file for linux to use for testing on my puppeee linux netbook. I'll try a similar file on Windows. Mac testing will have to wait a while.
But since things seemed to be working, I went ahead and made a small module in C to process the LDSEARCHDIR environment variable. It simply tacks a few more search paths onto the end of the list from ldrawini. I gave it a quick test on linux, then compiled it for Windows and substituted it for the ldglite 1.3.0 executable that was distributed with LPub3D. I don't know for sure if LPub3D was exercising the LDSEARCHDIR code, but it was able to make instructions for the car model. So just for fun I made another Windows ldglite with the blur filter, the black background, and the front lighting all enabled by default. I substituted that one into LPub3D and regenerated the car instructions. I think it ran a bit slower than the original ldglite on the ancient A30p laptop, but the resulting images were sharper.
Update April 9 2016:
I've still gotta run it myself on some less vintage hardware -- maybe something under 15 years old -- to see if there's any noticeable speed difference for more normal users. But meanwhile I think I've tested it enough to call it a real release. So I've updated the both the CVS and github repositories, and rolled out some release 1.3.1 executables for Windows on the ldglite homepage. There's a normal release executable, and a special one with the -2g,2x option hardcoded for offscreen rendering so you can get the sharper looking images from ldglite with the current LPub3D release. I've also tested this on OSX 10.4 Tiger. So if I can remember how to make the dmg file then I should be able to roll out a new Mac release as well, although who knows if it'll run on a current Mac...
Here's a test file rendered offscreen with -2g,2x on my old PowerPC emac.
Then, just for fun, I fetched the Lpub4 sources from the ldraw-linux github repository, installed the required QT dev tools, and gave it a run.
sudo apt-get install qt4-dev-tools libqt4-dev
qmake -o Makefile lpub.pro
make
The resulting shiny new linux lpub complained about a missing parts.list file when it started up, so I tweaked the mklist.c sources from ldglite, built a mklist executable, and ran it in the ldraw directory to generate the missing file. I think the ldraw-linux repo has a scripted version of mklist, but I went with what I already know... Anyhow, I set the lpub4 config settings to use a special ldglite executable with -2g,2x hardwired, and got this nice screenshot.
Afterwards I felt like I was on a roll, so I made a Mac dmg file and tested the install on an old macbook the kids left behind with OSX 10.5 leopard. It worked, but the dmg file could use some compression and autorun magic. I forgot the secret dmg tricks, so I'll have to do some reading and tweak it up. This time I'll save the dmg assembly instructions in the CVS repository...
Next I'll try and make a patch for lpub4 to pass the magic -2g,2x args to ldglite so you can use a vanilla ldglite v1.3.1 executable instead of the special one.
And that might be all for this project because I think I'm running outta steam.
