Tuesday, June 19, 2007

Was this for our own good?

Those of you already using Flight Simulator X will have noticed by now that there's a new concept of authorizing third party DLLs and GAUges before they are loaded into FSX.

The first time a user installs a third party addon, depending on how many of those DLLs the addon uses, a security warning dialog box pops up asking the user if they "want to run this software" with a "Run - Don't Run" button choice. If the software (read: DLL or GAU) is code-signed, there will be information on the software publisher and a "More Options" tab which leads to radio boxes allowing the user to "Always run software from 'Publisher'", "Never run..." or "Ask me every time". This is default behavior by the new Windows Cryptograpy API.

FSX adds another layer after the user has (presumably) selected "Run" which allows them to "designate this module as 'Trusted' software", thus making FSX automatically load this software any time it runs again in the future, without having to go through the security dialog pop up again.

All this is well and good - for normal users. Of course, if the software publisher produces an update, the entire process has to happen once more, since the new DLL versions will be detected as different and the user will need to allow them again.

For us, developers, this is a pain in the butt. We produce new builds of our DLLs and gauges many times daily, so we need to keep "allowing" them each time, because if we "always trust" our own code, we are doing our users a disservice in case something goes wrong when FSX first loads the gauge (and I've already been bitten by that happening - and it wasn't even my fault! - more on that perhaps in another blog entry).

If this pain in the butt wasn't enough, I spent the past two days trying to figure out why FSX suddenly stopped asking about verification and designating modules as 'trusted'. Instead, it refused to load any 3rd party addon.

Two entire days. Wasted. I even tried removing FSUIPC and reinstalling it, thinking that I might have done something to my certification process to cause this, so another addon should surely be OK. Nope. FSX never asked me for that either - it simply failed to load FSUIPC too! OK - go ahead and remove the [Trusted] sector from fsx.cfg? Nope. Still couldn't care less - FSX will not load anything. WHY?

In the meantime, I had noticed that, when running the Certification Manager, it would sometimes give me an error when viewing my code signing certificates about "Cryptographic Operation Failed due to a local security option setting". Hmm... Whah? Huh? I know they're bought and paid for - and checked by a valid Root Authority! What gives? This was surely a mystery... but it led me to goog- I mean search MSDN for any related issues...

I can now thank Steve Patrick (at Spat's Weblog) for providing what turned out to be a very simple solution to this issue. While apparently unrelated, it turns out that something had switched my


Trust Providers\Software Publishing

registry key to 0x63c00. This value is Microsoft's WinTrust policy selection flags, as described here. The 0x40000 flag means Allow only items in personal trust database and apparently, this clashes with FSX's ability to allow manual authorization of unrecognized DLLs, telling it instead to never trust anything outside the user's personal trust.

Unsetting this flag (putting it back to its default 0x23c00 value) returned all operations to normal and Flight Simulator was once again able to ask me to designate my modules as 'Trusted' software.

I never found who/what changed that registry key... but at least now I can return to my normal schedule of love with Microsoft's code signing necessities!

This is what I do for a living... who said programmers don't have "the life"! ?

Monday, June 11, 2007

Code, manifest yourself!

(Warning: Extreme nerdy article to follow - talks about C++ code and Visual Studio 2005) - You've been warned!

You might want to chalk this one up to "lessons that were learnt the hard way"...

I was investigating a report from some users on our PMDGSounds.DLL failing to load entirely. So I thought: I must have screwed something up with the release builds - possibly linking to the wrong SimConnect.DLL version. Nope -that wasn't it.

I discovered (after having to remote-login to a beta tester's PC and after going through all the pains of changing .ini settings so that logs can be created) that it was really an issue with dependencies, but of the wrong kind: I had done my latest build on Vista and VS2005 - which since I am regularly updating my setup, had been updated to include SP1 and the latest Vista patch. As such, all MFC and CRT libraries were now linking with the newer SxS assemblies, rather than the RTM builds that FSX links against. Ergo, the missing dependencies on only some users' PCs, ergo my frustration.

This wasn't the first time I had run into this issue, though, so I went back to my notes... I had discussed this with our friends at Microsoft who had suggested / recommended a flag in the Preprocessor settings: _USE_RTM_VERSION, which supposedly will instruct the linker to embed the RTM version of the CRT and MFC libs in the manifest files...

The hard lesson is (after 4 hours of tests) that this holds true for executables (.EXE files) but NOT for DLLs. As most of our addon code is, well, added on to FSX, it comes as multiple DLLs and thus dependencies fail.

The fix isn't what I consider "beautiful" - rather than depending on the auto-magical way VS2005 embeds manifests, I had to make a manifest file myself with the appropriate versions and include that as an RT_MANIFEST resource in the .rc file.

Seems to work, but what happens if later I'd like to add another library dependency? I'll lose the ability for VS2005 to pick up on the new lib automatically and I'll have to remember to add it in the manifest file now...

So much for progress, I guess...