Take a cube, delete one face, then run my double side operation creates an open box. However you cannot select any of the inside faces. If you invert the box you can only select the inside faces.
I suspect you're optimizing the code to stop after finding a face that shares a set of points. However there may be two faces, one with a normal facing the other way.
I figured it out. Fixing it is a little bit more involved. It will come, probably sooner rather than later since the fix will improve picking for more than just this case.
Paul, have you had any luck getting the picks from D3DX8's "Intersect", beyond the first one? I've tried all sorts of ways to coax it to give me more than one, but to no success.
I actually have some "interesting" code for getting more than one pick, but it won't find more than one face from the same plane.
The Visual Basic version of Intersect is more limited than the C++ version for some reason. I get the feeling sometimes that MS had the interns working on the Visual Basic version of the API.
I've written my own ray intersection code that I use for the boolean operations. It's more accurate than the one provided by MS, but it's just not fast enough.
I suppose I could ray pick myself only when I get more than 1 hit, but that isn't practical I think since virtually every time you hit one face, you hit one on the other side of the object too. I suppose if I just checked for more than two it would be OK.
I'll have to think a bit more, but I probably will have to leave this until I get DirectX 9 up and running.
There is a good work around for this that I posted in the forums: work around