Possibly driver related but hard to say. Being that we're dealing with a third party application to generate scan lines and a third party application to switch resolutions, there are a lot of moving parts here. I'm going to throw you a curve ball. Rather than faffing about trying to make this current solution work, have you looked into using MAME's inbuilt support for HLSL (or alternately OpenGL) shaders? I would say that this approach is much more in keeping with the "MAME vision" of how to properly apply shaders, filters and whatnot. It's just a thought, but it is the first approach that I would try. (Although fair warning, you would probably want to update to the most recent version of MAME)