When I tested it before, i did not have Gameex or 4DO full screen, so that was not a good like-for-like test...but it still told me that it was the right place to look.
When both were windowed, hitting ESC closed 4DO and GameEx came back up and was in focus. I found that without the [Mapping Off] Exit (Process Close) setting, ESC did not exit and instead changed the application from fullscreen to windowed mode as you stated. With it enabled, 4DO did close nicely and return to GameEx.
When both were full screen and that setting enabled, ESC closed out 4DO, but my PC would not quite return to GameEx. The screen would flash back and forth between a black screen and the desktop. It would continue cycling until I hit ESC again which in turn closed GameEx out. So, on Windows 7 this is sort of working...but is not useful. All of the other emulators I have utilize [Mapping Off] Exit (Process Close) or a variation of it and they exit nicely. So, I think that this is the option we need to leverage, but the functionality to work specifically for 4DO may not be present.
There may also be a difference between how GameEx is operating in Windows7 and Windows10 since our results are differing. Maybe someone else with Windows10 can test this as well.