Since the button works in PinballX but not while VPX is running, the issue is that VPX is taking control of the input and not passing it back to PBX.The clean way to fix this is:
Do not map Exit inside VPX.
In PinballX Setup map your controller button to Exit Emulator only (not Quit).
In VPX make sure the Exit key is set to ESC (default).
In PBX confirm that ESC is the key being sent to close the emulator.
Also make sure: VPX is not running as administrator (and PBX isn’t either — both should match).
In PBX settings “Hide Backglass” and “Hide DMD” are enabled if used.
“Launch Before/After” isn’t interfering.
If that still doesn’t work, JoyToKey mapping your button to ESC (like you are doing now) is honestly a perfectly fine and reliable solution. Many people use it for exactly this reason.