I don't have a quick answer. It looks like it is troubleshooting time.
You say that PinballFX2, VP, and FP run smooth. Is it also smooth with PinballX launching them? If you are setting up these programs to launch from PinballX for the first time then it is possible that you find these may not run as smoothly with PinballX. PinballX does take some resources while running and for your system it may be enough to tip performance over to a level that is not as smooth.
Are there any other programs that are running in the background taking resources like an anti-virus, Steam client, etc? In my cabinet, I don't install anything that is not related to running pinball. No email client, no web surfing, etc. Unload/uninstall any programs you don't have to have.
As a troubleshooting measure, what happens with the playback performance of the videos if you don't run the DMD?
I don't know the quality or age of the videos on the FTP. If they are old and have been on the FTP for a long time, then they likely are out of date and do not reflect the improvements in visual quality of the tables that are currently available at VPF, VPU, etc. The cool kids generate their own media. This tool is specifically designed to output videos that work with the LAV filters. What's more it allows you to maintain a video collection that truly reflects the tables as you like to play them including any customizations that you may make for yourself such as lighting or camera POV adjustments. You might give it a try and see if the videos it generates run more smoothly.
https://www.gameex.info/forums/topic/17684-pbx-recorder-captures-playfield-backglass-and-dmd-videos-and-images/