Games take ages to be displayed

[JK1974]

Hi,

I just created a script to mount psx games (see my post here) but it takes ages for the list of games to be loaded.

The games are collected on a NAS in subfolder, and I am filtering for *.cue, but using

dir /s *.cue

takes just about 10 or 20 seconds while GameEx takes several minutes. It is even faster to start a game (copy it to the local HD, decompress is, starting the emulator) than waiting for the list to be created. The NAS is being accessed by a "normal" windows drive letter, so no tricks with UNC-Paths etc.

More annoying, after having played a game, returning to the game list again takes ages - as if the temporary created map file is not working. I have copied a few games on my local hard disc and there it seems to be as fast as expected, and the caching seems to work.

Additionally, calling the starting *.bat file (located locally in the emulator´s dir) is quite fast if the game is on the local hard drive. If the game is on the NAS, it takes 5-10 seconds until the same *.bat (at the same location!) is being stared.

Any ideas what is going wrong or slowing down the process so much?

With kind regards in advance

JK

[chriss]

well i guess you use gigabit lan, correct? so it can't be due to the speed of your network connection! it sounds like gameex is rebuilding the index file every time you display the cue list. usually gameex should only do this on the first startup. do you have this issue only with psx roms?

[bkenobi]

Can you post a log file when you have the problem? The log file will show how long it takes to perform the different tasks. It might help show what's causing the slowdown.

[JK1974]

  chriss said:

well i guess you use gigabit lan, correct? so it can't be due to the speed of your network connection! it sounds like gameex is rebuilding the index file every time you display the cue list. usually gameex should only do this on the first startup. do you have this issue only with psx roms?

Not yet gigabit (the switch is outdated ), but as I said: Doing a simple a simple "dir /s *.cue" which should be enough for getting all the game names just takes 10s-20s.

I just logged in at home via VNC (so some bandwidth goes away) and I did some measurements:

1) Loading the game list: around 4 minutes!

2) Starting Gran Turismo 1, which is around 300 MByte, needs calling the locally saved epsxe.bat (takes around 20s-30s[!!!] until the debug window is being opened), copying the image file (takes around 45s) and ecm-ing it locally (around 60s).

3) Reading the directory again after closing the emulator: again around 4 minutes (just had here 3:45).

Playing around with SNES games, the caching also does not seem to work here - around 60s for reading the files, 30s for a re-reading. But as I said: There might be slightly measurement errors. But starting a game does not make any problems.

As the NAS drive is an USB drive, I could test the speed with non-LAN-"direct access", but this would not be a real solution for me...

Quite interesting: The psx directory is the biggest one by size, but by the number of files, there are directories with more files.

I finally started FileMon (get it from Microsoft) to see what´s going on, and I found out that GameEx is reading the directory several times (I stopped counting at the third repetition), accessing also the data.mdb.

Finally a bug e.g. with the new subdirectory scan option? Maybe even problems with all game data on LAN accessed by GameEx as caching does not seem to work here?

[JK1974]

Here I have added the relevant parts - you can see, that unfortunately there seems to be nothing to see, apart from that there is "no database data".

Or did I chose the wrong logfile?

16:15:39.5  10.03.2009:  Validating: Emulator_9: Custom Background: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Custom Background Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Snap Path: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Snap Path Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Custom Art Path 1: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Custom Art Path 1 Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Custom Art Path 2: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Custom Art Path 2 Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Custom Art Path 3: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Custom Art Path 3 Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Control Panel Path: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Control Panel Path Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Instructions Path: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Instructions Path Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Database: [Console] Sony Playstation
16:15:39.5  10.03.2009:  Validating: Emulator_9: Title Snap Path: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Title Snap Path Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Box Art Path: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Box Art Path Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Cart Art Path: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Cart Art Path Does not exist
16:15:39.5  10.03.2009:  Validating: Emulator_9: Manual Path: 
16:15:39.5  10.03.2009:  Warning: Emulator_9: Manual Path Does not exist
16:15:39.6  10.03.2009:  Restoring Emulator from Cache
16:15:39.7  10.03.2009:  Restoring Emulator Database From Cache
16:16:48.7  10.03.2009:  Restoring Emulator from Cache
16:16:48.7  10.03.2009:  No Database Data
16:21:30.8  10.03.2009:  Restoring Emulator from Cache
16:21:31.0  10.03.2009:  Restoring Emulator Database From Cache
16:21:31.1  10.03.2009:  Restoring Emulator from Cache
16:21:31.1  10.03.2009:  Restoring Emulator Database From Cache
16:21:31.2  10.03.2009:  Restoring Emulator from Cache
16:21:31.3  10.03.2009:  Restoring Emulator Database From Cache
16:21:31.3  10.03.2009:  Restoring Emulator from Cache
16:21:31.4  10.03.2009:  Restoring Emulator Database From Cache
16:21:31.4  10.03.2009:  Restoring Emulator from Cache
16:21:31.4  10.03.2009:  Restoring Emulator Database From Cache
16:21:31.5  10.03.2009:  Restoring Emulator from Cache
16:21:31.5  10.03.2009:  Restoring Emulator Database From Cache
16:22:19.5  10.03.2009:  Running: cmd.exe /c C: epsxe.bat "U:\Emulators\roms\Sony\Complete PSX PAL Collection (Redump.Org)\G\Gran Turismo (E) (No EDC) [SCES-00984]" "Gran Turismo (E) (No EDC) [SCES-00984].cue"
16:23:21.7  10.03.2009:  Running: cmd.exe /c C: epsxe.bat "U:\Emulators\roms\Sony\Complete PSX PAL Collection (Redump.Org)\G\Gran Turismo (E) (No EDC) [SCES-00984]" "Gran Turismo (E) (No EDC) [SCES-00984].cue"
16:41:04.5  10.03.2009:  Starting Attract Mode
...

[Jeff.Miller]

Just out of curiousity... are both machine's clocks set the same? I have seen issues when Windows boxen have clock sync issues.

  JK1974 said:

Not yet gigabit (the switch is outdated ), but as I said: Doing a simple a simple "dir /s *.cue" which should be enough for getting all the game names just takes 10s-20s.

I just logged in at home via VNC (so some bandwidth goes away) and I did some measurements:

1) Loading the game list: around 4 minutes!

2) Starting Gran Turismo 1, which is around 300 MByte, needs calling the locally saved epsxe.bat (takes around 20s-30s[!!!] until the debug window is being opened), copying the image file (takes around 45s) and ecm-ing it locally (around 60s).

3) Reading the directory again after closing the emulator: again around 4 minutes (just had here 3:45).

Playing around with SNES games, the caching also does not seem to work here - around 60s for reading the files, 30s for a re-reading. But as I said: There might be slightly measurement errors. But starting a game does not make any problems.

As the NAS drive is an USB drive, I could test the speed with non-LAN-"direct access", but this would not be a real solution for me...

Quite interesting: The psx directory is the biggest one by size, but by the number of files, there are directories with more files.

I finally started FileMon (get it from Microsoft) to see what´s going on, and I found out that GameEx is reading the directory several times (I stopped counting at the third repetition), accessing also the data.mdb.

Finally a bug e.g. with the new subdirectory scan option? Maybe even problems with all game data on LAN accessed by GameEx as caching does not seem to work here?

[JK1974]

I found the reason for this misbehaviour, and I would call it some kind of bug as it will have impact on every setup where the games are located on a slower drive (network, external USB hd etc.).

Again, I used filemon to check what is going on on when GameEx accesses the files, and get some strange error lines:

1763	23:43:17	GameEx.exe:5400	LOCK	C:\Program files\GameEx\DATA\data.ldb	SUCCESS	Excl: Yes Offset: 1073749505 Length: 1	
1764	23:43:17	GameEx.exe:5400	READ	C:\Program files\GameEx\DATA\data.mdb	SUCCESS	Offset: 61440 Length: 4096	
1765	23:43:17	GameEx.exe:5400	UNLOCK	C:\Program files\GameEx\DATA\data.ldb	SUCCESS	Offset: 1073749505 Length: 1	
1766	23:43:17	GameEx.exe:5400	LOCK	C:\Program files\GameEx\DATA\data.ldb	SUCCESS	Excl: Yes Offset: 1073749505 Length: 1	
1767	23:43:17	GameEx.exe:5400	READ	C:\Program files\GameEx\DATA\data.mdb	SUCCESS	Offset: 61440 Length: 4096	
1768	23:43:17	GameEx.exe:5400	UNLOCK	C:\Program files\GameEx\DATA\data.ldb	SUCCESS	Offset: 1073749505 Length: 1	
1769	23:43:17	GameEx.exe:5400	OPEN	U:\roms\Sony\psx\Colin McRae Rally (E) (v1.3) [SLES-00477].cue	NOT FOUND	Options: Open  Access: Read-Attributes	
...
1773	23:43:17	GameEx.exe:5400	DIRECTORY	U:\roms\Sony\psx\	NO SUCH FILE	FileBothDirectoryInformation: Colin McRae Rally (E) (v1.3) [SLES-00477].cue	
...
1789	23:43:17	GameEx.exe:5400	DIRECTORY	U:\roms\Sony\psx\#\	NO SUCH FILE	FileBothDirectoryInformation: Colin McRae Rally (E) (v1.3) [SLES-00477].cue	
...
1837	23:43:17	GameEx.exe:5400	DIRECTORY	U:\roms\Sony\psx\#\007 - The World is Not Enough (E) [SLES-03134]\	NO SUCH FILE	FileBothDirectoryInformation: Colin McRae 
Rally (E) (v1.3) [SLES-00477].cue	
...
...
14716	23:43:56	GameEx.exe:5400	DIRECTORY	U:\roms\Sony\psx\	NO SUCH FILE	FileBothDirectoryInformation: Gran Turismo (E) (No EDC) [SCES-00984].cue	
...
14732	23:43:56	GameEx.exe:5400	DIRECTORY	U:\roms\Sony\psx\#\	NO SUCH FILE	FileBothDirectoryInformation: Gran Turismo (E) (No EDC) [SCES-00984].cue	
...
27636	23:44:33	GameEx.exe:5400	DIRECTORY	U:\roms\Sony\psx\Z\Zero Divide 2 - The Secret Wish (S) [SCES-01290]\	NO SUCH FILE	FileBothDirectoryInformation: Gran Turismo (E) (No EDC) [SCES-00984].cue
...
27650	23:44:33	GameEx.exe:5400	OPEN	U:\roms\Sony\psx\G\Gran Turismo (E) (No EDC) [SCES-00984]\Gran Turismo (E) (No EDC) [SCES-00984].cue	SUCCESS	Options: Open  Access: Read-Attributes	
27651	23:44:33	GameEx.exe:5400	OPEN	U:\roms\Sony\psx\G\Gran Turismo (E) (No EDC) [SCES-00984]\	SUCCESS	Options: Open Directory  Access: 00100000	
27652	23:44:33	GameEx.exe:5400	QUERY INFORMATION	U:\roms\Sony\psx\G\Gran Turismo (E) (No EDC) [SCES-00984]\Gran Turismo (E) (No EDC) [SCES-00984].cue	SUCCESS	FileNetworkOpenInformation	
27653	23:44:33	GameEx.exe:5400	CLOSE	U:\roms\Sony\psx\G\Gran Turismo (E) (No EDC) [SCES-00984]\Gran Turismo (E) (No EDC) [SCES-00984].cue	SUCCESS

Result: The directory is read twice, once for searching Colin McRae Rally and once again for searching Gran Turismo.

Where are they from? Those two games were the ones that I started before - the "last played games". As a result: Having 10 last played games, the directory is read 10 times searching the CUE files. This also happens if you return from the detail view to the game listing and will make problems on all other emulators that you use.

Using a GoodTool archive even makes it worse as GameEx starts searching for the uncompressed image file in the whole directory.

Possible Bugfix: Storing also the directory or archive information in the data.mdb.

There is currently just one workaround until this problem is being fixed: Disabling the display of the "last played games" in the emulator settings.

[bkenobi]

Wow...you have done a lot of legwork here! This should help Tom speed things up quite a bit... I'll let Tom pick things up from here since I can't do anything to help the situation.

[JK1974]

  bkenobi said:

Wow...you have done a lot of legwork here! This should help Tom speed things up quite a bit... I'll let Tom pick things up from here since I can't do anything to help the situation.

Well, I just had a problem and wanted to solve it

I think that using NAS will become more common within the next years, and the problem that the unzip folders are also generated on the NAS and not on a local temp folder is also one thing that still needs to be resolved (maybe this can also be fixed temporarily with a "wrapper" *.bat).

As there is just low feedback on this and no one talked about those problems before, I can understand if this is currently low priority. So I hope that my findings will speed things up a little bit If not, with the workaround(s) mentioned, GameEx still is a fantastic piece of software.

BTW: I already gave Tom the link to this thread, but thanks for your help anyway.

[fRequEnCy]

Take a look here. Scroll down to number 18. This is gonna be the solution for displaying your list.

[bkenobi]

I think this may need to be addressed again if that's the case. Tom has said his goal is to move towards databases and away from MAP files. If that's true, then this fix needs to work for databases as well. I didn't realize MAP files would eliminate that much time in building the list, so this is good to know!

[fRequEnCy]

  bkenobi said:

I think this may need to be addressed again if that's the case. Tom has said his goal is to move towards databases and away from MAP files. If that's true, then this fix needs to work for databases as well. I didn't realize MAP files would eliminate that much time in building the list, so this is good to know!

It helps a lot. I do this all the time. There's always talk about MAP files being dated but it's still how Tom generates the list of roms so don't see it going away anytime soon.

[JK1974]

  fRequEnCy said:

It helps a lot. I do this all the time. There's always talk about MAP files being dated but it's still how Tom generates the list of roms so don't see it going away anytime soon.

I tried this before, and it did not help. Maybe I did something wrong, so I am going to check it again.

But I don´t think so because when the list is being opened, a temporary mapfile is created, so reading the file list again should be really fast, but it is not because on every played game, the directory is scanned once again.

Please try filemon yourself - it´s a freeware tool by the "famous" Mark Russinovich (and Bryce Cogswell) who had the sysinternals website before and now works for Microsoft.

Edited September 4, 2012 by Draco1962
This thread is now closed. If the problem re-occurs, please start a new thread.