The GameEx Database Project

[headkaze]

There are some issues that need to be fixed with the GameEx DBM software. It also needs to be updated to export to the new database format.

It does run under Mono for Windows though so I'm sure it wouldn't be too difficult to get it running on OS X (if it doesn't already).

[nullPointer]

First I’d like to say huge props to all that have worked on this project. This feature was one of the first technical aspects of GameEx that drew me in, and continues to be one of my favorite features of the program. The recent move to SQLite has only sweetened the deal.

In checking out the new database structure, I noticed that some fields that used to be unique to MAME now appear in the standard emulator tables as well. Among other things, this includes the Series field. I’m viewing this with cautious optimism, as I think it would be awesome to be able to leverage the Series field against all emulators. As a result, I’ve got a few questions (one of which I’ve been mulling over for a while):

• Are there any plans to add functionality in GameEx to utilize the Series field for standard emulators?
  

I’ve also been pondering another question in relation to GameEx database development (although it’s a bit of a doosey).

• Has there ever been any discussion of incorporating all the separate databases into a single database schema? Especially now that we’ve moved to SQLite, it seems to me that this could open up a lot of possibilities. For instance what if the ‘Series’ field was instead moved to its own table? All of the sudden it opens up the possibility of cross-platform series entries via foreign keys to the various system tables (or even a single foreign key to a single table for games if anyone were crazy enough to go down the database normalization route). The same idea could be applied to a ‘Playlist’ table enabling users to generate their own ad-hoc cross-platform playlists. I know a lot of users would be ecstatic to see that feature, and it seems to me that tweaking the structure of our database(s), might well put some of that functionality within reach.
  

Having asked those questions, I want to reiterate that I think our current database implementation is awesome; this is just me playing “What If?” Does any of the above sound feasible? Are there technical limitations preventing this sort of shift? Is anyone else hungry for a taco?

Edit: I should also note that I've always been interested in helping out with this project, so if there's currently a need for that, I'd be glad to do so. Feel free to let me know if you're looking for extra hands on deck; I'd be honored to assist in any way.

Edit 2: After having some time to think a bit more about this, I realized that I may be assuming a tighter integration between GameEx and the database(s) than actually exists. All the same it would be interesting to get perspective from one of the database gurus on this subject.

Edited July 20, 2012 by nullPointer

[u-man]

i would also like to help. i have no plan about SQLite, but no pain no gain .

if some task requires diligence in order to complete it, i am willing to do it.

[Draco1962]

Has there ever been any discussion of incorporating all the separate databases into a single database schema?

@null / GameEx authors - Not to throw a wet blanket on things as you are probably much more knowledgeable than I am with regards to databases, structure, and query. My concerns for such create questions as follows:

- With one large DB what additional drains would there be on memory, especially if data is being loaded for systems one doesn't employ?

- Since the DBs are loading at startup, wouldn't one large DB add to the boot time?

- If there is a corruption in one DB, can GameEx continue to run so long as that emu and its related DB is not invoked? If so, would corruption in a single DB, depending on where the corruption occurs within the data, possibly crash GameEx entirely?

I am also looking into using the SQLite tools so that I can increase my knowledge and abilities inside and outside of GameEx.

[celly]

With the implementation of SQLite DB in 12.71 should I just scrap the rest of my ScummVM DB for 1.4.1.? It's in MS Access file. That really stinks since it was almost completed......

Or is there a way of converting the file to a SQLite compatible file?

[Draco1962]

I believe that the file should be convertible as there are a slew of conversion utilities available. I am going to elevate this to Tom and Headkaze so that one of them can answer as to whether or not a straight conversion would be compatible or if they may need to convert to assure compatibility.

[celly]

Thanks Draco. I asked Mr. Google a few minutes ago and he said this. I'll just hang onto the file in the meantime.

[Tom Speirs]

Just wanted to say acess databases are still compatible. You don't have to convert it. Perhaps just convert it for a release version if you like but its not a requiremet..

[Tom Speirs]

If it would be me I would still continue to work on databases in Access as Access is much easier to use. Both are still supported and there should be no difference in performance. SQLite may be a little faster but the main reason for the change was to make GameEx more cross platform.

[Draco1962]

Thanks for that clarification, Tom! I haven't updated the Interactive Fiction DB in a while and find Access a tad easier to maintain since it is what I am most familiar with. I am also looking forward to Celly's ScummVM update!

[celly]

Cool, thanks Tom for the clarification.

@ Draco, it's coming along. Work has bee busy so my time has been limited.

[nullPointer]

My concerns for such create questions as follows:

Hi Draco! Sorry for an exceedingly late response. I was waiting to see if HK, Flash, or Tom had input on this, as they are obviously much more knowledgeable in relation to the system as it exists today. I’ll add my input (for what it’s worth), as I’m still pretty interested in opening up a dialog in this capacity (at least until one of the guys above states that a single database would be impossible/exceedingly implausible) . I’ll do the Q&A format as it’s just easier:

- With one large DB what additional drains would there be on memory, especially if data is being loaded for systems one doesn't employ?

I’m no database architect, so keep that grain of salt firmly in place. I don’t think the database itself would take up a lot more resources than it does today. The database won’t (or shouldn’t) cache all data in memory or swap space (otherwise what’s the point of the database). The database session has the potential to be resource intensive (particularly where we’re talking about multiple concurrent sessions – but we don’t need to worry about that here). If the model works in the following capacity the resource use should be close or identical to the way it is today:

• GameEx opens a session with the database
  
• GameEx queries X dataset and stores that data set in memory (probably a similar dataset to what’s in place today)
  
• GameEx closes the session to the database
  

- Since the DBs are loading at startup, wouldn't one large DB add to the boot time?

This is somewhat related to the question above. If the queries are written in an efficient manner I wouldn’t estimate that it would take longer than it does today. This actually gets to the root of my line of inquiry. XBMC utilizes a single SQLite database structure, which seems to be pretty efficient (seen here and here in various incarnations). So I was in XBMC one day checking out how movies are linked to multiple actors and thought, “Gee, wouldn’t it be awesome if a game in GameEx could be linked to multiple series, genres, or whatever and all of it was cross-platform?” This certainly doesn’t equate to me saying, “Hey let’s be XBMC,” it’s simply a point of comparison. XBMC seems to be relatively efficient in the capacity, and I have no reason to believe that we couldn’t do the same.

In fact it might even be quicker. Bear in mind that today the system has to open multiple Access databases in order to cache the necessary data. To what degree the opening and closing of multiple separate DB connections affects performance I don’t know (it seems fairly snappy), but I think it might save on processing time to be running multiple queries against one single database, rather than multiple queries against multiple databases.

- If there is a corruption in one DB, can GameEx continue to run so long as that emu and its related DB is not invoked? If so, would corruption in a single DB, depending on where the corruption occurs within the data, possibly crash GameEx entirely?

This is a very good question, and I think it boils down to two questions.

• What if the database is still (mostly) functional, but the data is corrupt? This would come down to how GameEx implements error handling for bad data. Only one of the guys above has a solid answer to that question. My guess is that there are already safe guards in place to protect against bad data without bringing the system down.
  

• What if the entire database is corrupt and cannot be accessed? Well this is certainly a pickle isn’t it! This may be a potential flaw in the single master DB idea, but I would predict that the odds of this happening are exceedingly slim. (I think the first scenario is much more plausible). If this was the case, the user would likely need to reinstall (or it might even be related to the db caching. Once again this is all pretty hypothetical)
  

OK, so if anyone is still awake, that’s my premise generated purely from whole cloth. Once again I’d like to hear if any of this is plausible, but currently I think that this could have the possibility to be awesome.

[nightwulf]

Sorry to resurrect an old topic ...

After noticing some inconsistencies between the current no-intro set and the database for the Super NES, I set about trying to update the database. Attached is my first effort.

There was a surprising amount of manual intervention needed. If I can script everything to the point of (almost) automation, I'll be sure to share. Be forewarned though I'm a scripter, not a GUI designer.

Console Nintendo SNES.db3.zip

[Draco1962]

nightwulf - thanks for your contribution! It is much appreciated!

[sbaby]

Hi, I wonder if there is a database of wii games into wbfs format, example SMNP01.wbfs = New SUPER MARIO BROS. Wii

 

thanks

[sbaby]

Anyone? 

 

I found this http://www.gametdb.com/wiitdb.zip?LANG=IT&WIIWARE=1&GAMECUBE=1

How to convert it to db3 format?

[Draco1962]

47 minutes ago, sbaby said:

Anyone? 

 

I found this http://www.gametdb.com/wiitdb.zip?LANG=IT&WIIWARE=1&GAMECUBE=1

How to convert it to db3 format?

Converting a db from one format to another shouldn't be difficult. I am sure that there are plenty of free programs to do so.  The problem you will run into is whether or not the db is formatted to work with GameEx as not all dbs are ordered the same. 

[sbaby]

with excel I managed to do this by the tdbgame.txt but only for names , better than nothing . see my attached file 

[Console] Nintendo Wii.mdb

[morgoth32]

Hi, here is a small one.

I only did my collection but you still can complete it.

nintendo wii.db3

[sbaby]

thanks . I want complete it but I don't know how to edit db3. Can you help me please ?

 

Or send me mdb

[Adultery]

Try SQLITE browser. It's free.