The GameEx Database Project

[headkaze]

GameEx Database Project

What is it?

This is a new project to create databases for GameEx that will include information about games for many different systems and emulators supported by GameEx.

It allows us to:

- Sort games by name, category, publisher, date and number of players

- Read descriptions about games

- Use 3 different naming conventions for our game sets (GoodTool/NoIntro/TOSEC)

- Create online databases for popular games

What about the current databases?

The current databases are incomplete and the format for them has changed. All the databases will be re-created.

How do we collect the data?

Using an advanced software program called "GameEx DBM" (GameEx Database Manager) which is used to manage, maintain and allow data entry via a friendly user interface as well as provide tools for conversion, searching and scraping the web, syncronising data across databases and management of naming conventions (ie. GoodTool/NoIntro/TOSEC)

Team Members

Project / Software Manager: HeadKaze

Data Manager: Flash

[chriss]

great idea, this tool might be quite handy for me... for example the c64 roms need a while to show up, with an mdb file it would take no time (if i get this right ... )

i don't know if this will be possible but since you had a similar option in whdloader i thought i might ask:

i have c64 collection as goodgb64. the naming is [gamename] (number).zip but the screenshots from gamebase64 are just like [gamename].zip

so will it be possible to rename the screenshots so they display with the roms?

[Stu1983]

looking good Headkaze, I don't mind helping out again and don't worry I don't claim ownership on anything I post here, it's a public forum and anything I do on it is for the benefit and use of anyone

Stu

[headkaze]

looking good Headkaze, I don't mind helping out again and don't worry I don't claim ownership on anything I post here, it's a public forum and anything I do on it is for the benefit and use of anyone

Stu

I know that mate, but recent events have made me have to re-think the way group projects are maintained here. There is probably nothing to worry about anymore, but it's just to be sure. Once bitten twice shy if you know what I mean.

Anyway the scrape has finished. The Microsoft Windows one took a bit longer than I expected. Another thing I noticed is the ESRB ratings didn't get scraped because it shows the rating system for Australia for me. So maybe if I could get someone from the US to scrape the Windows database again? It didn't take that long to scrape really. We might not even be using the ESRB ratings anyway so I'll need to find out from Tom what data he's going to be using from the databases before I worry too much about it.

I'll PM you with details on how to download the project when I need your help. Thanks Stu

[starbuck]

A while back I asked about this being incorporated into GameEx. Thought would be nice if on the fly you could update the datebase from the GameEx screen and then have it sent to the same palce as GameEx live goes. That way each time there was a update thru GameEx we all would get updated databases. With all the GameEx communty comtributing to updating and maintanining the database, we would have the most up to date dbs for any frontend.

Looks great

[bkenobi]

PM replied to.

[Tempest]

I'll help too...I would also require coaching!

[headkaze]

Well to be honest I was sort of hoping some of the people who helped create the old databases would be willing to help out with the new ones. That way I wouldn't need to explain what it's all about.

I need volunteer's to fill in missing names. I will explain how to do that in the PM to the people who volunteer. Just remember it's a very easy thing to do, all it is is pressing buttons and filling in data using a bit of common sense. Most of the names are matched using the fuzzy string matcher, so it's only the ones it misses that need filling in manually.

The actual amount of work involved depends on the database, so what I'll do is go through and sort out which ones will take the longest to fill in and which ones will take the least amount of time, and people can decide which databases they'd like to work on. Some databases have got a 100% name match while some have 10%.

I would prefer people who volunteer to understand what GoodTools, No-Intro and TOSEC is and what they mean. It would help if people understood how databases work in GameEx, what a Microsoft Access database is and how to view and edit them. So I do need people who actually understand what most of this stuff is already. If I'm desperate for volunteers I'll probably settle for anyone who can press a button lol

Right now though there are some things I'm working on with Tom to make the program function better and also I want to make sure he's happy with the output it will produce before I upload it for people here. I'll make another post when I'm ready to do that. In the mean time people are still welcome to post here suggestions and ideas for the program before it gets released. I'm sure people will come up with additional ideas once I've sent it out to people, so it will probably be released in new versions as that happens.

[Brian Hoffman]

Send me the info and what you need done still.

Im sure I can dig up some time..... somewhere

[headkaze]

Thanks PimpDaddyStu, bkenobi, shaunopp, Tempest and Brian. I'll be PMing you all with instructions when I've got everything ready.

I'm thinking once you figure out how easy it is to scrape and update a database, I'm sure you won't mind keeping them up-to-date too. Anyway I'll give you intructions on how in a PM soon.

Stay tuned and thanks again for all the responses!

[ReFleX]

I would like to chime in on this...

It would be nice to have a few databases combined in addition to their separated brethren. For example, since GoodTools has 32x and Genesis combined, a combined database would be helpful. Gameboy / Gameboy Color and NeoGeo Pocket / Pocket Color are also the same way.

This wouldn't take any extra work, just include an extra database with these combined so they match the goodtools.

[headkaze]

This project is still happening. I've re-scraped all the databases to fix some of the issues with the previous scrape. I've managed to scrape the ESRB data too.

Right now were just waiting on Tom to answer some of my questions related to the formatting of the databases. I will be PMing the volunteer's as soon as the software is ready.

I have uploaded a table that shows the total number of games in each database and the percentage of GoodTool/NoIntro/TOSEC fuzzy name matches. It should give us an indication of how much work will be needed to fill in the remaining names for each system.

[greatflash]

This project is still happening. I've re-scraped all the databases to fix some of the issues with the previous scrape. I've managed to scrape the ESRB data too.

Right now were just waiting on Tom to answer some of my questions related to the formatting of the databases. I will be PMing the volunteer's as soon as the software is ready.

I have uploaded a table that shows the total number of games in each database and the percentage of GoodTool/NoIntro/TOSEC fuzzy name matches. It should give us an indication of how much work will be needed to fill in the remaining names for each system.

Nice work mate,

Looks like the Saturn DB will need a bit of work (have all U, and all Pal that is not in the U set, and 60% jap)

and what about my poor little zx81 (poor thing)

Did you get anywhere regard alternative names? (did you see my reply re: that in my saturn db problem post?)

All the best you work-aholic.. Hic!

[headkaze]

Nice work mate,

Looks like the Saturn DB will need a bit of work (have all U, and all Pal that is not in the U set, and 60% jap)

and what about my poor little zx81 (poor thing)

Did you get anywhere regard alternative names? (did you see my reply re: that in my saturn db problem post?)

All the best you work-aholic.. Hic!

It's not really going to matter what set's people have because I'm using the dat files which have all games listed already. The databases I am concerned about though are the ones that don't have a GoodTools/NoIntro or TOSEC dat file at all. So we may need people with big sets to make dat's for them.

I'm not sure I understood your idea regarding alternate names. I'm still waiting for Tom to get back to me on how he wants to implement it (which is more important than how I do things in my program). I've proposed that we have 3 columns for each naming convention GoodName/NoIntro/TOSEC where you could set the naming system for each emulator. His suggestion was to separate the databases so that each naming convention had it's own database. My problem with the latter is managing all the different sets of databases. I personally liked the idea of using a column for each one but it's up to Tom how he implements it.

Perhaps you could try explaining it to me again? Bare in mind it's more important to sell the idea to Tom than me

[greatflash]

It's not really going to matter what set's people have because I'm using the dat files which have all games listed already. The databases I am concerned about though are the ones that don't have a GoodTools/NoIntro or TOSEC dat file at all. So we may need people with big sets to make dat's for them.

I'm not sure I understood your idea regarding alternate names. I'm still waiting for Tom to get back to me on how he wants to implement it (which is more important than how I do things in my program). I've proposed that we have 3 columns for each naming convention GoodName/NoIntro/TOSEC where you could set the naming system for each emulator. His suggestion was to separate the databases so that each naming convention had it's own database. My problem with the latter is managing all the different sets of databases. I personally liked the idea of using a column for each one but it's up to Tom how he implements it.

Perhaps you could try explaining it to me again? Bare in mind it's more important to sell the idea to Tom than me

I do prefer the idea of keeping the 3 naming conventions within the one database. Not every dat contains every file. So, some games are missing from one dat and present in another. by having 3 contained in one db then people are still able to mix and match. also, not everyone has a nice clean set.

The naming problems also happen with ISO images. We have tosec ISO, but by looking at the file for the saturn, it expects you to have files as cue, iso, and mp3/wave.. who has them like that??? Mine are all .cdi (selfboots) but named to follow the conventions set by other dats.

With regards my idea. It was to include a relational reference to the naming conventions (so you could have more than 3). It is hard to explain.. The idea was to have a (*) header file that contains a common name with the other data that the rom requires within one record (name, publisher, year, etc), adding to the end of the record a flag that cross references other records that contain just the name (tosec, good, no-intro, generic, etc). The master record (*) would then relate within your editor to the subsequent naming records.

When you output the db, you could then convert it from a relational to a list database by creating records of the various names and appending the data from the master record (*) to each.

All i was trying to suggest is that the 3 naming conventions are not enough. Take a system that has no dat's, and a game that has "the xxxx - yyyy of zzzz" this could be written in so many ways. I was just looking for a way to add alternatives to the db because if people have collected lots of no-dat roms then there is a good chance they will have named them to follow their own style and this would have been a way to add a few guesses to the db list.

This suggestion was mainly made in case Tom decided he would like to keep the original format for db's. Using this format would allow you to do as you suggested with the multi-names and to continue to use one dat. Using the flag to link the matching records would also allow you to input a list db and return it to a relational db and vise-versa. All you need is to only allow data editing of the master record (*) and name editing of the naming records, updating the maser edits to the naming records on output.

God, I know i go on.

I hope you can follow this?

Included a little piccy (hope that helps)

All the best

[Stu1983]

3 MDB's for each system would be a complete mess to look after but I know I suggested extra columns the last time we were working on the MDB's and he wasn't keen on doing anything with the current system then either, heh what can you do, I guess we just have to make the most of what we're given.

Stu

[bkenobi]

Just a thought, but if you *must* use 3 databases for each, I hope 2 (or maybe all 3) would be linked to a master database where the actual data resides. In this case, the extra databases wouldn't need updating, they would just be there to make things easier for Tom. I personally think it's pretty silly to do it that way unless it's a major undertaking to get the correct solution working, but hey, not my call.

I also had another thought... I don't know how much data there is contained in all the databases (I haven't looked recently), but perhaps this info could just be pulled from a server so it could be maintained at a central location and a local cached copy used if no update was desired/needed. Obviously it would be slower that way, but if data is small, then it wouldn't be too bad. Also, it would only need updating if a flag showed your version wasn't up to date. And, it would only need to update out of date info. I haven't messed with this stuff, so I don't know how hard it is putting hash tables together vs just eating the bandwidth of the download. Anyway, just a thought.

[Tom Speirs]

Just a thought, but if you *must* use 3 databases for each, I hope 2 (or maybe all 3) would be linked to a master database where the actual data resides. In this case, the extra databases wouldn't need updating, they would just be there to make things easier for Tom. I personally think it's pretty silly to do it that way unless it's a major undertaking to get the correct solution working, but hey, not my call.

I also had another thought... I don't know how much data there is contained in all the databases (I haven't looked recently), but perhaps this info could just be pulled from a server so it could be maintained at a central location and a local cached copy used if no update was desired/needed. Obviously it would be slower that way, but if data is small, then it wouldn't be too bad. Also, it would only need updating if a flag showed your version wasn't up to date. And, it would only need to update out of date info. I haven't messed with this stuff, so I don't know how hard it is putting hash tables together vs just eating the bandwidth of the download. Anyway, just a thought.

Just a quick update, we're not going to go with 3 seperate databases now.

Not on topic, but Im also looking at having online lists like we have for MAME but for the other emulators.

Now Im lucky enough to have headkaze heading this effort, we should be able to standardize the databases enough to make this possible.

[greatflash]

Just a quick update, we're not going to go with 3 seperate databases now.

Not on topic, but Im also looking at having online lists like we have for MAME but for the other emulators.

Now Im lucky enough to have headkaze heading this effort, we should be able to standardize the databases enough to make this possible.

Like the idea of online lists.

Glad to hear you have decided on the single database. Will be much easier to maintain. All it really needs (apart from complete rom data) is the ability to cull the correct name from the possible matches. Very exciting times. (yes - i am quite easily pleased)

Good 'ol Headkaze.

[headkaze]

greatflash: I get what your saying now, which is really how a relational database works. The (*) your referring to is called a primary key. In fact if you take a look at the GameBase Amiga database it's split up into tables just like you describe. Unfortunately I think it will over-complicate things. Were trying to standardise the way things work so in order to standardize we must use the most common naming conventions (ie. GoodTool's, NoIntro and TOSEC). What I am concerned about though are the systems that don't have any of those dat's. So in that case perhaps I should add one more column called "MiscName" or GameExName" where we can have our own dat's created by the community. If we do that, we could have single dat's (in ClrMamePro format) that we could host that would allow people to fix their set's to match the databases.

bkenobi: It would be nice to have all this data managed online so we can update. It is something I suggested to Tom to help automate things and I think we will eventually move towards that. But for now I'm hoping the volunteer's can get the databases up to speed. It will help if we can all work on a few sets of databases and keep them maintained until we can figure out how to automate things a bit better.

Okay got Tom's e-mail back and were now back on track. I've implemented another table into the databases called "Configuration". This was requested by Tom so that we use the databases for posting the online lists like we have for Mame. I think this is a great idea, and I've attached a image of what it looks like. I added an entry called "Biography" which will include info about the system. I'm sure there will be other uses for this table in the future.

Now I just need to finalise the program for first release

[greatflash]

Okay got Tom's e-mail back and were now back on track. I've implemented another table into the databases called "Configuration". This was requested by Tom so that we use the databases for posting the online lists like we have for Mame. I think this is a great idea, and I've attached a image of what it looks like. I added an entry called "Biography" which will include info about the system. I'm sure there will be other uses for this table in the future.

Now I just need to finalise the program for first release

Regarding the Biography part.... What a GREAT idea! I have been adding the system text to my latest theme on a bitmap for each emu in gameex. This would be a great feature to add to Gameex at a later stage. At least the facility would be there - great forward thinking.

Anyway, i must shut up now and leave it to ya. Getting lock-finger (it's like lock-jaw but in a more fingery manner )

[headkaze]

Regarding the Biography part.... What a GREAT idea! I have been adding the system text to my latest theme on a bitmap for each emu in gameex. This would be a great feature to add to Gameex at a later stage. At least the facility would be there - great forward thinking.

Anyway, i must shut up now and leave it to ya. Getting lock-finger (it's like lock-jaw but in a more fingery manner )

I hope you don't mind grabbing all that data again and pasting it into the new database. Unfortunately I haven't made it backward compatible as the working data is in xml format. Hopefully my program will make it easier to enter data though.

I've been wondering how we were going to manage the databases. As in who is going to be maintaining what databases. I guess for a start it will be a matter of seeing who understands the program and how it works and can actually create working databases. I hope there will be enough volunteers to at least re-create all the databases from the old set. That's the minimum we can hope to achieve since the databases have different requirements now and need to be re-created anyway.

I'm still not 100% sure on some of the requirements. Like I don't understand how the GoodName column was filled out for systems that didn't have a GoodTool dat in the old databases. I'm guessing that fake names were generated? I'm not sure if we should leave blank entries for games that don't exist in the dat files or not. Anyway I did add the ability to generate a fake name if necessary.

Anyway it's just about ready for release. I'm just waiting for Tom to give the go ahead.

[greatflash]

I hope you don't mind grabbing all that data again and pasting it into the new database. Unfortunately I haven't made it backward compatible as the working data is in xml format. Hopefully my program will make it easier to enter data though.

I've been wondering how we were going to manage the databases. As in who is going to be maintaining what databases. I guess for a start it will be a matter of seeing who understands the program and how it works and can actually create working databases. I hope there will be enough volunteers to at least re-create all the databases from the old set. That's the minimum we can hope to achieve since the databases have different requirements now and need to be re-created anyway.

I'm still not 100% sure on some of the requirements. Like I don't understand how the GoodName column was filled out for systems that didn't have a GoodTool dat in the old databases. I'm guessing that fake names were generated? I'm not sure if we should leave blank entries for games that don't exist in the dat files or not. Anyway I did add the ability to generate a fake name if necessary.

Anyway it's just about ready for release. I'm just waiting for Tom to give the go ahead.

I don't mind re-doing the game data. I am in the lucky position of having a laptop at work linked to my phone (hsdpa) and am also the manager, so I can get quite a bit done in the working day

Looking forward to cracking on!

[starbuck]

Just thinking of things in the database that would be neat to have. How bout a column where we can pick a five star rating for the game. Maybe that data can even be tabulated like in Mame Live so those of us who have never played a certain game can see what others have rated it before we play it.

Another neat thing to have is below the game desctiption would be a section where we could put cheats for the games.

[starbuck]

May have missed it. Didn't see a database for Spectravideo SVI 318 & svi328