The GameEx Database Project

[headkaze]

I think it's good to get recognition for the work you do. I read the post and really just thought my guess was a good as yours. Sometimes you gotta just go with gut instinct. I know I don't always get the feedback I'd like on some of my projects in the past. And it does get lonely working on these things. I've done a few projects that needed alot of data entry, and they're the sort of things that are best suited to "zone out" to because they can be very repetitive. When I made colors.ini for example I had to write the software to help automate selecting the colors of the CP, then I had to go through every CP snapshot and assign the colors for each button. I broke a mouse doing that, so yeah, I feel for ya, and most people don't even know what colors.ini is

It's important to enjoy these things that we do though otherwise there is no point in doing it. And I think we do get alot out of this hobby, and especially for someone with children, I can imagine it being a great way to relive childhood and share the experience with them. Can you imagine having an arcade machine as a kid with thousands of arcade quality games on it? That would totally have blown my mind when I was young.

[greatflash]

I think it's good to get recognition for the work you do. I read the post and really just thought my guess was a good as yours. Sometimes you gotta just go with gut instinct. I know I don't always get the feedback I'd like on some of my projects in the past. And it does get lonely working on these things. I've done a few projects that needed alot of data entry, and they're the sort of things that are best suited to "zone out" to because they can be very repetitive. When I made colors.ini for example I had to write the software to help automate selecting the colors of the CP, then I had to go through every CP snapshot and assign the colors for each button. I broke a mouse doing that, so yeah, I feel for ya, and most people don't even know what colors.ini is

It's important to enjoy these things that we do though otherwise there is no point in doing it. And I think we do get alot out of this hobby, and especially for someone with children, I can imagine it being a great way to relive childhood and share the experience with them. Can you imagine having an arcade machine as a kid with thousands of arcade quality games on it? That would totally have blown my mind when I was young.

Thanks HK!

I do understand and my heart goes out to you for the "Colours.ini" (some of us still remember English ). I agree that most people (apart from you, Tom, Brian, and a small handful) have no idea of the work that goes on behind the scenes. They only pipe up to say "this does not work", etc.

I have uploaded ALL the refined XML's (71) to the ftp. PLEASE can you back up the DATA directory in DBM and replace with the one contained in the archive please. All XML's have Biography's and there are a couple of new systems added (and C16/+4 to add later). The Database.xml has been modified to include these and the un-needed systems are removed for now (Brew, symbian, etc) to make the amalgamation of the pub and dev fields quicker (Still took well over 40 hours though). These systems can be put back at a later date (though I really cannot see the point).

Please have a look through for me if you get a chance. There is still a bit of work to do on the pub dev thing, but it is 99% complete (apart from psp which needs the Dev field sorting).

The Developers have been REDUCED. By this, i mean that location based studios have been combined.. Rockstar North, Rockstar London, Rockstar Wagga Wagga ( ) have all become Rockstar Games. I know this is not 100% correct, but it give the ability to look at the creative output of one software house as a whole. The same goes for Disney studios, Disney interactive studios, Sega Entertainment Europe, Sega UK, Inc. Sega, Inc. etc..

I hope this is ok and the benefits will far outweigh the slight loss of honesty. I don't think it really matters that they are combined, If I want to see a list of games developed by Sega - then that is what I should get, not a list of 30 software houses that start with Sega and have to look through each of them to find what I want.

I hope you agree and I am sure that everyone else will not worry one way or another (from recent experience )

To go back,

Please download the zip

Rename/backup your data directory

use the one from the zip

enjoy looking at the notes.

(mdb's are also in the zip for you)

PS.

The pull downs for both Developer and Publisher really do need to work GLOBALLY pretty please. I fear adding anymore games as I would have to run an "All games" mdb side by side to compare them and find the correct entry. With a global pull down - one click and the entry for that dev-pub will be the correct one to use. Pretty bloody please........ (big hug! <flutters eyes>)

cheers HK!

[headkaze]

No worries Flash I've been meaning to add that global dropdown for Publisher and Developer.

I agree on the naming system you've come up with. Sega should be Sega etc. Let's think about the practical use of the databases we don't want pages and pages of different names to scroll through. So yeah I'm happy about that decision.

Thanks I will update the xml's I have and hopefully get some time today to update DBM

[greatflash]

No worries Flash I've been meaning to add that global dropdown for Publisher and Developer.

I agree on the naming system you've come up with. Sega should be Sega etc. Let's think about the practical use of the databases we don't want pages and pages of different names to scroll through. So yeah I'm happy about that decision.

Thanks I will update the xml's I have and hopefully get some time today to update DBM

Thanks HK.

Let me know what you think of the new DB's.

I will wait eagerly for my next installment of the DBM saga..

[headkaze]

We should have done this sooner Flash. I'm adding a feature that would have saved you hours on the Search and Replace thing. I just thought of it about 10 minutes ago.

I've just added global lists for Publisher and Developer, but then I thought of a great idea for Search and Replace. I have now added those same drop down lists there so you don't need to use that large mdb to copy and paste the names across. Now you can select them directly from a list.

Have you done all of the Search and Replace? I feel bad that I didn't come up with this idea sooner

[greatflash]

We should have done this sooner Flash. I'm adding a feature that would have saved you hours on the Search and Replace thing. I just thought of it about 10 minutes ago.

I've just added global lists for Publisher and Developer, but then I thought of a great idea for Search and Replace. I have now added those same drop down lists there so you don't need to use that large mdb to copy and paste the names across. Now you can select them directly from a list.

Have you done all of the Search and Replace? I feel bad that I didn't come up with this idea sooner

Just about mr. "I will put the plug in the bath AFTER the hot water has run out" :)

I have a few to do. The main thing is to be able to keep it consistant with the alterations already made. I still have to go through ALL the games in the db's to make sure they all match "Tic Tac Toe" "Tic-Tac-Toe" "Tic, Tac, Toe" etc. Will a drop down on the game name be possible also??

When all is done, this is certainly going to be something that no other FE can offer

[greatflash]

I have the other missing db's (the ones that are of no use YET) to do and also of course, any others that I add (c16-Plus4).

[greatflash]

I have the other missing db's (the ones that are of no use YET) to do and also of course, any others that I add (c16-Plus4).

Anyway - no worries about the s&r. It may have taken longer, but it had the benefit of seeing it against the other fields that helped me to make a judgement on what naming to use, and to spot errors that were still present. So - it was not all bad to have done it that way. My Ctrl, C, & V keys are a bit shiny now on my NEW sony Z11 ....

Now off to bed, so If you get anywhere with this, I will catch up with it in the morning mate. Thanks for all your help (12:45 am up here)

[headkaze]

I thought it must have been late over there cos it's early over here.

I was just about to upload when I saw you say "can you add a dropdown list for name" and I thought "what a good idea" so I'm adding that now. Hopefully this will save a whole lot of keys on your Z11

[headkaze]

DBM Version 1.65 Released!

- Added global caching for Name, Publisher and Developer. It takes longer to load DBM now but will display a loading bar showing the progress of the caching. It only takes about 8 seconds to load up for me, no biggie (it's caching 44,492 games afterall).

- Added three new buttons to "Search and Replace", namely "N", "P" and "D" for Name, Publisher and Developer respectively. When you press these the global cache list will show allowing you to choose from the list (also added a new search bar for searching the list). This should help with syncing databases *alot*.

- Added a "..." button for Name in the GameInfo edit screen. When you click it, it will show the global cache list and do a fuzzy search and hilight the game (or closest match). Again should help alot with keeping names in sync.

[greatflash]

DBM Version 1.65 Released!

- Added global caching for Name, Publisher and Developer. It takes longer to load DBM now but will display a loading bar showing the progress of the caching. It only takes about 8 seconds to load up for me, no biggie (it's caching 44,492 games afterall).

- Added three new buttons to "Search and Replace", namely "N", "P" and "D" for Name, Publisher and Developer respectively. When you press these the global cache list will show allowing you to choose from the list (also added a new search bar for searching the list). This should help with syncing databases *alot*.

- Added a "..." button for Name in the GameInfo edit screen. When you click it, it will show the global cache list and do a fuzzy search and hilight the game (or closest match). Again should help alot with keeping names in sync.

Well done HK..

Now finishing the N64 one so all the pub&dev match.. Works a treat. Will still uses export to do the game names to build the list for global dropdown. This will then make matching newly added games a treat!!!

Will let you know how it gets on..

[greatflash]

An open invitation to all,

The N64 db is nearly complete...

Any suggestions to what one to do next????

Snes, Nes, Genesis, etc......

[bkenobi]

Any of those 3 would be great! SNES, NES, and Genesis are the 3 main systems for me! N64 has never played that well due to poor controls (course I just got my Logitech Rumble Pads the other day, so that may not be an issue anymore ).

[greatflash]

Any of those 3 would be great! SNES, NES, and Genesis are the 3 main systems for me! N64 has never played that well due to poor controls (course I just got my Logitech Rumble Pads the other day, so that may not be an issue anymore ).

I think I will start on the Snes. ? . ? .

When I do this one, I will have to omit all hacks and pd. If I don't - I will be a LONG time. When all the db's are done, I will go back and update them to add these extra's. I think the main concern is to get all known official games done!

[bkenobi]

I totally agree. There are hundreds of games to start with. If you add all the variants, the list would be easily in the thousands if not higher! I'm not sure if it's really a problem using the base game for the description of a hacked version anyway. Perhaps GameEx will automatically match them to the official game anyway?

[greatflash]

I totally agree. There are hundreds of games to start with. If you add all the variants, the list would be easily in the thousands if not higher! I'm not sure if it's really a problem using the base game for the description of a hacked version anyway. Perhaps GameEx will automatically match them to the official game anyway?

No, it will not generally match hacks unless the only difference between the two names is the addition of [hack] or (hack), which would not be regarded.

I does not really notice, and these will be added later. Best to get the REAL games done. Will still include prototypes and unreleased.

[bkenobi]

Heh, doesn't really matter to me anyway. I went with No-Intro...

[headkaze]

DBM Version 1.69 Released!

- Another large update which covers recent requests from Flash (which he described as a "quickie" lol yeah sure!)

- Searches now show results of matches in red. Also data displayed for a replace displays the replaced text instead of the found text.

- You can now run "File->Create Mdb(s)" to create a whole bunch of mdb's at one time. Just select the ones you want from the database list and it will create the mdb's. Now that were doing alot of multiple database editing now I think large groups of databases will need to be updated at a time.

- Now for the interesting part which I'll have to explain in detail Flash. The mysterious repeating names in the global cache.

Eg.

Blizzard Entertainment

Blizzard Entertainment

After some investigations the Publisher and Developer fields that appear exactly the same but are treated as different are caused by a non-breaking space (0xa0) character which looks the same as a normal space (0x20) but will cause them not to match. This is caused by the data being scraped from html. So I have added a new feature called "Tools->Fix Non-Breaking Spaces".

Here are some instructions on how to fix the non-breaking spaces problem:

1. Go into Tools->Search And Replace and select the Publisher or Developer buttons "P" or "D" you will notice a whole bunch of entries in red. These are the names which contain non-breaking spaces.

2. Now select Tools->Fix Non-Breaking Spaces and it will go through all the databases fixing the fields by replacing the non-breaking spaces with regular spaces.

3. If you go back and open the Publisher or Developer cache they should all now appear as black and the repeating names should not be there anymore.

Also just to note, now when you do a new scrape or scrape update I now filter out non-breaking spaces automatically. So you will probably not need to do this ever again. But if you do ever see a red entry in the global cache for Name, Publisher or Developer you know you will need to run the fix again.

- Other minor updates and fixes I can't think of right now

[greatflash]

DBM Version 1.69 Released!

- Another large update which covers recent requests from Flash (which he described as a "quickie" lol yeah sure!)

- Searches now show results of matches in red. Also data displayed for a replace displays the replaced text instead of the found text.

- You can now run "File->Create Mdb(s)" to create a whole bunch of mdb's at one time. Just select the ones you want from the database list and it will create the mdb's. Now that were doing alot of multiple database editing now I think large groups of databases will need to be updated at a time.

- Now for the interesting part which I'll have to explain in detail Flash. The mysterious repeating names in the global cache.

Eg.

Blizzard Entertainment

Blizzard Entertainment

After some investigations the Publisher and Developer fields that appear exactly the same but are treated as different are caused by a non-breaking space (0xa0) character which looks the same as a normal space (0x20) but will cause them not to match. This is caused by the data being scraped from html. So I have added a new feature called "Tools->Fix Non-Breaking Spaces".

Here are some instructions on how to fix the non-breaking spaces problem:

1. Go into Tools->Search And Replace and select the Publisher or Developer buttons "P" or "D" you will notice a whole bunch of entries in red. These are the names which contain non-breaking spaces.

2. Now select Tools->Fix Non-Breaking Spaces and it will go through all the databases fixing the fields by replacing the non-breaking spaces with regular spaces.

3. If you go back and open the Publisher or Developer cache they should all now appear as black and the repeating names should not be there anymore.

Also just to note, now when you do a new scrape or scrape update I now filter out non-breaking spaces automatically. So you will probably not need to do this ever again. But if you do ever see a red entry in the global cache for Name, Publisher or Developer you know you will need to run the fix again.

- Other minor updates and fixes I can't think of right now

Another great update HK. DBM is getting pretty impressive. It is funny how these tools can expand.. Had a look at the first release the other day - god, it really has changed.

Oh, just remembered the other thing ( ), it was context sensitive right click options and the use of ctrl-a in the db list to select all.

The save mdb(s) feature is a great idea that I failed to think of ( )

Thanks again HK.

N64 is done (somehow missed GP championship 2?), will check the strange spaces from the ether and will post tonight after work. Making a start on the SNES one now (2500 games at the mo - dare not expand goodname just yet).

Will have a play today and let you know how I get on..

[headkaze]

Version 1.70 Released

- Added a "Stop" button for tasks that take a while (like Add Missing) incase you want to cancel

- Added "Re-Load" button, again if you do something wrong you can just re-load the database without saving

- Most tasks that take a long time now show in the progress bar and show which game is being processed (Eg. Get Names / Add Missing)

- Should be no more tasks which hang DBM with "Not Responding"

What options would you want on the context menu? Ctrl-A should be easy enough to add. Yep it's come a long way

[greatflash]

Version 1.70 Released

- Added a "Stop" button for tasks that take a while (like Add Missing) incase you want to cancel

- Added "Re-Load" button, again if you do something wrong you can just re-load the database without saving

- Most tasks that take a long time now show in the progress bar and show which game is being processed (Eg. Get Names / Add Missing)

What options would you want on the context menu? Ctrl-A should be easy enough to add. Yep it's come a long way

Edit games, select all, delete, save - from the game list

and the functions of the File menu in the dat list.

That is pretty much it.. These are not really that important. It is really just me. You get so used to right-clicking on selections - and i STILL do it all the time..

PS. Good morning!

PPS. Still not registered on my website (Flash looks forlorn and dejected)

[greatflash]

Edit games, select all, delete, save - from the game list

and the functions of the File menu in the dat list.

That is pretty much it.. These are not really that important. It is really just me. You get so used to right-clicking on selections - and i STILL do it all the time..

PS. Good morning!

PPS. Still not registered on my website (Flash looks forlorn and dejected)

Quick note

The fix spaces function works a treat.. Love the influx of redness, could spend hours changing publishers names just to watch them blush!!

Great work as always HK!!!

[greatflash]

Quick note

The fix spaces function works a treat.. Love the influx of redness, could spend hours changing publishers names just to watch them blush!!

Great work as always HK!!!

Right, one more thing.

Search & replace work GREAT and need to work on whole fields, but... Search (as a single function) needs to work on a match within a field.

eg.

in SNES

"Acme Animation Factory" is listed as a single entry with one dat match (nointro), and I am sure both good and tosec will contain this game, so it must be in the list somewhere? (Right-Click) search could be my friend here (and god knows - i need one! ) and enable me to search for ACME in the name field to find what the other is called so that I can put the entries in one record.

This MUST NOT affect the function of Search&Replace, this must always work on a whole record..

Another great idea from the little world of FLASH © 2008, Alll rights relinquished!

[headkaze]

DMB Version 1.71 Released

- Added context menus

- Ctrl-A selects all (BTW shortcuts are easy to add so if you want any more like this just let me know)

Mornin Flash! Holy crap you've come up with another idea! To be honest I don't really understand what you mean by another search. As far as I can tell you can already search GoodTool/NoIntro/TOSEC databases when you edit a game with the fuzzy matching. I could be wrong but I don't really understand what you mean.

Anyway there are a couple other issues I wanted to bring up

- We need to scour the GoodTools website to see if there are new dat files as well as TOSEC and NoIntro dats. Update them as necessary. I'm sure there are some out of date already.

- I've done a few update scrapes and have noticed quite a few databases have new games already (some have 20+ new games). So are you doing scrape updates as you go?

[greatflash]

DMB Version 1.71 Released

- Added context menus

- Ctrl-A selects all

Mornin Flash! Holy crap you've come up with another idea! To be honest I don't really understand what you mean by another search. As far as I can tell you can already search GoodTool/NoIntro/TOSEC databases when you edit a game with the fuzzy matching. I could be wrong but I don't really understand what you mean.

Anyway there are a couple other issues I wanted to bring up

- We need to scour the GoodTools website to see if there are new dat files as well as TOSEC and NoIntro dats. Update them as necessary. I'm sure there are some out of date already.

- I've done a few update scrapes and have noticed quite a few databases have new games already (some have 20+ new games). So are you doing scrape updates as you go?

I do a scrape when I start a new db. I will not re-scrape until I have completed quite a few.. I will never get it done otherwise

The SEARCH was not for an online search, but for the SEARCH&REPLACE function.

Follow me here....

At the moment S&R checks for the FULL record and replaces with the FULL amendment - This is exactly as needs to be, perfect.

But a standard search (within S&R) needs to check for "within" a record, so a search of "ACME" will find "ACME Sausage Factory", "The ACME sausage Factory", etc. Using this I could find orphaned entries where a dat has used a slightly different name.

THIS MUST NOT affect S&R..

Implementation:-

Clicking "Replace" - Search for FULL matches and replace with FULL matches.

Clicking "Search" - Search for PART matches withing fields and display results only.

-------------------

Ok, now down to Scrapes again.

When you scrape, new data is entered into the current db. You have no way of knowing which are new and in this way the pub,dev, and name caches will become muddled again. Can a new scrape use a triangle (example) as an icon (instead of the coloured circle) in the game list, and remain (even when db saved) until that record is edited? This way, you will know what needs to be checked at all times. All the db's need is an "Edit Flag". Set to 0 by default and change to a 1 to mark a modified entry. Do ya catch me drift!!??

-------------------

In the games list of an open dat, can the devs and pubs of non-matching entries (dev is not in cache list, etc) be displayed in bold or similar (remember that I am colour blind). This way, a scan of the list will show me edits that are needed. Not sure how to implement this because any dev with a single entry in the cache would be marked in bold. Perhaps something could be implemented with the Modified flag? Or is this a unusable idea..

--------------------

Can the column headings be swapped around and remembered. So I can have the dat names at the back and the year after the name if i need to. On lists with large names the dat names are off the right of the screen In this event, I could move Name to the back so the first 3 are the dat names for easy checking etc... Or perhaps I could move the dats behind description so I have a better view of the data for a game?

-------------------

Do you like all my minus signs?

-------------------

I will update the tosec/good/nointro at some time, but am quite scared of using updated ones as I have removed and amalgamated so many entries that an "Add missing" could really mess a dat up. Try having a play with the 7800 or 5200 for an example of "Add Missing".

You also get multiple entries again...

-----------------------

More minuses and Thanks for ALL your work HK..