beta db's

[Nologic]

I guess the next step would be to rename the current primary cat field to "Non-Sport" or the like, so we can preserve the data we have in there for now. Then add a new cat field...and start importing AllGames cat's for use in GameEx till such a time comes to where we can use multi-select cat sorting.

After that point...probably add a new name field to match up with No-Intro...then later followed up with TOSEC...even tho I have a bitch with those guys.

That way Tom should be able to parse the roms and decide based on percentage which name field in the db should be used to match things up.

[Stu1983]

I guess the next step would be to rename the current primary cat field to "Non-Sport" or the like, so we can preserve the data we have in there for now. Then add a new cat field...and start importing AllGames cat's for use in GameEx till such a time comes to where we can use multi-select cat sorting.

After that point...probably add a new name field to match up with No-Intro...then later followed up with TOSEC...even tho I have a bitch with those guys.

That way Tom should be able to parse the roms and decide based on percentage which name field in the db should be used to match things up.

Sounds good, just wondering where you came up with the name "Non-Sport"? Would "Multi Catergory" not make more sense:) Anyway I'll have to leave the AllGames bit to you as I've no idea where to start writing a script to grab the Catergory but if you get 1 wrote I'll certainly help running it on the databases

As Tom ever said if he's willing to support multiple name fields?

Stu

[Nologic]

Well I think currently we have two cat's...tho I haven't looked at the db's recently...and I don't currently have Access installed. If I recall one was geared towards sports...and the other not...but I could be wrong.

However if is two...we would probably be better off making it a single field.

He didn't say "Hell No!" hehe but it didn't sound like its in the playing cards as of right now...and not looking good.

We all know Tom's looking at doing two possible additions that are rather intensive...and have been requested for some time...so probably once those are off the table he can review the request and see how possible it is.

More than likely it would come some time after v3 themes are done.

Yeah I'll see if I can put together a script tomorrow afternoon PST

[Stu1983]

Well I think currently we have two cat's...tho I haven't looked at the db's recently...and I don't currently have Access installed. If I recall one was geared towards sports...and the other not...but I could be wrong.

I see what your talking about now... currently we have 3 fields... we have Category, Sport and Non-Sport... GameEx only looks at the Category field so we shouldn't need to touch the Sport and Non Sport Fields though these could possibly be merged into a single Genre field or something?

I think we just need to rename the Category Field to "Multi Category" or similar and then add a new Category field for the AllGames categories

Stu

[Nologic]

Well may have to look at ether the script or the fields...but the current Category may very well be our pre done multi category...I may have just slopped them in there for the time being...which if I did...great.

If thats the case delete the other two and rename the current, and add another field and we are ready to start adding AllGames....well short of me writing a script that is.

[Nologic]

Okay managed to avoid writing the script tonight...but should have it by tomorrow.

I'll probably just dump everything to a text file for right now from AllGames....mainly as we'll likely need to use your fuzzy logic code again to zip through things at a reasonable pace.

[headkaze]

Okay managed to avoid writing the script tonight...but should have it by tomorrow.

I'll probably just dump everything to a text file for right now from AllGames....mainly as we'll likely need to use your fuzzy logic code again to zip through things at a reasonable pace.

Hey Nologic I just wrote a script that will scrape the entire AllGames website and output to xml files. It's still running (taking ages), if you want I can send you a copy of the files once it's done. Probably alot easier to process a bunch of xml files than the website directly. Up to you mate, but I'm really kinda scraping for the control info for CPWizard, your welcome to the data once I've finished.

[Nologic]

Yeah that would be great, thanks.

[headkaze]

Yeah that would be great, thanks.

No worries.. I'm currently up to "Sega Pico" that makes 27.9 MB of data scraped so far. Stay tuned

[Stu1983]

No worries.. I'm currently up to "Sega Pico" that makes 27.9 MB of data scraped so far. Stay tuned

I wouldn't mind a copy of that too please headkaze if you don't mind

Stu

[bkenobi]

That will zip down to about 300kb

[headkaze]

That will zip down to about 300kb

True, esp. using 7Zip text files are always good for compressing. Currently it's at 53.6 MB, but the "IBM PC Compatible" section is pretty big so the rip will take a while. I had to modify my scraper a bit to handle disconnections because before it would just start again if there was a disconnection on a system, this time I will have to have it continue off on the page it last tried to scrape. Still might take a day or two to finish.

[bkenobi]

Damn! A 50mb text file is pretty impressive. I've made bigger ones...but they are usually error logs from my analysis programs at work.

Out of curiosity, did you take into account the format of the file(s) so that it will be easy to import into a DB (e.g., csv or another flat DB format)? I haven't played with flat formats in a while and wish I knew more Being a .NET programmer, I'm guessing you chose XML.

[Nologic]

Yeah HK listed XML as the format...which I think Access can import from directly.

Tho I think Stu is going to work some magic on it and make things smooth as silk.

[bkenobi]

Should be pretty sweat! Where do these web sites get the info in the first place? Probably doing similar to what we've been doing here I suppose...

[headkaze]

The xml files are split up per system. So the largest file will be for PC games which is still being scraped.

Here is an example game from the PC xml :

  <Game>
	<Title>Grand Theft Auto</Title>
	<Platform>PC</Platform>
	<Rating>7</Rating>
	<ESRBRating Symbol="M">
	  <ContentDescriptors>
		<Descriptor Value="Strong Language" />
		<Descriptor Value="Animated Blood" />
	  </ContentDescriptors>
	</ESRBRating>
	<Genres>
	  <Genre Value="Racing" />
	</Genres>
	<Styles>
	  <Style Value="Mission-Based Racing" />
	</Styles>
	<Developer>DMA Design Ltd.</Developer>
	<Publisher>ASC Games</Publisher>
	<ReleaseDate>1997</ReleaseDate>
	<ControlType>Joystick/Gamepad, Keyboard, Mouse</ControlType>
	<Warnings>Animated Violence, Blood, Profanity</Warnings>
	<Flags>Voice</Flags>
	<IncludedInPackage>
	  <Item Value="2 City Maps" />
	  <Item Value="24-page Instruction Manual" />
	  <Item Value="Two Maps of The Cities" />
	  <Item Value="User's Manual" />
	</IncludedInPackage>
	<Synopsis>Grand Theft Auto is an action-packed excursion into the dark world of street crime. You romp through multiple cities on a crime spree in an effort to make the mob bosses recognize and promote you because of your ability and willingness to carry out any crime they might assign you. You start as a small fish in a very big pond and carry out the bosses' wishes by committing crimes such as car-jacking, murder, bribes, armed robbery (people and banks), extortion, revenge, road rage, thievery and more. 

You rise in the ranks to fulfill your dream of becoming a top mobster. You constantly receive instructions over pay phones located throughout the cities. To get the job done, you'll steal cars, listen to your police scanner to keep up with the cops and basically wreak havoc in your quest to become the best (baddest?) criminal in town. Over 200 missions and more than 6000 city miles are provided for your escapades. 

Grand Theft Auto has a high level of profanity and killing and deals with all aspects of the harsh world of crime. Whether pimping, robbing banks, slaughtering innocent bystanders or being the trigger-man on the crime of the day, you'll have a wide range of weapons to use and your choice of any car in the city as you hone your mobster skills. The game also features an original soundtrack appropriate to the action. Here's your chance to be the BAD guy!</Synopsis>
	<SimilarGames>
	  <SimilarGame Value="Death Rally" />
	  <SimilarGame Value="Grand Theft Auto -- Mission Pack #1: London 1969" />
	</SimilarGames>
	<Controls>
	  <ControlCategory Name="ACTION, DEFAULT KEY, CLASSIC KEY" />
	  <ControlCategory Name="TURN LEFT, LEFT ARROW, Z" />
	  <ControlCategory Name="TURN RIGHT, RIGHT ARROW, X" />
	  <ControlCategory Name="FORWARDS, UP ARROW, U" />
	  <ControlCategory Name="BACKWARDS, DOWN ARROW, I" />
	  <ControlCategory Name="BRAKE (WHEN IN VEHICLE), SPACEBAR, P" />
	  <ControlCategory Name="JUMP (WHEN WALKING), SPACEBAR, P" />
	  <ControlCategory Name="ENTER/EXIT VEHICLE, ENTER, W" />
	  <ControlCategory Name="ATTACK, CTRL, 9" />
	  <ControlCategory Name="NEXT WEAPON, X, TAB" />
	  <ControlCategory Name="PREVIOUS WEAPON, Z, LEFT SHIFT" />
	  <ControlCategory Name="SPECIAL, TAB, S" />
	  <ControlCategory Name="MAKE RUDE NOISES WHILE ON FOOT: S" />
	  <ControlCategory Name="HONK HORN WHILE IN VEHICLE: S" />
	  <ControlCategory Name="MOVE TURRET WHILE IN TANK: PRESS AND HOLD S KEY, SIMULTANEOUSLY PRESSING LEFT OR RIGHT ARROW">
		<Control Name="F1-F3" Value="send a message to an opponent during multi-player action" />
		<Control Name="F4" Value="send a message to ALL opponents during multi-player action" />
		<Control Name="F5" Value="tunes the radio to another station" />
		<Control Name="F6" Value="pauses the game (also gives status of your Target Score, Missions Completed and Secrets found)" />
		<Control Name="F7" Value="repeats the last pager message received" />
		<Control Name="F8" Value="toggles the Frame Limiter on/off during gameplay" />
		<Control Name="F9" Value="displays the current area name" />
		<Control Name="F10" Value="repeats the last message received" />
		<Control Name="F11" Value="calls up the on-screen Display Options during gameplay" />
		<Control Name="ESC" Value="lets you quit during gameplay" />
	  </ControlCategory>
	</Controls>
  </Game>

[bkenobi]

WOW! I didn't realize that DB had the controls and everything. That's a goldmine!

[headkaze]

Okay it's official the AllGame website is screwy as hell. Check out the "IBM PC Compatible" games.

Page 1 - Begins with "The $100,000 Pyramid"

Page 99 - Ends with "NHL Hockey [CD-ROM Classics]"

Page 100 to Page 109 - Begins with "Anarchy Online"

Page 110 to Page 119 - Begins with "Arcade Legends 3D"

Page 120 - Begins with "Art Attack: Digital/Art Attack: Comic Creator European"

Page 163 - Ends with "Battleship 2 - Replay European"

It only goes to page 163 so I can only scrap from A-N from the website. I think I'll send an e-mail to them and let them know their website is buggy. Hopefully they can fix it so I can do a proper scrape of the rest. It's a shame because their PC list is the most complete compared to every other system they have.