Omega back-end and game scraper

The past couple of days have been dedicated to the back-end of Omega, the front-end I’m developing for my bartop arcade. It has been a whole lot of work for one person, but the result will be awesome.

There’s not much left apart from adding the artwork downloader and further optimizing the code for even faster scraping. Just a few days ago the scraping process would take days to complete, but now it’s only a matter of hours from you launch your arcade for the first time until you have all the info you’ll find about any game.

Omega ScraperWhile the scraper is still not 100% done, it’s mostly functional. It takes about 20-30 seconds to read all of the locally stored HyperSpin XML’s files and building the sqlite database with 20k games.

Upon triggering a scrape, only the games and systems missing information will be processed. In order to save as much time as possible, escpecially when scraping the huge collection of MAME, files like nplayers.ini and GameInfo from HyperLaunch will be used in order to fetch most of the data offline.

Data that can’t be found on disk will of cause be downloaded automatically. Supporting many of the major game databases such as TheGameDBMobyGamesEmuMovies (Subscription needed) and GamesDBase (also working on GiantBomb support).

There’s nothing more awesome to watch a really process like this fully automated. 🙂

While currently a simple command-line tool, this application will eventually be tightly integrated with the rest of Omega.

Share this:

3 comments, add yours.

Avatar

Adam

is the scrapper available for download…?

Benjamin Krause

Benjamin Krause

Author

Not yet, but progress is made, so I hope to release it in the near future. Let me know if there are any particular features you would like to see in the initial release. 🙂

Avatar

Dave M

Please release this!

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.