RetroBox Scraper: Brainstorming

A front end independent rom metadata, image and video scraper.

About a year ago I created a game scraper as part of my custom arcade front end. Since developing the front end has been put on hold for the time being, I’m going to separate these two projects and continue to focus on the scraper as an independent tool. I’ve already used it for scraping my own rom collection and it works really nice, but it’s still far from being finished unfortunately.

For now the software is a command-line only tool for Windows, but I intend to create a nice looking UI for it once all other features are implemented. I think the tool should be able to parse a rom directory or a gamelist from different front ends, search online for information and save it in whatever format the user needs. The scraper will essentially be a gamelist viewer and converter with the possibility to gather all data from the web.

RetroBoxScraperObject

Providing the “Name” and “System” is enough to gather the rest of the data

Features

  • Import a list of games from a rom directory, a HyperSpin database or a EmulationStation gamelist file
  • Fast and fully automatic scraping of all necessary game information (look at the screenshot above), artwork and video
  • Export scraped data to any format/gamelist

User interface

I haven’t started on the user interface yet, but I guess it will look similar to OPLM or the Wii Backup Manager. If you have a brilliant idea I would love to hear it or see some concepts! 🙂

Providers

The software currently supports the following online sites:

 

Are there any features you would like to add to the list? Do you have suggestions for the UI? Let me know using the comment section below! If you would appreciate a tool like this, please consider subscribing. Stay tunes for more updates!

 

Share this:

2 comments, add yours.

CriticalCid

Hey I just found your blog and your idea sounds awesome!

I would love to use this tool for my favorite FrontEnd LaunchBox: https://www.launchbox-app.com/

They have their own database and a Wikipedia scraper (which doesn’t work that good) but I really would like to fill in all the gaps with data from Mobygames, Giantbomb, etc.

Some features I would really like to see:

– Configurable preferences for scrapers:
It would be nice to have the option to change the priorities for the different scrapers similar like the ROMCollectionBrowser for Kodi does. Screenshot: https://9a1f45d9a1d700c98c266bb31b9545f5435ee5a0.googledrive.com/host/0B-9ewpmLdTHqb25qSFpDWUJyYjA

So it would basically search for artwork and metadata from the first scraper the user has chosen. If the scraping is done and any artwork or metadata is missing it will jump to scraper #2 and look there for missing artwork and metadata. And so on and so on until all metadata and artwork is present or the last scraper is finished. The only thing that nagged me in RCB is that it features more scrapers than you can select for scraping. It would be nice to have the opportunity to use ALL available scrapers and not only just 3 like in RCB.

– Find the right cover according to the region tag in the ROM name:
Most ROMs from preservation groups like No-Intro or Redump have a tag for the region in the file name. It would be awesome if your tool would find the right cover based on that tag.

– Configurable preferences for the exact region/country for European and multi-region games:

This is more or less a combination of the other 2 suggestions. For example if I want to scrap a game with the region tag (Europe) in it I would want to specify from which European country I would prefer the artwork first.
As I’m from Germany I would prefer the German artwork. If it doesn’t find any German artwork for the game I would prefer the cover from the United Kingdom, next the cover from France, etc.
Same goes for multi-region versions like (USA, Europe).

– And of course the option to import everything into LaunchBox 🙂

Finally some databases I would recommend to add as well (they all have an official API):

– Steam: For PC games. It would be especially nice to get the trailer for the games as well as EmuMovies doesn’t have that much gameplay videos for PC games.

– Co-Optimus: They have great detailed info’s about the co-op and multiplayer capabilities (local and online) http://www.co-optimus.com/

– IGDB: The guys over there basically try to create an IMDB for games https://www.igdb.com/

-VNDB: A database specified on Visual Novels. https://vndb.org/

Benjamin Krause

Benjamin Krause

Author

Thank you so much for that comment. A lot of great suggestions and I’ll try to take a closer look at them in the coming days. Been on vacation for quite a while now, so sorry about my late approval/comment. 🙂

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.