Another MP Database!

[Rolf]

Hi everyone! Well this is weird. It seems great minds do think alike. I haven't been on this forum for a while, but recently I saw Blashyrk's thread (https://motorpsycho.fix.no/comeonin/forums/topic.php?id=10831) about the MP setlist database site he is building, and was quite surprised, because coincidentally I have also been working on a very similar project, a Motorpsycho Setlist Database! Although I wish I had seen the other posts earlier, because that might have saved Blashyrk from doing a lot of work.

My vision has been to make the ultimate MP setlist database, as complete and correct as possible, and I've been working on it for over a year (not full-time, obviously…). I've been putting off announcing it because I wanted it to get to a more finished state, but since we are now two people working on two separate, but very similar projects, I figured I should announce mine as well. Maybe we can join forces and cooperate? I'm not trying to steal Blashyrk's thunder here.

The main motivation for my project was the fact that a complete and correct MP setlist database didn't exist, so that's what I wanted to create. A lot is here on motorpsycho.fix.no and the forum, and on Dampsaba. There's also a lot of data on setlist.fm, but I absolutely hate that site, mainly the interface (including, but not limited to auto-playing videos). Another problem is that anyone can edit setlists on setlist.fm, which is both good and bad, the bad being what seems to be a lack of standardization, so everyone seems to come up with their own standards. I've personally corrected errors there, only for someone else to come along and change the correction back to (what I consider is) wrong!

Obvious examples are spelling and capitalization, and how to represent segues and partial performances. For example, the old Superstooge -> The Wheel combo. Is it "Superstooge -> The Wheel"? "SuperWheel"? "Super – Wheel"? "Super/Wheel"? I found all kinds of variations on matters like that on setlist.fm – so I'm happy to see that Blashyrk and other people in his thread have the same attention to these details as I have! In my project, Superstooge -> The Wheel is stored as two performances, one for each song, and Superstooge will have a segue flag set. This is modeled on how Grateful Dead (and other jamband) setlists are often written. I strongly believe it is wrong to store such song combos as one performance/title, but I know other setlist geeks probably disagree. In any case I'm planning to add support for handling songs that have alternate titles, and that will also let me handle song combos/sequences nicely. I will also add other flags/fields to the database, for example for partial performances of a song.

Another consideration for a project like this is what should be considered the sources of truth. A known complete recording is obviously the best source. The Dampsaba is another great source, since it comes straight from Bent/MP, although I have found a couple of instances where the Dampsaba setlist didn't match up with recordings or other sources. Then we have user/audience contributions, like posts here on this forum. A lot of the time (probably most of the time) these are correct, but sometimes all we have is someone posting something like "I was a bit drunk, but I remember them playing X, Y and Z, not sure if correct order". So because of all this I have started storing the state of each setlist – i.e. if it's known to be correct, incorrect, incomplete, etc. – and what the source is. (This is not yet shown in the interface/frontend.)

I see Blashyrk is entering all the data manually in his database. I didn't want to do that – we have computers for doing such jobs. Therefore I built an XML file from various sources, mainly setlist.fm's API. Using their API I could download all MP setlist data from setlist.fm in XML format. (Since my project is non-commercial I believe this is ok. It's public data after all, why re-invent the wheel? All setlists from the 90s seem to be taken from Dampsaba anyway. I'll add credit where credit is due.) I then spent a lot of time going through this data, correcting errors, adding missing data, standardizing, etc. – and I'm far from done checking the correctness of each setlist. So quite a bit of manual labour still.

To verify a setlist's correctness I have been comparing the entries in my giant XML file to all other sources I have found. This obviously takes a lot of time.

Anyway, to automate the database creation I wrote a program which converts the XML master data to an SQL database. I think that is pretty cool!

I then wrote the backend and frontend for the website. I made the frontend as a "single page application", designed to be fast, responsive, flexible and intuitive. (I should note that another motivation for this project was to teach myself modern web development, so many things can and will be improved. I have done everything from scratch, but I'm obviously using existing frameworks like Laravel and Vue.js)

And in all humbleness I must say I think my frontend/user interface is quite awesome. It takes a few seconds to load, but after that it's smooth sailing. The filtering feature is a thing of beauty imho.

I spent a lot of time working on the frontend, so the database itself doesn't have that many features yet, but here are some:

– Segue markers

– Automatic detection of first time a song is played

– A few statistics, like how many times a song has been played and most recent performance (more stats will be added).

– Times played for a song takes into account situations like "Hogwash -> Halleluwah -> Hogwash" – in this case Hogwash is only counted as one performance.

There are currently 977 setlists in my database! That's actually slightly more than setlist.fm.

I have many plans for this project, many/most of which overlap with those in the other threads. One cool thing will be an API so that all the data can be easily consumed/exported in various formats and used by other projects. But anything is possible!

Anyway, I'll keep on working on this project, and if @Blashyrk wants to collaborate then I'm open to that, but I suspect we are using very different approaches, so it's perhaps not feasible to join these twin projects into one. But that's ok Worst case scenario we get two MP databases!

Finally, my project is currently available at http://mpdb.herebemonsters.network

Note: It's running on a tiny free Amazon cloud server, and I don't know how much traffic it can handle, so be gentle. (also, the "add your own quick filters" feature currently doesn't save your quick filters across sessions.) Also it loads a random setlist the first time you enter the "Concerts" view. And you can't register a user, so you can't log in.

Any feedback is welcome, but again remember that the site is far from done!

EDIT: the site is down for now. The free server stopped being free, so I need to look into what options exist out there for free/cheap hosting.

[Punj Lizard]

Fantastic! I hope you and Blashrykh can collaborate in some way to make it easier to realise your individual visions of a setlist database.

[pfnuesel]

I really hope you guys would merge the two projects. Merging is hard, but it will only become harder with time.

[Blashyrkh]

I think it's going to be impossible to merge both project, because both of them are built in another technology. Also the way the databases are built is completely different.

[Rolf]

@Blashyrk Yeah, that's what I feared, I got the impression you are using a very different approach/technology. Great work anyhow, keep on truckin!

I'll probably open source my project when it's more mature.

[The Other Anders]

I love both of them. Please keep up the good work, both of you! Thanks :MPD:

[Author]

Posts