Saving 8track's fandom playlists info

[wedonnotcare]

Hello! Given the fact that 8tracks recently announced that they will be closing down on December 31st, fans have been trying to save all the fandom playlists they can so they don't get lost when the website shuts down. Good news! There is a way to easily extract the information of all the playlists in a 8tracks tag. Here's a post explaining how to do it.
This post is meant to be a place where people who are extracting URLs from tags (of any fandom!) on 8tracks before the closing of the site can post their progress in a comment to avoid extracting the same tags multiple times. Not everyone extracting URLs will know about this post so it will be incomplete, this is just one attempt to avoid unnecessary duplicates. Anonymous commenting is enabled, so you don’t need an account to let us know your progress!

UPDATE:
Thanks to VidderKidder, we now have an excel spreadsheet that will archive the playlist info (note: the playlist text file will be in a json format, but using the spreadsheet is a super fast way to getting the metadata and the covers).

**********************
Download: http://bit.ly/8trackback
(you will need to have Macros enabled to run the spreadsheet)
https://powerspreadsheets.com/how-to-enable-macros-excel/

Use the first sheet tab to paste your URLs and click the button to run.

What it will do:
*Extract relevant meta data from the playlist page and put it in your Excel Spreadsheet
*Download the JSON file with the playlist details to a folder on your desktop (8tracks)
*Download the cover photo for each playlist to a folder on your desktop (8tracks)

What you should do:
*Paste urls on the first column of the first sheet tab

*Create a folder on your desktop called 8tracks

*Click the button.

*When you are done and you want to run a new batch copy the info to the second tab

*Clear out the old urls and relevant info on the first tab and paste in the new urls and run again.

Here's a video showing how it works: https://youtu.be/4WlI4yGW-8M

 If you have any questions you can contact morgandawn at admin@vidders.net

Comments

Tags whose URLs I've extracted as of 5:45 PM UTC

[(Anonymous)]

"Uhura" "Nyota Uhura"

[morgandawn]

Just a quick reminder that with the infinite scroll that 8-track's website, if your tag has more than 1000 results, it will only grab a thousand.

I asked archive team about this: "when we were working with the infinite scroll pages, we ran into a 1000 limit. It didn't matter whether we specified 1 per page or 10 or 20. Still was limited to 1000 results in total divided by however many pages"

As a result they dropped trying to archive by tags , instead they're running 3.5 million random numbers in the hopes of grabbing what they can . On the back end of 8tracks, the playlist data is not stored by URL, or by tag, but by unique numeric identifier.

Because of the volume that they have to run, the archive team will not be archiving the "look and feel (user profile icons). only basic metadata and the raw cover art

https://8tracks.com/mixes/6439105/tracks_for_international.jsonh

Supernatural pairings and fandom tags

[morgandawn]

Trying to do various combinations of supernatural or "spn". Biggest problem is that there are 13,000 supernatural playlists, and even when we add additional tags to narrow that down, we're still above the 1000 limit.

Supernatural pairings
destiel
deancas
wincest
sam/dean
dean/sam
the winchesters
winchesters
sabriel

Fandom
Supernatural
spn

[wedonnotcare]

- jean-luc picard
- christopher pike
- pavel chekov
- worf
- star trek discovery
- daforge
- jaylah
- leonard mccoy

[wedonnotcare]

- Spirk
- Chakotay
- Chulu
- Christine Chapel
- Janice Rand
- Montgomery Scott
- Amanda Grayson
- Kathryn Janeway
- Tom Paris
- B'Elanna Torres
- Elim Garak

[morgandawn]

Can you add this info to your post?

Thanks to VidderKidder, we now have an excel spreadsheet that will archive the playlist info (note: the playlist text file will be in a json format, but using the spreadsheet is a super fast way to getting the metadata and the covers).

**********************
Download: http://bit.ly/8trackback
(you will need to have Macros enabled to run the spreadsheet)
https://powerspreadsheets.com/how-to-enable-macros-excel/

Use the first sheet tab to paste your URLs and click the button to run.

What it will do:
*Extract relevant meta data from the playlist page and put it in your Excel Spreadsheet
*Download the JSON file with the playlist details to a folder on your desktop (8tracks)
*Download the cover photo for each playlist to a folder on your desktop (8tracks)

What you should do:
*Paste urls on the first column of the first sheet tab

Create a folder on your desktop called 8tracks

Click the button.

When you are done and you want to run a new batch copy the info to the second tab

Clear out the old urls and relevant info on the first tab and paste in the new urls and run again.

If you have any questions let me know.

Just reply to this email. My email is admin@vidders.net.

I have assisted on projects to backup tumblr, Yahoo Groups and now 8tracks.

I am the admin of vidders.net the largest streaming vidding network online.

You can share this sheet with anyone who would like to do the same thing you are about to do

Again if there are any questions just email.

[(Anonymous)]

I'm adding this (and your above tag reminder) to my post on the macro over on Tumblr. -Meg starfleetdoesntfirefirst

[arcadianmaggie]

-Merlin
-Arthur Pendragon
-Merthur
-BBC_Merlin
-inception
-arthur/eames
-one direction
-larry stylinson
-The Social Network

[morgandawn]

If you have a list of the urls, email them to me morgandawn@gmail.com and I can have them run through the wayback machine

[(Anonymous)]

- A Series of Unfortunate Events / asoue
- Doki Doki Literature Club / ddlc
- Ava's Demon / avas demon
- Undertale (as much as possible considering there's >3000)

[haemophilus]

A lot of It's Always Sunny in Philadelphia tags - whatever I have time for.

Arrested Development

Transgender

I don't think I'll have time for anything else.

[morgandawn]

If you have a list of the urls, email them to me morgandawn@gmail.com and I can have them run through the wayback machine

[(Anonymous)]

Is anyone doing smaller SPN characters and ships?

So far I've collected links for variations of

Dean/Benny
Dean/Lisa
Sam/Castiel
Bobby/Crowley
Sam/Jessica
+some tiny ships

Plus whatever characters I can think of

Also got about 2,300 out of 3,600 for Dean Winchester

+ Almost Human (fandom)

[elendraug]

Thanks for looking out for smaller ships, too. Hook me up with the SPN rarepair list if no one else is doing it already and I can try to run the script to download if the site's still up tomorrow.

I'd personally love to see Crowstiel, Crevin, and Calthazar saved as well if you know if you got those.

ETA 4PM 1/1: I'm noting my progress on smaller SPN tags in the comments below, of the above I haven't yet gotten to Lisa Braeden, anything with Castiel other than the "Team Free Will" tag, or anything with Jessica Moore. I pulled everything under Bobby Singer, Crowley+Supernatural, and Benny Lafitte/Benny+Supernatural, so I think that covered those you listed above.

Vid show Playlists

[morgandawn]

we were able to pull 3000 out the 5000 existing playlists tagged fanfic and fanfiction

Another Tip: Collections

[morgandawn]

We're testing this but the 1000 limit does not seem to be as strictly enforced when scrolling through collections.

Ex:
https://8tracks.com/super_redhead/collections/a-big-star-wars-collection

https://8tracks.com/vaguelydownward/collections/star-trek

Collections are hard to find so if you come across any mention them here

Re: Another Tip: Collections

[haemophilus]

I had a hard time running my collections through the spreadsheet and ended up having to use the tag

Anime

[morgandawn]

a few thousand tagged with anime/. if people have specific anime fandoms, leave them here

Re: Anime

[chapbook]

Spice and Wolf
Aoi Hana
12 Kingdoms
Samurai 7
Kase-san series
Noein
Escaflowne
Haibane Renmei

Larger numbers of fanmixes, from which I've been grabbing just samples from atm. Feel free to contact me if you want to split any up:

Natsume Yuujinchou
Yuri on Ice
Mushi-shi
Princess Tutu

[paperairplanesopenwindows]

I've grabbed Clintasha links, but I haven't gotten the Macro to work (my excel is old though)

[vidderkidder]

Yeah, sorry to hear. There may be issues with older versions of Excel as this was created on Excel 2019. It should work with most of the newer versions like 2013, 2016. Just share them with me if you'd like and I can run them. My email admin@vidders.net

[tehhumi]

I'm going to post a series of comments as I do batches, but for now I've got:
- agent carter
- peggy carter
- rogue one
- feanorians

[tehhumi]

- celebrimbor
- silmarillion
- the silmarillion
- elrond
- winterhawk
- stevebucky (chosen over 'stucky' because it squeaks under the limit with 996 songs)
- girl genius
- robin mckinley
- patricia briggs
- discworld
- gunnerkrigg court
- tolkienreadalong

I'll grab major lord of the rings characters tomorrow, I don't know whose tags will be too big.

[(Anonymous)]

I've grabbed "macdennis" and "wicdiv"

[(Anonymous)]

ok so for Undertale I grabbed:
- undertale
- chara / chara (undertale)
- sans
- frisk
- mettaton / mettaton ex
- genocide route
- pacifist route
- papyrus
- asriel dreemurr
- flowey
- undertale spoilers
(basically all the top tags)

so I think that covers most of it

[morgandawn]

If you have a list of the urls, email them to me morgandawn@gmail.com and I can have them run through the wayback machine

[razyrs]

i'm currently planning to get links and download info for
-night in the woods
-we know the devil
-wktd
-fallen london
-echo bazaar
-echobazaar
-sunless sea
(i'm thinking of getting more, but the computer i'm using for this is really slow so we'll see.)
-rick and morty

[(Anonymous)]

Got up to 6463 links in Supernatural (excluding variations of destiel, wincest, "spn/supernatural") out of 9k+ (or 13k?)

Also did:
- Lucifer (FOX)
- Good Omens
- Sense8

But I'll have to find someone to grab the info because I can't run the macro

[(Anonymous)]

Also grabbed links for

- Buffy The Vampire Slayer

[(Anonymous)]

My computer isn't the fastest, but I'm getting what I can.
Currently working on the Les Mis tag (it does have over 1000, so I'm missing some). Got everything in the Combeferre tag as a test run to make sure I was doing it right. I'll probably try to grab some Classics-related tags as well.

[(Anonymous)]

Got the first 1000 in the Les Mis tag (sorted by trending) and everything in the Alexander the Great tag.

[morgandawn]

edited: if you are using vidderkidder's Excel VBA macro, he did a tutorial video
 https://youtu.be/4WlI4yGW-8M

[morgandawn]

If you have a list of the urls, email them to me morgandawn@gmail.com and I can have them run through the wayback machine

[comely_wench]

hey! let me just say that you're all doing the lord's work. furthermore:
"black sails" is ran through with the macro! there are some small benefits to being a small fandom.
"enjoltaire" is ran through meanwhile

edited because of the noob questions, hah

[comely_wench]

i think i'm going to periodically post updates, since some of these are big.
other stuff i ran through:

- valvert
- the terror
- dorian pavus
- phantom of the opera
- the vampire chronicles
- shakespeare
- fantastic beasts and where to find them
- barduil
- severus snape
- captive prince
- the witcher

(okay, two noob questions...why is the macro giving me error 500 on some playlists, is it just because 8tracks itself is giving me that page sometimes? i imagine they're facing quite a lot of traffic...re-running the macro fixes it usually, but i'm too slow as it is :d and i don't always notice
and i'm also experiencing the 0 byte json files thing -- i don't know if that is something that can be helped :( if i have time i'll just find them by id and download tracklists manually though, but i don't know how much time i have left.
my email didn't went through either, but i suppose i could be counted lucky for that since some of my questions were kinda stupid)

[(Anonymous)]

Have run the following tags, and once the last lot have finished processing will probably not have a chance to do more:
- fe* (where * is 1-15) (also fire emblem awakening)
- ryu ga gotoku (also yakuza, kiryu kazuma, goro majima, yakuza 0)
- the world ends with you (also twewy)
- noragami
- mdzs (also mo dao zu shi)
- kingdom hearts [whatever number it is]/4 days, chain of memories (roxas, ienzo, zexion, vexen)
- eragon
- time riders
- run lola run
- watership down
- bravely default

+ some random stuff I had in collections. There's a bit of steven unvierse and a bit of dragon age, some tolkien-related things, and some random other splatterings. Possibly inception? I didn't check it fully, just pulled the links when I was playing around.

Have also run them through the wayback machine using the email, just in case.

Also, mostly with the noragmi ones but some others, I have noticed some of the files come out with a size of zero. Opening them as text reads them empty. The pictures still download and the spreadsheet list fills in, but the tracklists and all that are not in the files. I've tried rerunning them and no banana. The unifying link between them seems to be that they all contain non-european characters (odd fonts, love hearts, Cyrillic, japanese characters, etc).

[(Anonymous)]

Was still working this morning so added
- rune factory (also rune factory 4, rf4)
- harvest moon
- discworld
- earthsea

Running the Excel VBA Macro

[vidderkidder]

Notes (Things I've discovered):

1) Won't work well with Mac versions of Excel.
2) Works better with the playlists links. What do I mean? There are different kind of playlist links. Version 2 got me results while version 1 returned "Not found"

Version 1 - https://8tracks.com/talathamanya/we-move-lightly#smart_id=tags:clintasha&play=1

Version 2 - https://8tracks.com/talathamanya/we-move-lightly

Vid show Playlists

[morgandawn]

correct that version 2 is better. In the scrape your links tutorial link above we tell people to remove everything after #smart_id=

[morgandawn]

If anyone is done with their fandoms and has room/time to run more through the Excel spreadsheet, send me an email at morgandawn@gmail.com

Ones I need help with (warning most are large and will take hours to run on a good machine)
*Homestuck (small)
*LOTR (3ooo groups, ready to go)
*Marvel (11,000 groups ready to go)
*random set of 'Goofy" - 200 multiple fandoms (small)
*Star Wars 2500 ready to go
*Good Omens (pending)
*Sense8 (pending)
*Buffy (pending)
*Another set of random ones - we need more keywords and pairings for Firefly/Serenity because the search I used on this one pulled in a lot of non-fandom
*Roleplay (pending..don;t know the size but it will be large)

UPDATED 08:00 UTC 1/4/2020 | Various fandoms & coding progress

[elendraug]

TIME IS RUNNING OUT. NEW WORKBOOK WITH IMPROVED FUNCTIONALITY SHOULD BE AVAILABLE ASAP ON JANUARY 4TH.

If you are ready to submit data, please email 8tracksbackup AT gmail DOT com ASAP with the following, and we'll send you a Dropbox link:

1. The name you want us to call you
2. How to reach you (presumably the email you send it from, but you can also list AO3/dreamwidth/tumblr/twitter/etc)
3. What fandoms/tags you gathered info for
4. If you backed up URLs and still need help running the macro
5. If you ran the macro successfully and confirmed you were not missing data (0kb text files, missing images, wordwrap issues, etc)
6. Whether you were signed into an account when pulling data, or if you think your results were limited to SFW playlists only
7. If you want us to reach to you to continue helping with this project (we will still contact with questions about your data set)
8. Any other questions you have

We will continue working out details once we've saved as many playlists as possible but data preservation is our priority for now.

THANK YOU, THANK YOU, THANK YOU!

IF YOU ARE AN ADULT AND YOU ARE COMFORTABLE BACKING UP NSFW PLAYLISTS, PLEASE BE SURE YOU ARE SIGNED INTO AN ACCOUNT WITH "SAFE BROWSE" TURNED OFF, OR YOU MAY BE MISSING PLAYLISTS THAT ARE MARKED AS EXPLICIT AND NOT REALIZING IT!

PLEASE CONFIRM IF YOUR TEXT FILES (ANSI) ARE SHOWING UP AS 0KB - WE HAVE AN UPDATED SCRIPT YOU'LL NEED TO RERUN TO CAPTURE TEXT IN UNICODE!

YOU MAY BE MISSING IMAGES! CHECK IF YOU HAVE AN EVEN NUMBER OF FILES IN THE FOLDER!

New versions are forthcoming with tools to help recapture files affected by these errors!

---

ETA 0: I've got something like the first 1000 Homestuck ones so far. It's still running and it'll be a while, but we're in progress.

(I'm already in contact with morgandawn but I'm posting here for the benefit of those reading.)

⛎🎉🎊 Cheers, Homestucks. 🎊🎉⛎

ETA 1 (11PM 12/31): Got the entire Homestuck list provided to me plus the dirkuu tag, caliborn tag, Mortior's playlists, Schellibie's playlists, a handful of others. Going to try for any additional cherub content, intermission fandom, etc.



ETA 2 (3AM 1/1): Grabbed cherubs, midnight crew, the felt, intermission, specialsari, bentsage. Calliope will take more sorting to separate out non-fandom before running.

Running overnight for six tabs provided to/from morgandawn for Firefly, Mad Max, Venom, Gossip Girl, Criminal Minds, Wonder Woman.

Done Intending to run tomorrow for Metalocalypse, Supernatural rarepairs. Other fandoms I'm familiar with are on my profile here on Dreamwidth, or on AO3 if anyone has a list ready to go and can't run the macro.

For Firefox, the add-on Link Extractor has been useful for this project, and obviously the Internet Archive/Wayback Machine extension

Thank you to everyone who's helping with this. I'm pretty fed up with Q4 meaning the collapse of various fandom platforms

ETA 3 (10AM 1/1): Successfully ran macro with list provided to me for Firefly, Mad Max, Venom, Gossip Girl, Criminal Minds, Wonder Woman.

Ran macro for Metalocalypse and began running for smaller ships/non-Winchester non-Castiel characters for SPN as follows:

SUCCESS, needs some 0kb and missing image cleanup Calthazar, Crowstiel, Crevin (none) Kevin Tran, Linda Tran (none), Mama Tran (none), Charlie Bradbury, Andy Gallagher, Bobby Singer, Jody Mills, Donna Hanscum (overlaps with Jody Mills), Claire Novak, Rowena MacLeod, Rowena, Jack Kline, Jack+Supernatural, Kelly Kline (none), Ellen Harvelle, Jo Harvelle, Ash+Supernatural, Benny+Supernatural, Bela Talbot, Gordon Walker, Azazel+Supernatural, Meg Masters, Megstiel, Ruby+Supernatural, Anna Milton, Abaddon+Supernatural, Balthazar+Supernatural, Crowley+Supernatural, Garth Fitzgerald IV, Adam Milligan

ETA 4 (4PM 1/1): Threw together a spreadsheet to help automatically clean up URL lists (HTTP>HTTPS, filtering to &play, removing text past # as a delimiter, quickly copying and then clearing text). Still potentially buggy but if I can get it stable I can share.

Working with morgandawn and vidderkidder to potentially identify what non-standard characters are throwing off the retrieval and resulting in 0kb text files; using Xenosaga/Xenogears as a test case, presence of Japanese text definitely results in failure. Possibly plain text vs unicode issue?

- 0kb ANSI vs Unicode issue in progress fixed, version 4 coming soon
- Working on updates to macros including automatic link formatting
- Images missing due to script seeking PNGs; being expanded to other file formats fixed, version 4 coming soon
- extract list of IDs from windows folder, then run those IDs to download missing json text (to quickly re-run Japanese/Cyrillic/etc character issues)

LIST GATHERED BY ME, MACRO TO BE RUN BY ME:
Galerians
Heavy Rain
Homestuck: Calliope, cherubs
Homestuck: working with a friend to completely redo the dataset to grab as many as possible of 8778 playlists on 8tracks; we are going by character first for beta kids, alpha kids, beta trolls, alpha trolls, cherubs, ancestors, guardians, carapacians, intermission, sprites, etc.
Supernatural subsets (Becky Rosen, Chuck Shurley, Amara+Supernatural)
Final Fantasy 1,2,3,4,5,6,7,8,9,10,10-2,11,12, Tactics and various naming conventions thereof (e.g. Final Fantasy IX, Final Fantasy 9, FFIX, FF9)

RUNNING MACRO:
homestuck alpha trolls, ancestors, guardians, carapacians, sprites

LIST GATHERED BY ME, MACRO ALREADY RUN BY ME, MAY NEED 0KB TEXT/MISSING IMAGE CLEANUP:
Ace Attorney (977)
Homestuck: Caliborn, dirkuu, the felt, midnight crew, intermission, various others (156+)
Metal Gear/Metal Gear Solid/MGS (351)
Metalocalypse (106)
Supernatural: "Team Free Will" (195)
Xenogears & Xenosaga (39, definitely needs missing text cleanup, missing images handled)
Silent Hill (329)
South Park by popular, 988 of 1,215 (exceeds 1000 limit, will try by trending/newest and remove duplicates if time permits)
Supernatural: various characters/rarepairs (many listed above)
Supernatural: Sastiel (77)
Mass Effect (1329/1819 playlists, missing 490)
Narnia/Chronicles of Narnia (~350)

LIST GATHERED BY OTHERS, MACRO ALREADY RUN BY ME, MAY NEED 0KB TEXT/MISSING IMAGE CLEANUP:
Homestuck (3078+ out of 8778, still working on it)

6 TABS (1831 rows) for:
Criminal Minds ((328)
Firefly (157)
Gossip Girl (624)
Mad Max (488)
Wonder Woman (192)
Venom (42)

JSONs so far

[camilasc]

hello, I have been at this since yesterday...have gotten a couple errors but I'm just trying to grab as much as possible... (didn't even realize some files were at 0kbs till I read one comment here ughh). I think most have saved fine though?

I have done:

-xena warrior princess
-femslash
-supercorp
-supergirl
-eurovision

And I'm trying to get *everything* from the doctor who tag. I got everything that wasn't tagged with a distinct tag, and have done these <1000s: doctor x rose, romana, romana x doctor, academy era, mickey smith, vastra/jenny, sally sparrow .torchwood, trock, osgood, classic who, faction paradox, big finish, doctor who eu . koschei .trock .companions .the rani . cybermen . charley pollard . dw . the companions the ponds .rory williams . gallifrey . bad wolf . tardis . clara x danny .twissy .time hunter .tentoo x rose . blake’s seven .

still got a lot more to do but, keeping at it... I hope someone gets some Glee and Harry Potter ones. I sadly only have time to do 1 big fandom ):

Re: JSONs so far

[(Anonymous)]

I'm grabbing the first 1000 of Harry Potter + popular

I probably won't have time to do more than that.