I was worried that getting this data would be difficult,
but unironically, they literally have a single div for all
of the text in the article and a single div for the header
that contains the article name, author and date released.
IT'S LIKE THEY WANTED ME TO SCRAPE THE SITE.
Why such a conclusion, and why is this "unironically"? A normal blog page, MIT or not, which is not obsessed with monetization or showing ads after each paragraph, is literally just a text with some very minor formatting (adding a few HTML tags here and there). It's very much expected that you would have a single <div class="content"> and an actual content inside of it.
¯\(ツ)/¯ I was expecting it to be vauge and or have many divisions that would have made scraping harder, I was surprised that it did not. I also just have not had that much experience with large professional sites like that before
Actual Title: MIT Webscraper
"...please use the original title, unless it is misleading or linkbait; don't editorialize."
https://news.ycombinator.com/newsguidelines.html
My bad, bad decision on my part, its fixed!
¯\(ツ)/¯ I was expecting it to be vauge and or have many divisions that would have made scraping harder, I was surprised that it did not. I also just have not had that much experience with large professional sites like that before