Class times and due dates

Reading quiz, live coding assignment, and lab assignment due dates:

Sunday, July 14, 11:59pm

We will have a Special Class on Kickstarting your data science career on Tuesday, July 2, 1:00-2:30pm in the Data Science Building room 305. This event is in person during our regular class time.

Our class live session for this module will be on Tuesday, July 9, 1:00-3:45pm in the Data Science Building room 305. You are expected to attend both classes in person if you are able to do so. I will be using Zoom to record our classes for future reference (see the video on the Home page for instructions on accessing these videos), and also to allow students to attend class if they are sick:

Zoom link: https://virginia.zoom.us/j/94503322732?pwd=BlxptMIeQfCIZNk7fAaxYWoy8jHGkC.1

Textbook, Video Lectures, and Slides

Please read Chapter 5 of "Surfing the Data Pipeline with Python": https://jkropko.github.io/surfing-the-data-pipeline/ch5.html

The following video lectures follow the discussion in chapter 5 of "Surfing the Data Pipeline with Python", and use these slides:

module5slides.pdf

Lecture 5.1: Legal, Ethical, and Practical Issues of Web Scraping

Lecture 5.2: Web Scraping with BeautifulSoup

Lecture 5.3: Putting HTML Data into a Dataframe and Building a Spider

External Readings and Reading Quiz

The readings for this module are

Official documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

Textbook: Nair, chapter 8 https://learning.oreilly.com/library/view/getting-started-with/9781783289554/

Blog: https://towardsdatascience.com/ethics-in-web-scraping-b96b18136f01

You may have these readings open as you take the following reading quiz:

Reading Quiz 5

Lab Assignment

You can find this module's lab assignment on Gradescope (don't forget the [instructions for submitting your assignments to Gradescope] ) :

Lab assignment on Gradescope

You can download the PDF version of the lab on Gradescope, but here is the .ipynb notebook file that you can use as a template to write your assignment:

labassignment5.ipynb

Live Coding Exercise

When you are ready, please submit your own Python notebook or script for this live coding exercise using the following link:

Live Coding Assignment: Module 5

Do you read movie reviews before deciding on whether to see the movie? Or maybe you do what I do -- I read reviews after having seen a movie, just to see if the critics agree with me.

Judging the quality of a film is a complex task because there are so many levels on which to engage with and judge a film. There's the plot, the script, the acting, the special effects and cinematography, the music, the costumes, and so on. Sometimes a movie excels in one area and falls flat in another. (Interstellar, for example, was a beautiful movie starring beautiful people saying stupid things.)

Film reviews are an art in and of themselves. Good critics can think about all the facets of a film and comment on how they work individually and as a symbiotic whole. And the very best critics avoid snobbishness while using elegant prose to convey to readers what the experience of the movie is like, and properly advise them about whether seeing the film is worth the time and money.

So it is a strange thing to boil down the complex process of reviewing a film to a number, scale, or binary value. Critics and newspaper editors have argued about whether to include a rating along with a review since 1928, when Irene Thirer first used a 0-3 star scale for rating movies along with her reviews in the New York Daily News. Some critics suggest that star ratings are too simple to convey good information about mixed reviews, and point out that two movies can arrive at the same rating despite providing very different experiences. Roger Ebert, who for many years reviewed movies by having epic arguments with his colleague Gene Siskel, wrote

I cringe when people say, "How could you give that movie four stars?" I reply, "What in my review did you disagree with?" Invariably, they're stuck for an answer. One thing I try to do is provide an accurate account of what you will see, and how I feel about it. I cannot speak for you. Any worthwhile review is subjective. If we completely disagree, my words might nevertheless be useful or provocative. If you disagree with what I write, be my guest. If you disagree with how many stars I gave it, you can mail your opinion to where the sun don't shine.

And among critics who use ratings, there's wide disagreement on whether to use a four star or five star scale, or even to use stars at all. The San Francisco Chronicle has for many years used the Little Man instead of stars: a little bald man who is jumping out of his chair, clapping enthusiastically, paying close attention, dozing off, or has left the theater.

Although, even the use of the Little Man is the subject of current controversy and debate.

Gene Siskel believed in a binary metric, the thumbs-up or thumbs-down, as Ebert recalled him saying: "What's the first thing people ask you? Should I see this movie? They don't want a speech on the director's career. Thumbs up--yes. Thumbs down--no."

In the past the most influential film critics were the ones employed by major newspapers in important markets, or ones with nationally syndicated television shows. But today, websites that aggregate these reviews and generate scores are dominant. The two most well-known aggregator websites are Rotten Tomatoes and Metacritic, both of which summarize all of the reviews for a film (or TV show or video game) to a score from 0 to 100. However, these two websites employ very different methodologies to derive the score. Rotten Tomatoes assesses whether each review is positive or negative, with no middle ground, and calculates the percent of positive reviews for a film out of all reviews. Metacritic assigns a score to each review denoting the degree to which that review is positive towards the movie, then takes the mean across all scores to generate its overall metric. These two approaches can differ in notable ways. Suppose a movie is universally reviewed as pretty good, but not outstanding. If all of the reviews are positive, then the Rotten Tomatoes score will be 100, a score that denotes an all-time excellent movie. But if each of these reviews is only slightly positive, Metacritic might assess the average rating across these reviews to be somewhere around 60 to 70.

In addition, both Metacritic and Rotten Tomatoes also compile ratings and short reviews from their website's users, leading to some movies having significantly lower audience scores than critic scores, sometimes for overtly sexist and bigoted reasons.

In this live coding exercise, we will be web-scraping data from the Rotten Tomatoes pages for all movies that are currently in theaters:

https://www.rottentomatoes.com/browse/movies_in_theaters/sort:a_z?page=8

Our goals are:

To check the robots.txt file for Rotten Tomatoes, to make sure web-scraping this webpage is permitted.
To use BeautifulSoup to get the following features for each movie:
To build a spider that finds the URL for each film's Rotten Tomatoes page, and extract the following additional features:
- Rating (G, PG, PG-13, R)
- Genre
- Original Language
- Director
- Producer
- Writer
- Release Date (Theaters and Streaming)
- Box Office (Gross USA)
- Runtime
- Distributor
- Sound Mix
- Aspect Ratio
- and the text that summarizes each review

While the point of this exercise is to practice the techniques involved in web-scraping, we can imagine some things we could do with the resulting dataframe. We can correlate the scores with budgets, rating, or genre, we can see if certain compositions of the cast lead to greater disparities between the critic and audience scores, we can mine the reviews for sentiment and compare that index against the critics' score, and so on.