IMDb datasets: 3 centuries of movie rankings visualized

The Question

I am a sucker for IMDb ratings so don't judge me. They are my priors before watching almost anything on a screen (home screen that is). But between movies (feature films), TV movies and TV (mini) series IMDb ratings are highly inconsistent. For example, series The Boys has rating 8.7 and so does movie Goodfellas by Martin Scorsese. Does it make sense The Boys ranked as high as #16 rated movie title in the whole IMDb database (among those with at least 25,000 user votes)? Or, in other words, if and how much apples vs. oranges those ratings are?

 

Rating Distributions

To start I downloaded IMDb datasets (here). Let's show distributions of title ratings depending on the types: movie (i.e. feature film), TV movie, TV mini series, and TV series between fiction and documentaries:

Title ratings drift towards higher values depending on their types (shown on the right): movie, TV movie, TV mini series, and TV series. So indeed ratings of movies and TV series come from different distributions representing different things like apples and oranges. But how much different they are? (we will focus on fiction titles only from this point on.)

 

Percentiles

If a title has all time best rating then no doubt it's worth giving a try (let's say among titles with at least 1000 votes - number of votes is rather important consideration but we let it slide here and may come back to votes later). Why? Because 100% of other titles are rated below or at best the same and that indicates exceptional qualities. In statistics such rating has a name: 100th percentile. Following the same logic 99th percentile represents rating above 99% of all titles in the database (again, don't forget about minimum threshold for number of votes to be considered).

Based on above we can assign IMDb titles to groups based on the highest percentile they belong to: 99% percentile suggests that the title is very best, 95% - excellent, 90% - very good, 75% - good, 50% - average, and 25% - bad. Feel free to assign and name percentiles differently in your analysis but we stick with this convention for this post. Last piece of the puzzle is taking percentiles not across whole IMDb set but rather for each title type separately and compare them:

Going back to our example, 8.7 in TV Series places The Boys firmly in "Excellent" (95th percentile), while Goodfellas at 8.7 sits at the top of "Very Best" (99th percentile) in movies - noticeable difference between the two.

The difference becomes even more meaningful when looking at the lower tiers "Very Good" (90th percentile) and below: while rating of 7.6 suffices for a movie (e.g. Love Actually) to place in "Very Good", a TV series must achieve rating of 8.4 to qualify for the same 90th percentile. In fact, a TV Series with 7.6 rating (like Grey's Anatomy) places just above "Average" 50th percentile. Furthermore, the rating of 8 would place a movie firmly in top 5% while the same 8 for a TV series barely cracks top 25%.

 

Percentiles Extra

Comparing and analyzing ratings between title types can be helped by organizing and visualizing the same percentile data in a few different ways:

• Overlapping bar charts by title types:
  
  
  
  
  

• Line chart by title types:
  
  
  
  
  

• Line chart by percentiles:
  
  
  
  
  

What About Documentaries?

The title percentiles above excluded documentaries. To be able to compare ratings between fiction and documentary titles the following visual computes and dissects rating percentiles between fiction and documentaries by title types:

For whatever reason IMDb users rate documentaries more generously than their fiction counterparts across all title types.

 

Historical Perspective Mixed with Film Trivia

The oldest film on IMDb is Passage de Venus made in 1874, is ranked 6.9 with 1282 votes (as of January 2020), and is filed under title type short and genre Documentary. In chronological order it is followed by 2 titles in 1878 (short animation Le singe musicien and short documentary Sallie Gardner at a Gallop), 1 in 1881 (short documentary Athlete Swinging a Pick), 1 in 1883 (short documentary Buffalo Running), and 1 in 1885 (short animation L'homme machine). Starting with 1887 that cranked up 45 titles total there are no more gap years, but such production feast will be surpassed only 1894 with 97 titles. First movie title (and only that year) Reproduction of the Corbett and Fitzsimmons Fight was filmed in 1897 under Documentary, News, and Sport genres. Lastly, first year when total number of titles exceeded its year numerical value is 1952 with 2059 shorts, movies, etc. under the belt. Did I just say last? One more factoid if you excuse me: movie production in 2020 (35,109 titles total) dropped us exactly 10 years back when 35,062 titles were produced in 2010, while the absolute record belongs to 2017 with 51231 films total.

What about visualizing film production over time?

 

Final Thoughts

IMDb dataset turned out to be richer and deeper than I expected and I just scratched the surface. There is plenty to play with - genres, runtimes, adult movies (yes, probably for compliance IMDb flags each title as adult or not), and, of course, ratings. IMDb uses adjusted (weighted) rating formula (based on averages and number of user votes) in their rankings (see Weighted Average Ratings) so the title averageRating we looked at can't be taken at the face value after all.

Source Code

IMDb R notebook with all data prep and visualizations for this post found here: https://github.com/grigory93/r-notebooks/blob/master/IMDB-movie-ratings.Rmd

Information courtesy of IMDb (http://www.imdb.com).
Used with permission.

Finally, You Can Plot H2O Decision Trees in R

Creating and plotting decision trees (like one below) for the models created in H2O will be main objective of this post:

Figure 1. Decision Tree Visualization

Decision Trees with H₂O

With release 3.22.0.1 H₂O-3 (a.k.a. open source H₂O or simply H₂O) added to its family of tree-based algorithms (which already included DRF, GBM, and XGBoost) support for one more: Isolation Forest (random forest for unsupervised anomaly detection). There were no simple way to visualize H₂O trees except following clunky (albeit reliable) method of creating a MOJO object and running combination of Java and dot commands.

That changed in 3.22.0.1 too with introduction of unified Tree API to work with any of the tree-based algorithms above. Data scientists are now able to utilize powerful visualization tools in R (or Python) without resorting to producing intermediate artifacts like MOJO and running external utilities. Please read this article by Pavel Pscheidl who did superb job of explaining H₂O Tree API and S4 classes in R before coming back to take it a step further to visualize trees.

The Workflow: from Data to Decision Tree

Whether you are still here or came back after reading Pavel's excellent post let's set goal straight: create single decision tree model in H₂O and visualize its tree graph. With H₂O there is always a choice between using Python or R - the choice for R here will become clear when discussing its graphical and analytical capabilities later.

CART models operate on labeled data (classification and regression) and offer arguably unmatched model interpretability by means of analyzing a tree graph. In data science there is never single way to solve given problem so let's define end-to-end logical workflow from "raw" data to visualized decision tree:

Figure 2. Workflow of tasks in this post

One may argue that the choice of executing steps inside H₂O or R could be different but let's follow outlined plan for this post. Next diagram adds implementation details:

• R package data.table for data munging
• H₂O grid for hyper-parameter search
• H₂O GBM for modeling single decision tree algorithm
• H₂O Tree API for tree model representation
• R package data.tree for visualization  

Figure 3. Workflow of tasks in this post with implementation details

Discussion of this workflow continues for the rest of this post.

Titanic Dataset

The famous Titanic dataset contains information about the fate of passengers of the RMS Titanic that sank after colliding with an iceberg. It regularly serves as toy data for blog exercises like this.

H₂O public S3 bucket holds the Titanic dataset readly available and using package data.table makes it fast one-liner to load into R:

Data Engineering

Passenger features from the Titanic dataset are discussed at length online, e.g. see Predicting the Survival of Titanic Passengers and Predicting Titanic Survival using Five Algorithms. To summarize the following features were selected and engineered for decision tree model:

• survived indicates if passenger survived the wreck
• boat and body leak survival outcome and were dropped completely before modeling
• name and cabin are too noisy as they are and only used to derive new features
• title is parsed from name
• cabin_type is parsed from cabin
• family_size and family_type are derived from combination of count features sibsp (siblings+spouse) and parch (parents+children)
• ticket and home.dest are dropped to preserve simplicity of the model
• missing values in age and fare are imputed using target encoding (mean) over grouping by survived, sex, and embarked columns. 

Data load and data munging steps above are implemented in R using data.table:

Starting with H₂O

Creating models with H₂O requires running a server process (remote or local) and a client (package h2o in R available from CRAN) where the latter connects and sends commands to the former. The Tree API was introduced with release 3.22.0.1 (10/26/2018) but due to CRAN policies h2o package usually lags several versions behind (on the time of this writing CRAN hosted version 3.20.0.8). There are two ways to work around this:

1. Install and run package available from CRAN and use strict_version_check=FALSE inside h2o.connect() to communicate with newer version running on server
2. Or install the latest version of h2o available from H₂O repository either to connect to remote server or to both connect and run server locally.

Tree API is available only with 2d option because it requires access to new classes and functions in h2o package (remember, I asked you read Pavel's blog). Below code from the official H₂O download page shows how to download and install the latest version of the package: 

Building Decision Tree with H₂O

While H2O offers no dedicated single decision tree algorithm there two approaches using superseding models:

• Distributed Random Forest (DRF) function h2o.randomForest() with arguments
  ntrees = 1
  mtries = number of features (would be determined dynamically at runtime)
  sample_rate = 1
  min_rows = 1
• Gradient Boosting Machine (GBM) function h2o.gbm() with arguments
  ntrees = 1
  min_rows = 1
  sample_rate = 1
  col_sample_rate = 1

Choosing GBM option requires one less line of code (no need to calculate number of features to set mtries) so it was used for this post. Otherwise both ways result in the same decision tree with the steps below fully reproducible using h2o.randomForest() instead of h2o.gbm().

Decision Tree Depth

When building single decision tree models maximum tree depth stands as the most important parameter to pick. Shallow trees tend to underfit by failing to capture important relationships in data producing similar trees despite varying training data (error due to high bias). On the other hand trees grown too deep overfit by reacting to noise and slight changes in data (error due to high variance). Tuning H₂O model's parameter max_depth that limits decision tree depth aims at balancing the effects of bias and variance. In R using H₂O to split data and to tune the model, then visualizing results with ggplot to look for right value unfolds like this:   

1. split Titanic data into training and validation sets
2. define grid search object with parameter max_depth
3. launch grid search on GBM models and grid object to obtain AUC values (model performance)
4. plot grid model AUC'es vs. max_depth values to determine "inflection point" where AUC growth stops or saturates (see plot below)
5. register tree depth value at inflection point to use in the final model

Code below implements these steps:

and produces chart that points to inflection point for maximum tree depth at 5:

Figure 4. Visualization of AUC vs. maximum tree depth hyper-parameter trend
extracted from the H2O grid object after running grid search in H2O.
Marked inflection point indicates when increasing maximum tree depth
no longer improves model performance on validation set

 

Creating Decision Tree

As evident from the Figure 4 optimal decision tree depth is 5. The code below constructs single decision tree model in H₂O and then retrieves tree representation from a GBM model with Tree API function h2o.getModelTree(), which creates an instance of S4 class H2OTree and assigns to variable titanicH2oTree:

At this point all action moved back inside R with its unparalleled access to analytical and visualization tools. So before navigating and plotting a decision tree - final goal for this post - let's have brief intro to networks in R.

Overview of Network Analysis in R

R offers arguably the richest functionality when it comes to analyzing and visualizing network (graph, tree) objects. Before taking on the task of conquering it spend time visiting a couple of comprehensive articles describing vast landscape of tools and approaches available: Static and dynamic network visualization with R by Katya Ognyanova and Introduction to Network Analysis with R by Jesse Sadler.

To summarize there are two commonly used packages to manage and analyze networks in R: network (part of statnet family) and igraph (family in itself). Each package implements namesake classes to represent network structures so there is significant overlap between the two and  they mask each other's functions. Preferred approach is picking only one of two: it appears that igraph is more common for general-purpose applications while network is preferred for social network and statistical analysis (my subjective assessment). And while researching these packages do not forget about package intergraph that seamlessly transforms objects between network and igraph classes. (And this analysis stopped short of expanding into universe of R packages hosted on Bioconductor).

When it comes to visualizing networks choices quickly proliferate. Both network and igraph offer graphical functions that use R base plotting system but it doesn't stop here. Following packages specialize in advanced visualizations for at least one or both of the classes:

• ggraph
• ggnet2
• ggnetwork
• visNetwork
• DiagrammeR
• networkD3

Finally, there is packagedata.tree designed specifically to create and analyze trees in R. It fits the bill of representing and visualizing decision trees perfectly, so it became a tool of choice for this post. Still, visualizing H2O model trees could be fully reproduced with any of network and visualization packages mentioned above. 

Visualizing H₂O Trees

In the last step a decision tree for the model created by GBM moved from H₂O cluster memory to H2OTree object in R by means of Tree API. Still, specific to H₂O the H2OTree object now contains necessary details about decision tree, but not in the format understood by R packages such asdata.tree.

To fill this gap function createDataTree(H2OTree) created that traverses a tree and translates it from H2OTreeinto data.tree accumulating information about decision tree splits and predictions into node and edge attributes of a tree:

Finally everything lined up and ready for the final step of plotting decision tree:

• single decision tree model created in H2O
• its structure made available in R
• and translated to specialized data.tree for network analysis.

Styling and plotting data.tree objects is built around rich functionality of the DiagrammerR package. For anything that goes beyond simple plotting read documentation here but also remember that for plotting data.tree takes advantage of:

• hierarchical nature of tree structures
• GraphViz attributes to style graph, node and edge properties
• and dynamic callback functions (in this example GetEdgeLabel(node), GetNodeShape(node), GetFontName(node)) to customize tree's feel and look 

The following code will produce this moderately customized decision tree for our H₂O model: 

Figure 5. H2O Decision Tree for Titanic Model Visualized in R using data.tree package

     

References

• Anomaly Detection with Isolation Forests using H2O
• Changes in H2O Xia (3.22.0.1) - 10/26/2018
• Visualizing H2O GBM and Random Forest MOJO Model Trees
• Inspecting Decision Trees in H2O 
• Classification and Regression Trees 
• Predicting the Survival of Titanic Passengers 
• Predicting Titanic Survival using Five Algorithms 
• Distributed Random Forest (DRF)
• Gradient Boosting Machine (GBM)
• Inflection Point
• Static and dynamic network visualization with R by Katya Ognyanova
• Introduction to Network Analysis with R by Jesse Sadler 
• CRAN Graphical Models in R Task View
• Bioconductor GraphAndNetwork packages 
• Introduction to data.tree 
• DiagrammeR package on github
• Node, Edge, and Graph attributes for Graphviz tools
• Public GitHub gist with source code