Lesson Objectives

• Import (customized) data from flat, rectangular data files using readr
• Identify and fix common data import challenges using readr
• Apply different strategies for Exploratory Data Analysis (EDA)
  • Graphical Summaries
  • Numerical Summaries with and without tables
• Develop ideas for further analysis

Loading data

So far, we’ve been using packages and data() to load in new data to our working session. But what if I want to load in some other data that I have? Go ahead and download the following files and let’s work on how to load them into R.

avengers.csv (source)
crf33.dta (source)
steak-risk-survey.xlsx (source)

Creating tibbles

As you have learned, a tibble and a dataframe are almost the same thing, except that tibbles have much nicer output and are easier to work with. From now on know that anytime I say data frame or tibble, I really mean a type of data that has rows and columns that is usually saved as a tibble.

How can I create a tibble from scratch? There are two main ways, using tibble and tribble.

Tibble creates dataframes in column formats while tribble creates them in rows. Many new learners say tribble is more intuitive.

library(tidyverse)

cats_tibble <- tibble(coat = c("calico", "black", "tabby"),
                      weight = c(2.1, 5.0, 3.2),
                      likes_string = c(1, 0, 1))
cats_tibble

## # A tibble: 3 x 3
##   coat   weight likes_string
##   <chr>   <dbl>        <dbl>
## 1 calico    2.1            1
## 2 black     5              0
## 3 tabby     3.2            1

cats_tribble <- tribble(
  ~coat, ~weight, ~likes_string,
  "calico", 2.1, 1,
  "black", 5.0, 0,
  "tabby", 3.2, 1
)
cats_tribble

## # A tibble: 3 x 3
##   coat   weight likes_string
##   <chr>   <dbl>        <dbl>
## 1 calico    2.1            1
## 2 black     5              0
## 3 tabby     3.2            1

#these are equivalent
cats_tibble == cats_tribble

##      coat weight likes_string
## [1,] TRUE   TRUE         TRUE
## [2,] TRUE   TRUE         TRUE
## [3,] TRUE   TRUE         TRUE

However, in normal practice creating data from scratch is rare. More often than not, we will load in out data from a premade source.

Loading .csv and .tsv

• A lot of datasets are available as flat files, in comma-separated or tab-separated formats, with the column or variable names in the first row of data.

• The different file types are usually identifiable by their extension.

• The extension “.csv” stands for “comma-separated values” where each column is separated by a comma. Each row is separated as a new line.

• readr is a tidyverse package to import data from a variety of file formats into tibbles faster (10x) and more accurately than base R’s read.csv()

  • readr converts flat files into data frames (tibbles)
  • readr does NOT convert character vectors to factors automatically (like R version 3.0 read.csv())
  • readr functions usually give more informative error messages than base R functions like read.csv()

Use read_csv() to read a CSV (comma-separated values) file into R

avengers <- read_csv(file = here::here("static","data","avengers.csv"))

If the .CSV is online and you know the URL, you can use the URL for the file argument.

avengers <- 
      read_csv(file ="https://raw.githubusercontent.com/fivethirtyeight/data/master/avengers/avengers.csv")

glimpse(avengers)

## Rows: 173
## Columns: 21
## $ URL                           <chr> "http://marvel.wikia.com/Henry_Pym_(E...
## $ `Name/Alias`                  <chr> "Henry Jonathan \"Hank\" Pym", "Janet...
## $ Appearances                   <dbl> 1269, 1165, 3068, 2089, 2402, 612, 34...
## $ `Current?`                    <chr> "YES", "YES", "YES", "YES", "YES", "Y...
## $ Gender                        <chr> "MALE", "FEMALE", "MALE", "MALE", "MA...
## $ `Probationary Introl`         <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N...
## $ `Full/Reserve Avengers Intro` <chr> "Sep-63", "Sep-63", "Sep-63", "Sep-63...
## $ Year                          <dbl> 1963, 1963, 1963, 1963, 1963, 1963, 1...
## $ `Years since joining`         <dbl> 52, 52, 52, 52, 52, 52, 51, 50, 50, 5...
## $ Honorary                      <chr> "Full", "Full", "Full", "Full", "Full...
## $ Death1                        <chr> "YES", "YES", "YES", "YES", "YES", "N...
## $ Return1                       <chr> "NO", "YES", "YES", "YES", "YES", NA,...
## $ Death2                        <chr> NA, NA, NA, NA, "YES", NA, NA, "YES",...
## $ Return2                       <chr> NA, NA, NA, NA, "NO", NA, NA, "YES", ...
## $ Death3                        <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N...
## $ Return3                       <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N...
## $ Death4                        <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N...
## $ Return4                       <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N...
## $ Death5                        <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N...
## $ Return5                       <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N...
## $ Notes                         <chr> "Merged with Ultron in Rage of Ultron...

readr tells you how it converted the columns - also known as parsing the data. you can uncover the exact ways it was read in with spec

spec(avengers)

## cols(
##   URL = col_character(),
##   `Name/Alias` = col_character(),
##   Appearances = col_double(),
##   `Current?` = col_character(),
##   Gender = col_character(),
##   `Probationary Introl` = col_character(),
##   `Full/Reserve Avengers Intro` = col_character(),
##   Year = col_double(),
##   `Years since joining` = col_double(),
##   Honorary = col_character(),
##   Death1 = col_character(),
##   Return1 = col_character(),
##   Death2 = col_character(),
##   Return2 = col_character(),
##   Death3 = col_character(),
##   Return3 = col_character(),
##   Death4 = col_character(),
##   Return4 = col_character(),
##   Death5 = col_character(),
##   Return5 = col_character(),
##   Notes = col_character()
## )

Readr automatically previews about 1000 rows of each column to guess the column type. If the parsing is incorrect, you can use the optional coltypes = argument within read_csv() to correct the column type.

Loading other data types

Haven for STATA

Sometimes you’ll come across data from SPSS, STATA, or SAS and need to import it into r. The package Haven comes installed with the tidyverse and enables clean import of these sorts of proprietary statistical file structures. Although you have it installed, you must load the package when you want to use it.

library(haven)
crf33 <- read_dta(file = here::here("static","data","crf33.dta"))

#While glimpse doesn't look any different
glimpse(crf33)

## Rows: 45
## Columns: 4
## $ a     <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2...
## $ b     <dbl> 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 2...
## $ y     <dbl> 24, 33, 37, 29, 42, 44, 36, 25, 27, 43, 38, 29, 28, 47, 48, 3...
## $ order <dbl> 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1...

#but str() will uncover these special attribute labels
str(crf33)

## tibble [45 x 4] (S3: tbl_df/tbl/data.frame)
##  $ a    : num [1:45] 1 1 1 1 1 1 1 1 1 1 ...
##   ..- attr(*, "format.stata")= chr "%8.0g"
##  $ b    : num [1:45] 1 1 1 1 1 2 2 2 2 2 ...
##   ..- attr(*, "format.stata")= chr "%8.0g"
##  $ y    : num [1:45] 24 33 37 29 42 44 36 25 27 43 ...
##   ..- attr(*, "format.stata")= chr "%8.0g"
##  $ order: num [1:45] 1 2 3 4 5 1 2 3 4 5 ...
##   ..- attr(*, "format.stata")= chr "%8.0g"

readxl for loading excel data

Often Not a Good Idea to Import Directly from Excel

• RStudio is getting better at importing data from Excel as are other packages
• You may want to import data directly from Excel? Not recommended.
• Excel is designed for human data input and data analysis and not efficient data management

• Potential for errors or excess time spent adjusting the data in Excel
• People tend to color code information in Excel or be inconsistent in their formatting

• Instead, export the data from the Excel worksheet as a .CSV. Then read the .CSV file into R.

• Edit the data so the information is encoded by a new variable.

library(readxl)
steak <- read_excel(path = here::here("static","data","steak-risk-survey.xlsx"))
glimpse(steak)

## Rows: 550
## Columns: 15
## $ RespondentID           <dbl> 3237565956, 3234982343, 3234973379, 32349723...
## $ hypothetical_scenario  <chr> "Lottery B", "Lottery A", "Lottery A", "Lott...
## $ smoke_cigarettes       <chr> NA, "No", "No", "Yes", "No", "No", "No", "No...
## $ drink_alcohol          <chr> NA, "Yes", "Yes", "Yes", "Yes", "No", "Yes",...
## $ gamble                 <chr> NA, "No", "Yes", "Yes", "No", "No", "No", "Y...
## $ skydiving              <chr> NA, "No", "No", "No", "No", "No", "No", "Yes...
## $ drive_above_speedlimit <chr> NA, "No", "Yes", "Yes", "Yes", "Yes", "Yes",...
## $ cheated_partner        <chr> NA, "No", "Yes", "Yes", "Yes", "No", "Yes", ...
## $ eat_steak              <chr> NA, "Yes", "Yes", "Yes", "Yes", "Yes", "No",...
## $ steak_preference       <chr> NA, "Medium rare", "Rare", "Medium", "Medium...
## $ gender                 <chr> NA, "Male", "Male", "Male", "Male", "Male", ...
## $ age                    <chr> NA, "> 60", "> 60", "> 60", "> 60", "18-29",...
## $ income                 <chr> NA, "$50,000 - $99,999", "$150,000+", "$50,0...
## $ education              <chr> NA, "Some college or Associate degree", "Gra...
## $ census_region          <chr> NA, "East North Central", "South Atlantic", ...

Again, be very careful when importing excel data. So often, there’s a random value in cell DM1334 that will throw off how it is read in or two rows for column names.

HW & Lab Preview

For your homework this week, you’ll take a deep dive into what is available for a dataset for your final project. I’m having you browse multiple different repositories that I think contain fun and useful data so you may become inspired for you final project. You will be sharing these datasets with your classmates in a ‘data speed-dating’ activity in the lab on Thursday, so do find a few you like.

EDA

Introduction

• Once you have loaded and checked your data for completeness and consistency you want to begin to look at it.
• You may have some initial questions or hypotheses about your question of interest
• EDA is a process for exploring your data to assess initial hypotheses and generate or uncover new ones
• You have to be careful about “data snooping” from a statistical perspective
• It helps to follow a general strategy for EDA

General Strategies

• Plot the distribution of every variable.

• Look for symmetry, skewness, modality, etc..

• Plot the bi-variate distribution of every pair of variables (to find which variables are associated).

• Again, look for patterns and relationships, skewness, curvature, modality, gaps, discontinuities, , etc..

• Color-code by variables to see if relationships appear more clearly.

• Calculate lots of numerical summary statistics.

• Look at “missingness.”

• Look at extreme values for potential “outliers” and patterns

• EDA is about curiosity.

• Ask many questions, use many plots, investigate many aspects of your data.

• This will let you hone in on the few interesting questions you want to pursue deeper.

• Keep track of what you are doing with your .Rmd file text chunks and code chunks so you can protext yourself from becoming a victim of data snooping - only cherry picking the “good results”

Automated EDA

skimr

skimr provides a frictionless approach to summary statistics which conforms to the principle of least surprise, displaying summary statistics the user can skim quickly to understand their data. It handles different data types and returns a skim_df object which can be included in a pipeline or displayed nicely for the human reader.

# install.packages("skimr")
library(skimr)
skimr::skim(avengers)

Table 1: Data summary

Variable type: character

Variable type: numeric

diamonds %>% 
  group_by(cut) %>% 
  skimr::skim()

Table 1: Data summary

Variable type: factor

Variable type: numeric

visdat:

helps you visualise a dataframe and “get a look at the data” by displaying the variable classes in a dataframe as a plot with vis_dat, and getting a brief look into missing data patterns using vis_miss.

# install.packages("visdat")
library(visdat)

#overall viz
vis_dat(steak)

vis_dat(avengers)

#missing viz
vis_miss(avengers, 
         cluster = TRUE)

#comparison viz
vis_compare(cats_tibble, cats_tribble)

Why EDA

Why do we want to visualize our data?

• All the graphs are based on data with the same summary statistics!

We can see this more explicitly with the “DatasauRus Dozen” Package. We have nearly identical summary statistics, but the actual data looks vastly different from eachother.

library(datasauRus)
datasaurus_dozen %>%
  group_by(dataset) %>%
  summarize(
    mean_x    = mean(x),
    mean_y    = mean(y),
    std_dev_x = sd(x),
    std_dev_y = sd(y),
    corr_x_y  = cor(x, y)
  )

## # A tibble: 13 x 6
##    dataset    mean_x mean_y std_dev_x std_dev_y corr_x_y
##  * <chr>       <dbl>  <dbl>     <dbl>     <dbl>    <dbl>
##  1 away         54.3   47.8      16.8      26.9  -0.0641
##  2 bullseye     54.3   47.8      16.8      26.9  -0.0686
##  3 circle       54.3   47.8      16.8      26.9  -0.0683
##  4 dino         54.3   47.8      16.8      26.9  -0.0645
##  5 dots         54.3   47.8      16.8      26.9  -0.0603
##  6 h_lines      54.3   47.8      16.8      26.9  -0.0617
##  7 high_lines   54.3   47.8      16.8      26.9  -0.0685
##  8 slant_down   54.3   47.8      16.8      26.9  -0.0690
##  9 slant_up     54.3   47.8      16.8      26.9  -0.0686
## 10 star         54.3   47.8      16.8      26.9  -0.0630
## 11 v_lines      54.3   47.8      16.8      26.9  -0.0694
## 12 wide_lines   54.3   47.8      16.8      26.9  -0.0666
## 13 x_shape      54.3   47.8      16.8      26.9  -0.0656

ggplot(datasaurus_dozen, 
       aes(x=x, y=y, color=dataset))+
  geom_point(size = .5)+
  theme_void()+
  theme(legend.position = "none")+
  facet_wrap(~dataset, ncol=3)

¹

Data Export

write_csv(), write_csv2(), and write_tsv()

• You can write comma-separated and tab-separated files using write_csv(), write_csv2(), and write_tsv().

steak %>% 
  filter(hypothetical_scenario == "Lottery B") %>% 
  write_csv("../output/nyc_region_data.csv")

• The defaults are usually fine.

Reading/Writing R Objects with readRDS() and saveRDS().

• You can save and reload arbitrary R objects (data frames, matrices, lists, vectors) using readRDS() and saveRDS(). - .Rds files are compressed data files so enable much faster loading and more compact storage. - These are what usually go into the /data folder as opposed to the /data_raw folder

steak %>% 
  filter(hypothetical_scenario == "Lottery B") %>% 
  saveRDS("../output/nyc_region_data.rds")

References

• Chapter 11 of RDS
• Chapter 7 of RDS
• Data Import Cheatsheet
• Readr Overview
• Haven Documentation
• readxl Documentation