how household surveys could be used to measure poverty and income levels.

In this chapter we’re going to focus on how to use the dplyr package, another core member of the tidyverse. We’ll illustrate the key ideas using a data on India taken from IPUMS. To download the IPUMS dataset, you need to take the following steps:

1. Go to the IPUMS International homepage: https://international.ipums.org/international/ and log in with your user-name and password.
2. Click on “Account” at the top, and then “Join a class” using the class code RA5PHL.
3. Click on “My Data” and then on “Class Extracts.”

You should now be able to view the data for the exercise. Do the following steps to get it into R:

1. Download the file Assignment2_Start.ipynb from Canvas and place it to your preferred working directory (project folder).
2. On the IPUMS page, double click on Download.DAT to download a dat.gz file. Put the .gz file in your project folder, and extract it to your working directory. Rename it IPUMS.dat.
3. Again on IPUMS, right-click on “DDI” (Data Documentation Initiative) to download an .xml file using save link as. Put it in your project folder and rename it IPUMS.xml.
4. Open the file Assignment2_Start.ipynb and get started with your assignment. The file provides the code that loads the data and key packages. You should type your answers to the assignment in this jupyter notebook file in the provided cells, and rename it FirstName_LastName_A1.ipynb.

Take careful note of the conflicts message that’s printed when you load the tidyverse. It tells you that dplyr overwrites some functions in base R. If you want to use the base version of these functions after loading dplyr, you’ll need to use their full names: stats::filter() and stats::lag().

india04

To explore the basic data manipulation verbs of dplyr, we’ll use india04. This data frame contains a representative sample of all Indian individuals between 15 and 75 years of age, that were employed with a positive registered wage in 2004. People with zero wages or missing wages have their income coded as missing. The data comes from IPUMS, and you can access documentation about the variables on the IPUMS International. To do so, enter browse data on the IPUMS webpage and find your variables. Click on a variable to obtain information about that variable. For information on finding variable documentation, check the recorded film on Canvas.

india04

## # A tibble: 167,079 x 9
##     year urban   age sex     geolev2 occisco             edattain incwage  perwt
##    <int> <chr> <int> <chr>     <dbl> <chr>               <chr>      <dbl>  <dbl>
##  1  2004 Rural    27 Male     3.56e8 Skilled agricultur~ <primary      NA  2467.
##  2  2004 Rural    25 Female   3.56e8 NIU (not in univer~ <primary      NA  2467.
##  3  2004 Rural    25 Male     3.56e8 Skilled agricultur~ <primary      NA  2467.
##  4  2004 Rural    22 Female   3.56e8 NIU (not in univer~ <primary      NA  2467.
##  5  2004 Rural    45 Male     3.56e8 Service workers an~ <primary     240  1850.
##  6  2004 Rural    42 Female   3.56e8 Skilled agricultur~ <primary      NA  1850.
##  7  2004 Rural    16 Male     3.56e8 NIU (not in univer~ primary       NA  1850.
##  8  2004 Rural    32 Male     3.56e8 NIU (not in univer~ <primary      NA  1850.
##  9  2004 Rural    30 Female   3.56e8 NIU (not in univer~ <primary      NA  1850.
## 10  2004 Rural    55 Male     3.56e8 Skilled agricultur~ <primary      NA 16187.
## # ... with 167,069 more rows

The dataframe will show slightly differently in your Jupyter Notebook, but the first 10 rows will be the almost same, up to some rounding for numbers.

You might also have noticed the row of three (or four) letter abbreviations under the column names above. These describe the type of each variable:

• int stands for integers.

• chr stands for character vectors, or strings.

• dbl stands for doubles, or real numbers.

There are four other common types of variables that aren’t used in this dataset but you’ll encounter later in the book:

• lgl stands for logical, vectors that contain only TRUE or FALSE.

• fctr stands for factors, which R uses to represent categorical variables with fixed possible values.

• date stands for dates.

• dttm stands for date-times (a date + a time).

To get the type of each variable in your Jupyter Notebook session, you can either run str(india04), or do it separately for each variable by running typeof('variable_name'), for example typeof('year'). Try it out.

dplyr basics

In this chapter you are going to learn the five key dplyr functions that allow you to solve the vast majority of your data manipulation challenges:

• Pick observations by their values (filter()).
• Reorder the rows (arrange()).
• Pick variables by their names (select()).
• Create new variables with functions of existing variables (mutate()).
• Collapse many values down to a single summary (summarise()).

These can all be used in conjunction with group_by() which changes the scope of each function from operating on the entire dataset to operating on it group-by-group. These six functions provide the verbs for a language of data manipulation.

All verbs work similarly:

1. The first argument is a data frame.

2. The subsequent arguments describe what to do with the data frame, using the variable names (without quotes).

3. The result is a new data frame.

Together these properties make it easy to chain together multiple simple steps to achieve a complex result. Let’s dive in and see how these verbs work.