Course: Exploratory Data Analysis
Except where otherwise indicated, this work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License . Your use of this material constitutes acceptance of that license and the conditions of use of materials on this site.
The materials below are used by students in this course. To sign up to take the course online, please visit the Johns Hopkins Data Science Specialization.
» Course Project 1
This assignment uses data from the UC Irvine Machine Learning Repository, a popular repository for machine learning datasets. In particular, we will be using the “Individual household electric power consumption Data Set” which I have made available on the course web site:
Dataset: Electric power consumption [20Mb]
Description: Measurements of electric power consumption in one household with a one-minute sampling rate over a period of almost 4 years. Different electrical quantities and some sub-metering values are available.
The following descriptions of the 9 variables in the dataset are taken from the UCI web site:
- Date: Date in format dd/mm/yyyy
- Time: time in format hh:mm:ss
- Global_active_power: household global minute-averaged active power (in kilowatt)
- Global_reactive_power: household global minute-averaged reactive power (in kilowatt)
- Voltage: minute-averaged voltage (in volt)
- Global_intensity: household global minute-averaged current intensity (in ampere)
- Sub_metering_1: energy sub-metering No. 1 (in watt-hour of active energy). It corresponds to the kitchen, containing mainly a dishwasher, an oven and a microwave (hot plates are not electric but gas powered).
- Sub_metering_2: energy sub-metering No. 2 (in watt-hour of active energy). It corresponds to the laundry room, containing a washing-machine, a tumble-drier, a refrigerator and a light.
- Sub_metering_3: energy sub-metering No. 3 (in watt-hour of active energy). It corresponds to an electric water-heater and an air-conditioner.
Loading the data
When loading the dataset into R, please consider the following:
The dataset has 2,075,259 rows and 9 columns. First calculate a rough estimate of how much memory the dataset will require in memory before reading into R. Make sure your computer has enough memory (most modern computers should be fine).
We will only be using data from the dates 2007-02-01 and 2007-02-02. One alternative is to read the data from just those dates rather than reading in the entire dataset and subsetting to those dates.
You may find it useful to convert the Date and Time variables to Date/Time classes in R using the strptime() and as.Date() functions.
Note that in this dataset missing values are coded as ?.
Our overall goal here is simply to examine how household energy usage varies over a 2-day period in February, 2007. Your task is to reconstruct the following plots below, all of which were constructed using the base plotting system.
First you will need to fork and clone the following GitHub repository: https://github.com/rdpeng/ExData_Plotting1
For each plot you should
Construct the plot and save it to a PNG file with a width of 480 pixels and a height of 480 pixels.
Name each of the plot files as plot1.png, plot2.png, etc.
Create a separate R code file (plot1.R, plot2.R, etc.) that constructs the corresponding plot, i.e. code in plot1.R constructs the plot1.png plot. Your code file should include code for reading the data so that the plot can be fully reproduced. You should also include the code that creates the PNG file.
Add the PNG file and R code file to your git repository
When you are finished with the assignment, push your git repository to GitHub so that the GitHub version of your repository is up to date. There should be four PNG files and four R code files.
The four plots that you will need to construct are shown below.
» Course Project 2
Fine particulate matter (PM2.5) is an ambient air pollutant for which there is strong evidence that it is harmful to human health. In the United States, the Environmental Protection Agency (EPA) is tasked with setting national ambient air quality standards for fine PM and for tracking the emissions of this pollutant into the atmosphere. Approximatly every 3 years, the EPA releases its database on emissions of PM2.5. This database is known as the National Emissions Inventory (NEI). You can read more information about the NEI at theÂ EPA National Emissions Inventory web site.
For each year and for each type of PM source, the NEI records how many tons of PM2.5 were emitted from that source over the course of the entire year. The data that you will use for this assignment are for 1999, 2002, 2005, and 2008.
The data for this assignment are available from the course web site as a single zip file:
The zip file contains two files:
PM2.5 Emissions Data (summarySCC_PM25.rds): This file contains a data frame with all of the PM2.5 emissions data for 1999, 2002, 2005, and 2008. For each year, the table contains number ofÂ tonsÂ of PM2.5 emitted from a specific type of source for the entire year. Here are the first few rows.## fips SCC Pollutant Emissions type year ## 4 09001 10100401 PM25-PRI 15.714 POINT 1999 ## 8 09001 10100404 PM25-PRI 234.178 POINT 1999 ## 12 09001 10100501 PM25-PRI 0.128 POINT 1999 ## 16 09001 10200401 PM25-PRI 2.036 POINT 1999 ## 20 09001 10200504 PM25-PRI 0.388 POINT 1999 ## 24 09001 10200602 PM25-PRI 1.490 POINT 1999
fips: A five-digit number (represented as a string) indicating the U.S. county
SCC: The name of the source as indicated by a digit string (see source code classification table)
Pollutant: A string indicating the pollutant
Emissions: Amount of PM2.5 emitted, in tons
type: The type of source (point, non-point, on-road, or non-road)
year: The year of emissions recorded
Source Classification Code Table (Source_Classification_Code.rds): This table provides a mapping from the SCC digit strings int he Emissions table to the actual name of the PM2.5 source. The sources are categorized in a few different ways from more general to more specific and you may choose to explore whatever categories you think are most useful. For example, source “10100101” is known as “Ext Comb /Electric Gen /Anthracite Coal /Pulverized Coal”.
You can read each of the two files using theÂ readRDS()Â function in R. For example, reading in each file can be done with the following code:## This first line will likely take a few seconds. Be patient! NEI <- readRDS("summarySCC_PM25.rds") SCC <- readRDS("Source_Classification_Code.rds")
as long as each of those files is in your current working directory (check by callingÂ dir()Â and see if those files are in the listing).
The overall goal of this assignment is to explore the National Emissions Inventory database and see what it say about fine particulate matter pollution in the United states over the 10-year period 1999â€“2008. You may use any R package you want to support your analysis.
You must address the following questions and tasks in your exploratory analysis. For each question/task you will need to make a single plot. Unless specified, you can use any plotting system in R to make your plot.
Have total emissions from PM2.5 decreased in the United States from 1999 to 2008? Using theÂ baseÂ plotting system, make a plot showing theÂ totalÂ PM2.5 emission from all sources for each of the years 1999, 2002, 2005, and 2008.
Have total emissions from PM2.5 decreased in theÂ Baltimore City, Maryland (fips == "24510") from 1999 to 2008? Use theÂ baseÂ plotting system to make a plot answering this question.
Of the four types of sources indicated by theÂ typeÂ (point, nonpoint, onroad, nonroad) variable, which of these four sources have seen decreases in emissions from 1999â€“2008 forÂ Baltimore City? Which have seen increases in emissions from 1999â€“2008? Use theÂ ggplot2Â plotting system to make a plot answer this question.
Across the United States, how have emissions from coal combustion-related sources changed from 1999â€“2008?
How have emissions from motor vehicle sources changed from 1999â€“2008 inÂ Baltimore City?
Compare emissions from motor vehicle sources in Baltimore City with emissions from motor vehicle sources inÂ Los Angeles County, California (fips == "06037"). Which city has seen greater changes over time in motor vehicle emissions?
Making and Submitting Plots
For each plot you should
Construct the plot and save it to aÂ PNG file.
Create a separate R code file (plot1.R,Â plot2.R, etc.) that constructs the corresponding plot, i.e. code in plot1.R constructs the plot1.png plot. Your code file should include code for reading the data so that the plot can be fully reproduced. You should also include the code that creates the PNG file. Only include the code for a single plot (i.e.Â plot1.RÂ should only include code for producingÂ plot1.png)
Upload the PNG file on the Assignment submission page
Copy and paste the R code from the corresponding R file into the text box at the appropriate point in the peer assessment.