Data Science II

Session 3: sf Essentials, Choropleths, and Interactive Maps

Bogdan G. Popescu

Tec de Monterrey

What You’ll Learn Today

  1. What the sf package is, and the three classes inside it
  2. Joining a table to boundaries — and the join that silently fails
  3. Building a choropleth from the result
  4. Choosing colour — and why viridis rather than a rainbow
  5. Classification: the same data, three schemes, three different stories
  6. The count-versus-rate trap, and the small-numbers trap behind it
  7. Interactive maps with leaflet, embedded in your Quarto report
  8. Choosing a file format, and writing your layer back out

Before We Start

Code reads ../data/..., exactly as in Sessions 1 and 2:

#Step1: Libraries for today
library(sf)          # spatial objects
library(dplyr)       # data wrangling
library(ggplot2)     # static maps
library(classInt)    # classification schemes
library(viridis)     # colour palettes
library(leaflet)     # interactive maps
library(patchwork)   # side-by-side plots

#Step2: Treat lon/lat as planar coordinates -- S2, sf's spherical engine,
#       rejects the small topology errors that real boundary files are full of
sf_use_s2(FALSE)

A Word on sp

The first R package for vector data was sp (2005), with its companions rgdal and rgeos.

They dominated spatial analysis in R for fifteen years. rgdal and rgeos were retired from CRAN in 2023. sp itself survives, but only in maintenance.

Warning

Do not start new work with them — but recognise them. Most Stack Overflow answers you find were written for sp, and anything calling readOGR() or spTransform() will not run today. This is also the single thing AI gets wrong most often here.

The sf Package

sf is the standard package for working with vector data in R.

It relies on three external components: GDAL (reading formats), GEOS (geometry operations), and PROJ (projections).

It contains three classes, and they nest:

  • sfg — a single geometry
  • sfc — a geometry column: many sfg plus a CRS
  • sf — a layer: an sfc inside a dataframe of attributes

When an error says “not an sfc”, this is the sentence it is complaining about.

Today’s Data

../data/mexico_homicides.gpkg — download first: mexico_homicides.gpkg

A GeoPackage with two layers:

Layer Rows Contents
states 32 boundaries, population, homicides, rate
municipios 2,469 the same, at municipio level
  • Homicides — SESNSP homicidio doloso, annual mean over 2021–2023
  • Population — INEGI Censo 2020, via the mxmaps package
  • Boundaries — INEGI geometry with CVEGEO codes

A three-year mean, not a single year, because one bad year in a small municipio is noise, not signal.

Reading It In

#Step1: A GeoPackage can hold several layers -- ask what is inside
gpkg <- "../data/mexico_homicides.gpkg"
st_layers(gpkg)$name
[1] "municipios" "states"    
#Step2: With several layers you MUST name the one you want, or st_read takes
#       the first and only warns you about it
states <- st_read(gpkg, layer = "states", quiet = TRUE)

#Step3: Look at it
head(st_drop_geometry(states), 3)
  cve_ent          state_name     pop  homicides      rate
1      01      Aguascalientes 1425607   76.66667  5.377826
2      02     Baja California 3769020 2389.66667 63.402865
3      03 Baja California Sur  798447   37.00000  4.633996

Part A: From a Table to a Choropleth

Choropleth Maps

A choropleth is a thematic map where regions are coloured according to a statistical variable.

The colour carries the number. Everything else on the map is context.

Three Decisions

Three decisions, and every one of them changes the story:

  1. What variable — a count, or a rate?
  1. What colours — which palette, in which direction?
  1. What breaks — how do continuous values become discrete classes?

Most bad maps are bad because of decision 1 or decision 3.

The Join Key

Homicides come from SESNSP. Population comes from the census. Boundaries come from INEGI. Three sources, three files.

They are joined on INEGI’s geostatistical code. 31050 is Mérida, Yucatán: 31 for the state, 050 for the municipio.

In our file the state layer calls it cve_ent and the municipio layer calls it cvegeo — the same code, one with the municipio digits attached.

Warning

Never join Mexican data on place names. GADM calls the capital Distrito Federal; INEGI calls it Ciudad de México. There are two municipios called Guadalupe, and several spellings of Tláhuac. Codes do not have this problem.

What a Left Join Does

left_join(x, y, by = "key") keeps every row of x, and pulls in the matching columns from y.

A row of x with no match in y is still kept — the new columns just come back NA. Nothing is dropped, and nothing warns you.

The Two Halves

Our GeoPackage arrives pre-joined. Take it apart to see what was joined to what:

#Step1: The boundaries -- the key, a name, and the geometry
boundaries <- states %>%
  select(cve_ent, state_name)
head(boundaries, 3)
Simple feature collection with 3 features and 2 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -117.1239 ymin: 21.62226 xmax: -101.836 ymax: 32.71865
Geodetic CRS:  WGS 84
  cve_ent          state_name                           geom
1      01      Aguascalientes MULTIPOLYGON (((-102.0681 2...
2      02     Baja California MULTIPOLYGON (((-115.1639 2...
3      03 Baja California Sur MULTIPOLYGON (((-110.8699 2...
#Step2: The numbers -- an ordinary dataframe, no geometry at all
rates_table <- states %>%
  st_drop_geometry() %>%
  select(cve_ent, pop, homicides, rate)
head(rates_table, 3)
  cve_ent     pop  homicides      rate
1      01 1425607   76.66667  5.377826
2      02 3769020 2389.66667 63.402865
3      03  798447   37.00000  4.633996

Doing the Join

#Step1: The sf object goes on the LEFT. That is what keeps the geometry.
joined <- left_join(boundaries, rates_table, by = "cve_ent")
head(joined, 3)
Simple feature collection with 3 features and 5 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -117.1239 ymin: 21.62226 xmax: -101.836 ymax: 32.71865
Geodetic CRS:  WGS 84
  cve_ent          state_name     pop  homicides      rate
1      01      Aguascalientes 1425607   76.66667  5.377826
2      02     Baja California 3769020 2389.66667 63.402865
3      03 Baja California Sur  798447   37.00000  4.633996
                            geom
1 MULTIPOLYGON (((-102.0681 2...
2 MULTIPOLYGON (((-115.1639 2...
3 MULTIPOLYGON (((-110.8699 2...

Two fields became five, and geom came along without being asked.

Which Side Goes Left

#Step1: sf on the left -- the result is still an sf object
class(joined)
[1] "sf"         "data.frame"
#Step2: Now put the plain table on the left instead
flipped <- left_join(rates_table, boundaries, by = "cve_ent")

#Step3: The geometry column is still there...
names(flipped)
[1] "cve_ent"    "pop"        "homicides"  "rate"       "state_name"
[6] "geom"      
#Step4: ...but the object is not an sf object any more
class(flipped)
[1] "data.frame"

Every column survives, geom included. The class does not — and geom_sf() refuses anything that is not an sf object.

Why "01" Is Not 1

#Step1: The boundary file keeps the state code as text
head(boundaries$cve_ent, 4)
[1] "01" "02" "03" "04"
#Step2: read.csv sees a column of digits and reads it as numbers.
#       A leading zero has nowhere to live inside a number.
as.numeric(head(rates_table$cve_ent, 4))
[1] 1 2 3 4
#Step3: Turning it back into text does not bring the zero back
rates_table_bad <- rates_table %>%
  mutate(cve_ent = as.character(as.numeric(cve_ent)))
head(rates_table_bad$cve_ent, 4)
[1] "1" "2" "3" "4"
#Step4: And to R these are simply two different keys
"01" == "1"
[1] FALSE

The Trap: Leading Zeros

#Step1: The join runs. No error, no warning.
joined_bad <- left_join(boundaries, rates_table_bad, by = "cve_ent")

#Step2: Nine of the thirty-two matched nothing
sum(is.na(joined_bad$rate))
[1] 9
#Step3: And the map draws anyway
ggplot(joined_bad) +
  geom_sf(aes(fill = rate))

The grey states are 01 through 09. From 10 on, the keys match.

Not Losing Nine States

Read code columns as text, and the zero survives:

read.csv("estados.csv", colClasses = c(cve_ent = "character"))

Or use readr::read_csv(), which leaves them alone.

Important

Then check sum(is.na()) after every join. One line, and it is the only thing between you and a map with nine silent holes in it.

Your First Choropleth

ggplot(states) +
  geom_sf(aes(fill = rate))

It works. The north-west and the Pacific centre are visibly worse off than the south-east.

Now we make it readable.

Part B: Colour

Three Kinds of Palette

Type Use when Example
Sequential low → high homicide rate
Diverging departure from a midpoint change since 2015
Qualitative unordered categories dominant party, land use

Using a qualitative palette for ordered data — or a rainbow for anything — invents structure that is not in the data.

The viridis Family

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_c(option = "magma")

The viridis Family

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_c(option = "inferno")

The viridis Family

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_c(option = "plasma")

The viridis Family

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_c(option = "viridis")

The viridis Family

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_c(option = "cividis")

The viridis Family

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_c(option = "turbo")

Why viridis?

  • Perceptually uniform — equal steps in data look like equal steps in colour
  • Colourblind-safe — about 8% of men have some form of colour vision deficiency
  • Prints in greyscale — the order survives a black-and-white printer

Warning

turbo is the exception — it is a rainbow, and it fails all three tests. Look back at the turbo map: the eye jumps to the green states, which are in the middle of the distribution. Do not use it for a policy map.

Direction Matters

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_c(direction = -1)

direction = -1 makes dark mean high. For a variable like homicide, dark-is-bad matches what readers expect — and it survives photocopying better.

_c, _b, or _d?

Function Gives you Use for
scale_fill_viridis_c() continuous gradient smooth variables
scale_fill_viridis_b() binned steps choropleths
scale_fill_viridis_d() discrete categories factors

For choropleths you almost always want binned — the eye cannot read a continuous ramp accurately. Which raises the real question: where do the bins go?

Part C: Classification

Continuous to Discrete

A binned choropleth reduces a continuous variable to a handful of colours.

That reduction is a choice, and it is the single biggest lever on what the reader sees.

Never pick breaks before you have looked at the distribution.

The Distribution

ggplot(states, aes(rate)) +
  geom_histogram(bins = 20) +
  labs(x = "Homicides per 100,000") +
  theme_bw()

Right-skewed, but not degenerate. Mean 23.7, standard deviation 21.7. Eleven states are below 10; four are above 50.

classInt: Three Ways to Cut

#Step1: Same data, same number of classes, very different cut points
round(classIntervals(states$rate, n = 5, style = "equal")$brks, 1)
[1]  1.7 19.6 37.5 55.5 73.4 91.3
round(classIntervals(states$rate, n = 5, style = "quantile")$brks, 1)
[1]  1.7  7.9 12.3 19.2 42.7 91.3
round(classIntervals(states$rate, n = 5, style = "jenks")$brks, 1)
[1]  1.7  9.6 21.1 49.0 63.4 91.3

Now let us see what each one does to the map.

What the Three Schemes Do

The rule The picture
Equal interval cut the range into equal widths a ruler chopped into five equal pieces
Quantile cut so each class holds the same number of states 32 guests seated at 5 tables, about 6 each
Jenks cut where the data already has gaps five natural clumps, split where they thin out

Equal interval is the only one whose legend a reader can predict. Quantile is the only one that guarantees every colour appears. Jenks is the only one that looks at the shape of the data.

None of them is neutral. Choosing a scheme is a claim about what counts as high.

Equal Interval

#Step1: Cut the RANGE into five equal-width slices
breaks <- classIntervals(states$rate, n = 5, style = "equal")$brks

#Step2: Use those breaks as the colour steps
ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_b(breaks = breaks, direction = -1)

Equal Interval: What It Does

breaks <- classIntervals(states$rate, n = 5, style = "equal")$brks
table(cut(states$rate, breaks = breaks, include.lowest = TRUE))

[1.65,19.6] (19.6,37.5] (37.5,55.5] (55.5,73.4] (73.4,91.3] 
         21           3           4           3           1 

Twenty-one of thirty-two states in the first class. Equal interval divides the range, and the range is stretched by Colima at 91.

It is honest about magnitude — the classes really are equally wide — but it wastes four of your five colours on eight states.

Use it when data are roughly uniform, or when round-number breaks matter more than resolution.

Quantile

#Step1: Cut so each class holds the same NUMBER of states
breaks <- classIntervals(states$rate, n = 5, style = "quantile")$brks
breaks <- classIntervals(states$rate, n = 5, style = "quantile")$brks
table(cut(states$rate, breaks = breaks, include.lowest = TRUE))

[1.65,7.87] (7.87,12.3] (12.3,19.2] (19.2,42.7] (42.7,91.3] 
          7           6           6           6           7 

Quantile: The Trade-Off

Six or seven states per class, so the whole map varies and you can read the middle of the distribution.

But look at the colour bar: the swatches are wildly unequal in width. The top class runs from 43 to 91; the bottom from 1.7 to 8.

Warning

Quantile maps make rank visible and magnitude invisible. A state at 44 and a state at 91 get the same colour, though one is twice as violent.

Jenks Natural Breaks

#Step1: Let the algorithm find the gaps in the distribution
breaks <- classIntervals(states$rate, n = 5, style = "jenks")$brks

Jenks minimises variance within classes and maximises it between them — it puts the cuts where the data already has gaps. Here: 11, 11, 6, 3, 1.

Which Should You Use?

Scheme Good when Fails when
Equal interval data are uniform; breaks must be round numbers outliers stretch the range
Quantile you care about rank; want a map that varies magnitude matters
Std. deviation data are symmetric; diverging maps data are skewed
Jenks you want the data’s own structure maps must be comparable

Important

If you make several maps that must be compared — 2015 next to 2023, say — fix the breaks by hand and reuse them. Jenks chooses different cuts for every map, and the maps stop being comparable.

Part D: Counts, Rates, and Small Numbers

The Most Common Mistake

#Step1: The five states with the most homicides
states %>%
  st_drop_geometry() %>%
  arrange(desc(homicides)) %>%
  transmute(state_name, homicides = round(homicides), rate = round(rate, 1)) %>%
  head(5)
       state_name homicides rate
1      Guanajuato      2679 43.4
2 Baja California      2390 63.4
3          México      2286 13.5
4       Michoacán      1880 39.6
5       Chihuahua      1835 49.0

The Most Common Mistake

#Step2: Now the five states with the highest rate
states %>%
  st_drop_geometry() %>%
  arrange(desc(rate)) %>%
  transmute(state_name, homicides = round(homicides), rate = round(rate, 1)) %>%
  head(5)
       state_name homicides rate
1          Colima       668 91.3
2 Baja California      2390 63.4
3       Zacatecas       941 58.0
4         Morelos      1126 57.1
5       Chihuahua      1835 49.0

Almost none of the same states.

Two Different Countries

By count, the State of México is third — 2,286 homicides a year.

By rate, it is 13.5 per 100,000, well below the national average. It has 17 million people.

Colima has 668 homicides — barely a quarter of México’s — and the worst rate in the country at 91.3.

Important

A count map of almost any human event is, mostly, a map of where the people are. Almost always, you want a rate.

Count and Rate Compared

#Step1: Jenks breaks for each variable separately
breaks_count <- classIntervals(states$homicides, n = 5, style = "jenks")$brks
breaks_rate <- classIntervals(states$rate,      n = 5, style = "jenks")$brks

#Step2: One map each, side by side with patchwork
map_count <- ggplot(states) + geom_sf(aes(fill = homicides)) +
  scale_fill_viridis_b(breaks = unique(breaks_count), direction = -1) + ggtitle("Count")
map_rate <- ggplot(states) + geom_sf(aes(fill = rate)) +
  scale_fill_viridis_b(breaks = unique(breaks_rate), direction = -1) + ggtitle("Per 100,000")

map_count | map_rate

Count and Rate Compared

But Rates Have Their Own Trap

#Step1: Same file, the other layer -- 2,469 units instead of 32
municipios <- st_read(gpkg, layer = "municipios", quiet = TRUE)

#Step2: The five worst rates in the country
municipios %>%
  st_drop_geometry() %>%
  arrange(desc(rate)) %>%
  transmute(municipio_name, state_name, pop,
            homicides = round(homicides, 1), rate = round(rate)) %>%
  head(5)
           municipio_name state_name  pop homicides rate
1             Doctor Coss Nuevo León 1360      10.0  735
2                 Oquitoa     Sonora  496       1.7  336
3     Sitio de Xitlapehua     Oaxaca  713       1.7  234
4 Santa María Mixtequilla     Oaxaca 4690       9.3  199
5 General Enrique Estrada  Zacatecas 6644      13.0  196

The Small-Numbers Problem

Doctor Coss, Nuevo León has the worst homicide rate in Mexico: 735 per 100,000 — eight times Colima.

It has 1,360 inhabitants and averaged 10 homicides a year. One more incident moves the rate by 74 points.

Warning

A rate with a tiny denominator is unstable, not extreme. Ranking municipios by raw rate puts the smallest ones on top almost every time — and a map of it is a map of small populations.

What To Do About It

  • Pool years — we already used a three-year mean rather than one year
  • Suppress small denominators — grey out units below a population threshold and say so in the caption
  • Aggregate — report at state level where municipio counts are thin
  • Smooth — empirical Bayes shrinks unreliable rates toward the regional mean

The map on the next slide does the simplest of these: it shows municipios, but greys out those under 10,000 people.

Municipios, With Small Ones Suppressed

#Step1: Flag the units we do not trust
municipios <- municipios %>%
  mutate(rate_shown = ifelse(pop < 10000, NA, rate))

#Step2: na.value gives the suppressed units their own colour
ggplot(municipios) +
  geom_sf(aes(fill = rate_shown), colour = NA) +
  scale_fill_viridis_b(breaks = c(0, 10, 25, 50, 100), direction = -1,
                       na.value = "grey85", name = "per 100,000")

Municipios, With Small Ones Suppressed

Part E: Making It Publishable

Everything a Reader Needs

breaks <- classIntervals(states$rate, n = 5,
                       style = "jenks")$brks

ggplot(states) +
  geom_sf(aes(fill = rate)) +
  scale_fill_viridis_b(
    breaks = unique(breaks),
    direction = -1,
    name = "per 100,000") +
  labs(
    title = "Homicide is concentrated in the north-west and the Pacific centre",
    subtitle = "Intentional homicides per 100,000, annual mean 2021-2023",
    caption = "Sources: SESNSP; INEGI Censo 2020") +
  theme_bw()

The Checklist

  • Title states the finding, not the variable name — “Homicide is concentrated in…”, not “Homicide rate by state”
  • Legend has units — per 100,000, not rate
  • Sources in the caption, always, and the period in the subtitle
  • theme_void() — latitude and longitude gridlines are chart junk on a thematic map
  • coord_sf(crs = 6372) — never publish a Mexican map in Web Mercator

Which Format Should You Use?

Use it when
Shapefile someone insists — it is the old standard
GeoJSON small data, web maps, version control
GeoPackage your default — one file, no size limit, multiple layers

Warning

Shapefile limitations that will bite you: column names truncated to 10 characters, a 2 GB size cap, and no proper support for accented characters. Nombre_del_Municipio becomes Nombre_del.

Writing It Back Out

#Step1: The format comes from the extension -- nothing else to set
st_write(states, "homicidios_estados.gpkg", delete_dsn = TRUE)
st_write(states, "homicidios_estados.geojson", delete_dsn = TRUE)

delete_dsn = TRUE overwrites; without it, a second run errors instead of replacing the file.

Tip

Write the layer you analysed, not just the picture of it. A reader who has your .gpkg can check your work; a reader who has your PNG cannot.

Part F: Interactive Maps with leaflet

leaflet: Three Ingredients

#Step1: leaflet only accepts lon/lat. It does its own projecting, so hand it
#       WGS84 -- give it EPSG:6372 and you get a blank map, with no error.
states_wgs84 <- st_transform(states, 4326)

#Step2: colorBin builds a FUNCTION. Give it a rate, it hands back a colour.
#       Same breaks as the static map, so the two tell the same story.
pal <- colorBin("viridis", domain = states_wgs84$rate,
                bins = unique(breaks), reverse = TRUE)

#Step3: One HTML string per state, in the same row order as the data.
#       <strong> is bold, <br/> is a line break -- this is a tooltip, not a plot.
labels <- sprintf("<strong>%s</strong><br/>%.1f per 100,000",
                  states_wgs84$state_name, states_wgs84$rate) %>%
  lapply(htmltools::HTML)

leaflet: Building the Map

#Step4: Start the map and remember the data -- like ggplot(states_wgs84)
leaflet(states_wgs84) %>%
  #Step5: The street map underneath. This is the layer that gives context.
  addTiles() %>%
  #Step6: Draw the states on top of it
  addPolygons(fillColor = ~pal(rate),   # ~ means "look this up in the data"
              fillOpacity = 0.8,        # let the streets show through
              color = "white",          # the border colour
              weight = 1,               # the border thickness
              highlightOptions =        # what happens when the mouse is over
                highlightOptions(weight = 3, color = "#333"),
              label = labels) %>%       # the tooltip from Step3
  #Step7: The key, so the colours mean something
  addLegend(pal = pal, values = ~rate,
            title = "per 100,000", position = "bottomright")

The ~ is leaflet’s version of aes(): this is a column, go and find it. Everything without a ~ is a plain value you are setting once.

The Finished Map

Hover over a state. That is the whole point.

Do Not Forget This

A leaflet map is JavaScript. In a Quarto HTML document it depends on external files — unless you say otherwise:

---
format:
  html:
    embed-resources: true
---

Important

This is the trap from Session 1, and this is the week it bites. Without embed-resources: true, your reader opens the file and sees a blank white rectangle.

What Have We Learned?

  • An sf object nests three classes: sfg → sfc → sf — a geometry, a geometry column, a table
  • A choropleth is three decisions: variable, colour, and breaks
  • Join Mexican data on the INEGI code, never on place names
  • The sf object goes on the left of left_join, or you lose the class
  • read.csv turns "01" into 1: check sum(is.na()) after every join
  • viridis is perceptually uniform, colourblind-safe, and prints in grey — turbo is not
  • Classification changes the story: equal interval put 21 of 32 states in one class; quantile spread them 7-6-6-6-7
  • A count map is a population map in disguise — México is 3rd by count, 13.5 by rate; Colima is 1st by rate with a quarter the homicides
  • leaflet for the reader who wants their own municipio, ggplot for the printed report — put both in the reto
  • But rates on small denominators are unstable: Doctor Coss, population 1,360, “leads” the country at 735
  • GeoPackage is your default format, not shapefile — and st_write picks it from the extension