Drop unused factor levels in a subsetted data frame

RDataframeR FactorR Faq

R Problem Overview

I have a data frame containing a factor. When I create a subset of this dataframe using subset or another indexing function, a new data frame is created. However, the factor variable retains all of its original levels, even when/if they do not exist in the new dataframe.

This causes problems when doing faceted plotting or using functions that rely on factor levels.

What is the most succinct way to remove levels from a factor in the new dataframe?

Here's an example:

df <- data.frame(letters=letters[1:5],

## [1] "a" "b" "c" "d" "e"

subdf <- subset(df, numbers <= 3)
##   letters numbers
## 1       a       1
## 2       b       2
## 3       c       3    

# all levels are still there!
## [1] "a" "b" "c" "d" "e"

R Solutions

Solution 1 - R

Since R version 2.12, there's a droplevels() function.


Solution 2 - R

All you should have to do is to apply factor() to your variable again after subsetting:

> subdf$letters
[1] a b c
Levels: a b c d e
subdf$letters <- factor(subdf$letters)
> subdf$letters
[1] a b c
Levels: a b c


From the factor page example:

factor(ff)      # drops the levels that do not occur

For dropping levels from all factor columns in a dataframe, you can use:

subdf <- subset(df, numbers <= 3)
subdf[] <- lapply(subdf, function(x) if(is.factor(x)) factor(x) else x)

Solution 3 - R

If you don't want this behaviour, don't use factors, use character vectors instead. I think this makes more sense than patching things up afterwards. Try the following before loading your data with read.table or read.csv:

options(stringsAsFactors = FALSE)

The disadvantage is that you're restricted to alphabetical ordering. (reorder is your friend for plots)

Solution 4 - R

It is a known issue, and one possible remedy is provided by drop.levels() in the gdata package where your example becomes

> drop.levels(subdf)
  letters numbers
1       a       1
2       b       2
3       c       3
> levels(drop.levels(subdf)$letters)
[1] "a" "b" "c"

There is also the dropUnusedLevels function in the Hmisc package. However, it only works by altering the subset operator [ and is not applicable here.

As a corollary, a direct approach on a per-column basis is a simple as.factor(as.character(data)):

> levels(subdf$letters)
[1] "a" "b" "c" "d" "e"
> subdf$letters <- as.factor(as.character(subdf$letters))
> levels(subdf$letters)
[1] "a" "b" "c"

Solution 5 - R

Another way of doing the same but with dplyr

subdf <- df %>% filter(numbers <= 3) %>% droplevels()


Also Works ! Thanks to agenis

subdf <- df %>% filter(numbers <= 3) %>% droplevels

Solution 6 - R

For the sake of completeness, now there is also fct_drop in the forcats package http://forcats.tidyverse.org/reference/fct_drop.html.

It differs from droplevels in the way it deals with NA:

f <- factor(c("a", "b", NA), exclude = NULL)

# [1] a    b    <NA>
# Levels: a b <NA>

# [1] a    b    <NA>
# Levels: a b

Solution 7 - R

Here's another way, which I believe is equivalent to the factor(..) approach:

> df <- data.frame(let=letters[1:5], num=1:5)
> subdf <- df[df$num <= 3, ]

> subdf$let <- subdf$let[ , drop=TRUE]

> levels(subdf$let)
[1] "a" "b" "c"

Solution 8 - R

This is obnoxious. This is how I usually do it, to avoid loading other packages:


which gets you:

> subdf$letters
[1] a b c
Levels: a b c

Note that the new levels will replace whatever occupies their index in the old levels(subdf$letters), so something like:


won't work.

This is obviously not ideal when you have lots of levels, but for a few, it's quick and easy.

Solution 9 - R

Looking at the droplevels methods code in the R source you can see it wraps to factor function. That means you can basically recreate the column with factor function.
Below the data.table way to drop levels from all the factor columns.

dt = data.table(letters=factor(letters[1:5]), numbers=seq(1:5))
#[1] "a" "b" "c" "d" "e"
subdt = dt[numbers <= 3]
#[1] "a" "b" "c" "d" "e"

upd.cols = sapply(subdt, is.factor)
subdt[, names(subdt)[upd.cols] := lapply(.SD, factor), .SDcols = upd.cols]
#[1] "a" "b" "c"

Solution 10 - R

here is a way of doing that

varFactor <- factor(letters[1:15])
varFactor <- varFactor[1:5]
varFactor <- varFactor[drop=T]

Solution 11 - R

I wrote utility functions to do this. Now that I know about gdata's drop.levels, it looks pretty similar. Here they are (from here):

present_levels <- function(x) intersect(levels(x), x)

trim_levels <- function(...) UseMethod("trim_levels")

trim_levels.factor <- function(x)  factor(x, levels=present_levels(x))

trim_levels.data.frame <- function(x) {
  for (n in names(x))
    if (is.factor(x[,n]))
      x[,n] = trim_levels(x[,n])

Solution 12 - R

Very interesting thread, I especially liked idea to just factor subselection again. I had the similar problem before and I just converted to character and then back to factor.

   df <- data.frame(letters=letters[1:5],numbers=seq(1:5))
   ## [1] "a" "b" "c" "d" "e"
   subdf <- df[df$numbers <= 3]

Solution 13 - R

Thank you for posting this question. However, none of the above solutions worked for me. I made a workaround for this problem, sharing it in case some else stumbles upon this problem:

For all factor columns that contain levels having zero values in them, you can first convert those columns into character type and then convert them back into factors.

For the above-posted question, just add the following lines of code:

# Convert into character
subdf$letters = as.character(subdf$letters)

# Convert back into factor
subdf$letters = as.factor(subdf$letters)

# Verify the levels in the subset

Solution 14 - R

Unfortunately factor() doesn't seem to work when using rxDataStep of RevoScaleR. I do it in two steps:

  1. Convert to character and store in temporary external data frame (.xdf).

  2. Convert back to factor and store in definitive external data frame. This eliminates any unused factor levels, without loading all the data into memory.

    Step 1) Converts to character, in temporary xdf file:

    rxDataStep(inData = "input.xdf", outFile = "temp.xdf", transforms = list(VAR_X = as.character(VAR_X)), overwrite = T)

    Step 2) Converts back to factor:

    rxDataStep(inData = "temp.xdf", outFile = "output.xdf", transforms = list(VAR_X = as.factor(VAR_X)), overwrite = T)

Solution 15 - R

Have tried most of the examples here if not all but none seem to be working in my case. After struggling for quite some time I have tried using as.character() on the factor column to change it to a col with strings which seems to working just fine.

Not sure for performance issues.

Solution 16 - R

A genuine droplevels function that is much faster than droplevels and does not perform any kind of unnecessary matching or tabulation of values is collapse::fdroplevels. Example:


# wlddev data supplied in collapse, iso3c is a factor
data <- fsubset(wlddev, iso3c %!in% "USA")

microbenchmark(fdroplevels(data), droplevels(data), unit = "relative")
## Unit: relative
##               expr  min       lq     mean   median       uq      max neval cld
##  fdroplevels(data)  1.0  1.00000  1.00000  1.00000  1.00000  1.00000   100  a 
##   droplevels(data) 30.2 29.15873 24.54175 24.86147 22.11553 14.23274   100   b


All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionmedriscollView Question on Stackoverflow
Solution 1 - RRoman LuštrikView Answer on Stackoverflow
Solution 2 - RhatmatrixView Answer on Stackoverflow
Solution 3 - RhadleyView Answer on Stackoverflow
Solution 4 - RDirk EddelbuettelView Answer on Stackoverflow
Solution 5 - RPrradepView Answer on Stackoverflow
Solution 6 - RAurèleView Answer on Stackoverflow
Solution 7 - RarsView Answer on Stackoverflow
Solution 8 - RMatt ParkerView Answer on Stackoverflow
Solution 9 - RjangoreckiView Answer on Stackoverflow
Solution 10 - RDiogoView Answer on Stackoverflow
Solution 11 - RBrendan OConnorView Answer on Stackoverflow
Solution 12 - RDfACView Answer on Stackoverflow
Solution 13 - RSandyView Answer on Stackoverflow
Solution 14 - RJerome SmithView Answer on Stackoverflow
Solution 15 - RNaga PakalapatiView Answer on Stackoverflow
Solution 16 - RSebastianView Answer on Stackoverflow