Efficient list recursion in R with {rrapply}
Introduction
The nested list below shows a small extract from the Mathematics Genealogy Project highlighting the advisor/student genealogy of several famous mathematicians. The mathematician’s given names are present in the "given" attribute of each list element. The numeric values at the leaf elements are the total number of student descendants according to the website as of June 2022. If no descendants are available there is a missing value present at the leaf node.
students <- list(
Bernoulli = structure(list(
Bernoulli = structure(list(
Bernoulli = structure(1L, given = "Daniel"),
Euler = structure(list(
Euler = structure(NA, given = "Johann"),
Lagrange = structure(list(
Fourier = structure(73788L, given = "Jean-Baptiste"),
Plana = structure(NA, given = "Giovanni"),
Poisson = structure(128235L, given = "Simeon")
), given = "Joseph")
), given = "Leonhard")
), given = "Johann"),
Bernoulli = structure(NA, given = "Nikolaus")
), given = "Jacob")
)
str(students, give.attr = FALSE)
#> List of 1
#> $ Bernoulli:List of 2
#> ..$ Bernoulli:List of 2
#> .. ..$ Bernoulli: int 1
#> .. ..$ Euler :List of 2
#> .. .. ..$ Euler : logi NA
#> .. .. ..$ Lagrange:List of 3
#> .. .. .. ..$ Fourier: int 73788
#> .. .. .. ..$ Plana : logi NA
#> .. .. .. ..$ Poisson: int 128235
#> ..$ Bernoulli: logi NAAs an exercise in list recursion, consider the following simple data exploration question:
Filter all descendants of ‘Leonhard Euler’ and replace all missing values by zero while maintaining the list structure.
Here is a possible (not so efficient) base R solution using recursion with the Recall() function:
filter_desc_euler <- \(x) {
i <- 1
while(i <= length(x)) {
if(identical(names(x)[i], "Euler") & identical(attr(x[[i]], "given"), "Leonhard")) {
x[[i]] <- rapply(x[[i]], f = \(x) replace(x, is.na(x), 0), how = "replace")
i <- i + 1
} else {
if(is.list(x[[i]])) {
val <- Recall(x[[i]])
x[[i]] <- val
i <- i + !is.null(val)
} else {
x[[i]] <- NULL
}
if(all(sapply(x, is.null))) {
x <- NULL
}
}
}
return(x)
}
str(filter_desc_euler(students), give.attr = FALSE)
#> List of 1
#> $ Bernoulli:List of 1
#> ..$ Bernoulli:List of 1
#> .. ..$ Euler:List of 2
#> .. .. ..$ Euler : num 0
#> .. .. ..$ Lagrange:List of 3
#> .. .. .. ..$ Fourier: num 73788
#> .. .. .. ..$ Plana : num 0
#> .. .. .. ..$ Poisson: num 128235This works, but is hardly the kind of convoluted code we would like to write for such a seemingly simple question. Moreover, this code is not very easy to follow, which can make updating or modifying it quite a time-consuming and error-prone task.
An alternative approach would be to unnest the list into a more manageable (e.g. rectangular) format or use specialized packages, such as igraph or data.tree, to make pruning or modifying node entries more straightforward. Note that attention must be paid to correctly include the node attributes in the transformed object as the node names themselves are not unique in this example. This is a sensible approach and usually the way to go when cleaning or tidying up the data, but for fast prototyping and data exploration tasks we may want to keep the list in its original format to reduce the number of processing steps and minimize the code complexity. Another reason to maintain a nested data structure may be that we wish to use a certain data visualization or data exporting function and the function expects its input in a nested format.
The recursive function above makes use of base rapply(), a member of the apply-family of functions in R, that allows to apply a function recursively to the elements of a nested list and decide how the returned result is structured. Although sometimes useful, the rapply() function is not sufficiently flexible for many list recursion tasks in practice, as also demonstrated in the above example. In this context, the rrapply() function in the minimal rrapply-package attempts to revisit and extend base rapply() to make it more generally applicable for list recursion in the wild. The rrapply() function builds upon R’s native C implementation ofrapply() and for this reason requires no other external dependencies.
When to use rrapply()
Below, we make use of the two datasets renewable_energy_by_country and pokedex included in the rrapply-package.
renewable_energy_by_countryis a nested list containing the renewable energy shares per country (% of total energy consumption) in 2016. The data is publicly available at the United Nations Open SDG Data Hub. The 249 countries and areas are structured based on their geographical locations according to the United Nations M49 standard. The numeric values listed for each country are percentages, if no data is available the value of the country isNA.pokedexis a nested list containing various property values for each of the 151 original Pokémon available (in .json) from https://github.com/Biuni/PokemonGO-Pokedex.
library(rrapply)
data("renewable_energy_by_country")For convenience, we subset only the values for countries and areas in Oceania from renewable_energy_by_country,
renewable_oceania <- renewable_energy_by_country[["World"]]["Oceania"]
str(renewable_oceania, list.len = 3, give.attr = FALSE)
#> List of 1
#> $ Oceania:List of 4
#> ..$ Australia and New Zealand:List of 6
#> .. ..$ Australia : num 9.32
#> .. ..$ Christmas Island : logi NA
#> .. ..$ Cocos (Keeling) Islands : logi NA
#> .. .. [list output truncated]
#> ..$ Melanesia :List of 5
#> .. ..$ Fiji : num 24.4
#> .. ..$ New Caledonia : num 4.03
#> .. ..$ Papua New Guinea: num 50.3
#> .. .. [list output truncated]
#> ..$ Micronesia :List of 8
#> .. ..$ Guam : num 3.03
#> .. ..$ Kiribati : num 45.4
#> .. ..$ Marshall Islands : num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]List pruning and unnesting
how = "prune"
With base rapply(), there is no convenient way to prune or filter elements from the input list. The rrapply() function adds an option how = "prune" to prune all list elements not subject to application of the function f from a nested list. The original list structure is retained, similar to the non-pruned versions how = "replace" and how = "list". Using how = "prune" and the same syntax as in rapply(), we can easily drop all missing values from the list while preserving the nested list structure:
## drop all logical NA's while preserving list structure
rrapply(
renewable_oceania,
f = \(x) x,
classes = "numeric",
how = "prune"
) |>
str(list.len = 3, give.attr = FALSE)
#> List of 1
#> $ Oceania:List of 4
#> ..$ Australia and New Zealand:List of 2
#> .. ..$ Australia : num 9.32
#> .. ..$ New Zealand: num 32.8
#> ..$ Melanesia :List of 5
#> .. ..$ Fiji : num 24.4
#> .. ..$ New Caledonia : num 4.03
#> .. ..$ Papua New Guinea: num 50.3
#> .. .. [list output truncated]
#> ..$ Micronesia :List of 7
#> .. ..$ Guam : num 3.03
#> .. ..$ Kiribati : num 45.4
#> .. ..$ Marshall Islands : num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]Remark: if the f function is missing, it defaults to the identity function. That is, the f argument can be dropped when no (non-trivial) function is applied to the list elements.
how = "flatten"
Instead, we can set how = "flatten" to return a flattened unnested version of the pruned list. This is more efficient than first returning the pruned list with how = "prune" and unlisting or flattening the list in a subsequent step.
## drop all logical NA's and return unnested list
rrapply(
renewable_oceania,
classes = "numeric",
how = "flatten"
) |>
head(n = 10)
#> Australia New Zealand Fiji New Caledonia
#> 9.32 32.76 24.36 4.03
#> Papua New Guinea Solomon Islands Vanuatu Guam
#> 50.34 65.73 33.67 3.03
#> Kiribati Marshall Islands
#> 45.43 11.75Hint: the options argument allows to tune several options specific to certain choices of how. With how = "flatten", we can choose to not coerce the flattened list to a vector and/or to include all parent list names in the result similar to how = "unlist" but then with a custom name separator.
## flatten to simple list with full names
rrapply(
renewable_oceania,
classes = "numeric",
how = "flatten",
options = list(namesep = ".", simplify = FALSE)
) |>
str(list.len = 10, give.attr = FALSE)
#> List of 22
#> $ Oceania.Australia and New Zealand.Australia : num 9.32
#> $ Oceania.Australia and New Zealand.New Zealand : num 32.8
#> $ Oceania.Melanesia.Fiji : num 24.4
#> $ Oceania.Melanesia.New Caledonia : num 4.03
#> $ Oceania.Melanesia.Papua New Guinea : num 50.3
#> $ Oceania.Melanesia.Solomon Islands : num 65.7
#> $ Oceania.Melanesia.Vanuatu : num 33.7
#> $ Oceania.Micronesia.Guam : num 3.03
#> $ Oceania.Micronesia.Kiribati : num 45.4
#> $ Oceania.Micronesia.Marshall Islands : num 11.8
#> [list output truncated]how = "melt"
Using how = "melt", we can return a melted data.frame of the pruned list similar in format to reshape2::melt() applied to a nested list. The rows of the melted data.frame contain the parent node paths of the elements in the pruned list. The "value" column contains the values of the terminal or leaf nodes analogous to the flattened list returned by how = "flatten".
## drop all logical NA's and return melted data.frame
oceania_melt <- rrapply(
renewable_oceania,
classes = "numeric",
how = "melt"
)
head(oceania_melt, n = 10)
#> L1 L2 L3 value
#> 1 Oceania Australia and New Zealand Australia 9.32
#> 2 Oceania Australia and New Zealand New Zealand 32.76
#> 3 Oceania Melanesia Fiji 24.36
#> 4 Oceania Melanesia New Caledonia 4.03
#> 5 Oceania Melanesia Papua New Guinea 50.34
#> 6 Oceania Melanesia Solomon Islands 65.73
#> 7 Oceania Melanesia Vanuatu 33.67
#> 8 Oceania Micronesia Guam 3.03
#> 9 Oceania Micronesia Kiribati 45.43
#> 10 Oceania Micronesia Marshall Islands 11.75Remark: if no names are present in a certain sublist of the input list, how = "melt" replaces the names in the melted data.frame by list element indices "1", "2", etc.
## drop some area names
renewable_oceania1 <- renewable_oceania
renewable_oceania1[[1]] <- unname(renewable_oceania[[1]])
## drop all logical NA's and return melted data.frame
rrapply(
renewable_oceania1,
classes = "numeric",
how = "melt"
) |>
head(n = 10)
#> L1 L2 L3 value
#> 1 Oceania 1 Australia 9.32
#> 2 Oceania 1 New Zealand 32.76
#> 3 Oceania 2 Fiji 24.36
#> 4 Oceania 2 New Caledonia 4.03
#> 5 Oceania 2 Papua New Guinea 50.34
#> 6 Oceania 2 Solomon Islands 65.73
#> 7 Oceania 2 Vanuatu 33.67
#> 8 Oceania 3 Guam 3.03
#> 9 Oceania 3 Kiribati 45.43
#> 10 Oceania 3 Marshall Islands 11.75A melted data.frame can be used to reconstruct a nested list with how = "unmelt". No skeleton object as e.g. required by relist() is needed, only an ordinary data.frame in the format returned by how = "melt". This option can be convenient to construct nested lists from a rectangular data.frame format without having to resort to recursive function definitions.
## reconstruct nested list from melted data.frame
rrapply(oceania_melt, how = "unmelt") |>
str(list.len = 3, give.attr = FALSE)
#> List of 1
#> $ Oceania:List of 4
#> ..$ Australia and New Zealand:List of 2
#> .. ..$ Australia : num 9.32
#> .. ..$ New Zealand: num 32.8
#> ..$ Melanesia :List of 5
#> .. ..$ Fiji : num 24.4
#> .. ..$ New Caledonia : num 4.03
#> .. ..$ Papua New Guinea: num 50.3
#> .. .. [list output truncated]
#> ..$ Micronesia :List of 7
#> .. ..$ Guam : num 3.03
#> .. ..$ Kiribati : num 45.4
#> .. ..$ Marshall Islands : num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]how = "bind"
Nested lists containing repeated observations can be unnested with how = "bind". Each repeated sublist is expanded
as a single row in a wide data.frame and identical sublist component names are aligned as individual columns. By default, the list layer containing the repeated observations is identified by the minimal depth detected across leaf elements, but this can also be overridden using the coldepth option in the options argument. Note that the returned data.frame is similar in format to repeated application of tidyr::unnest_wider() to a nested data.frame, with the same coercion rules applied to the individual columns as `how = “unlist”.
data("pokedex")
str(pokedex, list.len = 3)
#> List of 1
#> $ pokemon:List of 151
#> ..$ :List of 16
#> .. ..$ id : int 1
#> .. ..$ num : chr "001"
#> .. ..$ name : chr "Bulbasaur"
#> .. .. [list output truncated]
#> ..$ :List of 17
#> .. ..$ id : int 2
#> .. ..$ num : chr "002"
#> .. ..$ name : chr "Ivysaur"
#> .. .. [list output truncated]
#> ..$ :List of 15
#> .. ..$ id : int 3
#> .. ..$ num : chr "003"
#> .. ..$ name : chr "Venusaur"
#> .. .. [list output truncated]
#> .. [list output truncated]## unnest list to wide data.frame
rrapply(pokedex, how = "bind")[, c(1:3, 5:8)] |>
head(n = 10)
#> id num name type height weight candy
#> 1 1 001 Bulbasaur Grass, Poison 0.71 m 6.9 kg Bulbasaur Candy
#> 2 2 002 Ivysaur Grass, Poison 0.99 m 13.0 kg Bulbasaur Candy
#> 3 3 003 Venusaur Grass, Poison 2.01 m 100.0 kg Bulbasaur Candy
#> 4 4 004 Charmander Fire 0.61 m 8.5 kg Charmander Candy
#> 5 5 005 Charmeleon Fire 1.09 m 19.0 kg Charmander Candy
#> 6 6 006 Charizard Fire, Flying 1.70 m 90.5 kg Charmander Candy
#> 7 7 007 Squirtle Water 0.51 m 9.0 kg Squirtle Candy
#> 8 8 008 Wartortle Water 0.99 m 22.5 kg Squirtle Candy
#> 9 9 009 Blastoise Water 1.60 m 85.5 kg Squirtle Candy
#> 10 10 010 Caterpie Bug 0.30 m 2.9 kg Caterpie CandyHint: setting namecols = TRUE in the options argument includes the parent list names associated to each row in the wide data.frame as individual columns L1, L2, etc.
## bind to data.frame including parent columns
pokemon_evolutions <- rrapply(
pokedex,
how = "bind",
options = list(namecols = TRUE, coldepth = 5)
)
head(pokemon_evolutions, n = 10)
#> L1 L2 L3 L4 num name
#> 1 pokemon 1 next_evolution 1 002 Ivysaur
#> 2 pokemon 1 next_evolution 2 003 Venusaur
#> 3 pokemon 2 prev_evolution 1 001 Bulbasaur
#> 4 pokemon 2 next_evolution 1 003 Venusaur
#> 5 pokemon 3 prev_evolution 1 001 Bulbasaur
#> 6 pokemon 3 prev_evolution 2 002 Ivysaur
#> 7 pokemon 4 next_evolution 1 005 Charmeleon
#> 8 pokemon 4 next_evolution 2 006 Charizard
#> 9 pokemon 5 prev_evolution 1 004 Charmander
#> 10 pokemon 5 next_evolution 1 006 CharizardThis can be useful to unnest repeated list elements at multiple nested list levels and join the results into a single data.frame:
## merge pokemon evolutions with pokemon names
rrapply(
pokedex,
how = "bind",
options = list(namecols = TRUE)
)[, c("L1", "L2", "name")] |>
merge(
pokemon_evolutions[, c("L1", "L2", "L3", "name")],
by = c("L1", "L2"),
suffixes = c("", ".evolution")
) |>
head(n = 10)
#> L1 L2 name L3 name.evolution
#> 1 pokemon 1 Bulbasaur next_evolution Ivysaur
#> 2 pokemon 1 Bulbasaur next_evolution Venusaur
#> 3 pokemon 10 Caterpie next_evolution Metapod
#> 4 pokemon 10 Caterpie next_evolution Butterfree
#> 5 pokemon 100 Voltorb next_evolution Electrode
#> 6 pokemon 101 Electrode prev_evolution Voltorb
#> 7 pokemon 102 Exeggcute next_evolution Exeggutor
#> 8 pokemon 103 Exeggutor prev_evolution Exeggcute
#> 9 pokemon 104 Cubone next_evolution Marowak
#> 10 pokemon 105 Marowak prev_evolution CuboneCondition function
Base rapply() allows to apply a function f to list elements of certain types or classes via the classes argument. rrapply() generalizes this concept via an additional condition argument, which accepts any function to use as a condition or predicate to apply f to a subset of list elements. Conceptually, the f function is applied to all leaf elements for which the condition function exactly evaluates to TRUE similar to isTRUE(). If the condition argument is missing, f is applied to all leaf elements. In combination with how = "prune", the condition function provides additional flexibility in selecting and filtering elements from a nested list,
## drop all NA's using condition function
rrapply(
renewable_oceania,
condition = \(x) !is.na(x),
how = "prune"
) |>
str(list.len = 3, give.attr = FALSE)
#> List of 1
#> $ Oceania:List of 4
#> ..$ Australia and New Zealand:List of 2
#> .. ..$ Australia : num 9.32
#> .. ..$ New Zealand: num 32.8
#> ..$ Melanesia :List of 5
#> .. ..$ Fiji : num 24.4
#> .. ..$ New Caledonia : num 4.03
#> .. ..$ Papua New Guinea: num 50.3
#> .. .. [list output truncated]
#> ..$ Micronesia :List of 7
#> .. ..$ Guam : num 3.03
#> .. ..$ Kiribati : num 45.4
#> .. ..$ Marshall Islands : num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]More interesting is to consider a condition that cannot also be defined using the classes argument. For instance, we can filter all countries with values that satisfy a certain numeric condition:
## filter all countries with values above 85%
rrapply(
renewable_energy_by_country,
condition = \(x) x > 85,
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ World:List of 1
#> ..$ Africa:List of 1
#> .. ..$ Sub-Saharan Africa:List of 3
#> .. .. ..$ Eastern Africa:List of 7
#> .. .. .. ..$ Burundi : num 89.2
#> .. .. .. ..$ Ethiopia : num 91.9
#> .. .. .. ..$ Rwanda : num 86
#> .. .. .. ..$ Somalia : num 94.7
#> .. .. .. ..$ Uganda : num 88.6
#> .. .. .. ..$ United Republic of Tanzania: num 86.1
#> .. .. .. ..$ Zambia : num 88.5
#> .. .. ..$ Middle Africa :List of 2
#> .. .. .. ..$ Chad : num 85.3
#> .. .. .. ..$ Democratic Republic of the Congo: num 97
#> .. .. ..$ Western Africa:List of 1
#> .. .. .. ..$ Guinea-Bissau: num 86.5
## or by passing arguments to condition via ...
rrapply(
renewable_energy_by_country,
condition = "==",
e2 = 0,
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ World:List of 4
#> ..$ Americas:List of 1
#> .. ..$ Latin America and the Caribbean:List of 1
#> .. .. ..$ Caribbean:List of 1
#> .. .. .. ..$ Antigua and Barbuda: num 0
#> ..$ Asia :List of 1
#> .. ..$ Western Asia:List of 4
#> .. .. ..$ Bahrain: num 0
#> .. .. ..$ Kuwait : num 0
#> .. .. ..$ Oman : num 0
#> .. .. ..$ Qatar : num 0
#> ..$ Europe :List of 2
#> .. ..$ Northern Europe:List of 1
#> .. .. ..$ Channel Islands:List of 1
#> .. .. .. ..$ Guernsey: num 0
#> .. ..$ Southern Europe:List of 1
#> .. .. ..$ Gibraltar: num 0
#> ..$ Oceania :List of 2
#> .. ..$ Micronesia:List of 1
#> .. .. ..$ Northern Mariana Islands: num 0
#> .. ..$ Polynesia :List of 1
#> .. .. ..$ Wallis and Futuna Islands: num 0Note that the NA elements are not returned, as the condition function does not evaluate to TRUE for NA values.
As the condition function is a generalization of the classes argument, it remains possible to use deflt together with how = "list" or how = "unlist" to set a default value to all leaf elements for which the condition is not TRUE:
## replace all NA elements by zero
rrapply(
renewable_oceania,
condition = Negate(is.na),
deflt = 0,
how = "list"
) |>
str(list.len = 3, give.attr = FALSE)
#> List of 1
#> $ Oceania:List of 4
#> ..$ Australia and New Zealand:List of 6
#> .. ..$ Australia : num 9.32
#> .. ..$ Christmas Island : num 0
#> .. ..$ Cocos (Keeling) Islands : num 0
#> .. .. [list output truncated]
#> ..$ Melanesia :List of 5
#> .. ..$ Fiji : num 24.4
#> .. ..$ New Caledonia : num 4.03
#> .. ..$ Papua New Guinea: num 50.3
#> .. .. [list output truncated]
#> ..$ Micronesia :List of 8
#> .. ..$ Guam : num 3.03
#> .. ..$ Kiribati : num 45.4
#> .. ..$ Marshall Islands : num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]To be consistent with base rapply(), the deflt argument can still only be used in combination with how = "list" or how = "unlist".
Using the ... argument
The first argument to f always evaluates to the content of the list element to which f is applied. Any further arguments, (besides the special arguments .xname, .xpos, .xparents and .xsiblings discussed below), that are independent of the list content can be supplied via the ... argument. Since rrapply() accepts a function in two of its arguments f and condition, any arguments defined via the ... need to be defined as function arguments in both the f and condition functions (if existing), even if they are not used in the function itself.
To clarify, consider the following example which replaces all missing values by a value defined in a separate argument newvalue:
## this is not ok!
tryCatch({
rrapply(
renewable_oceania,
condition = is.na,
f = \(x, newvalue) newvalue,
newvalue = 0,
how = "replace"
)
}, error = function(error) error$message)
#> [1] "2 arguments passed to 'is.na' which requires 1"
## this is ok
rrapply(
renewable_oceania,
condition = \(x, newvalue) is.na(x),
f = \(x, newvalue) newvalue,
newvalue = 0,
how = "replace"
) |>
str(list.len = 3, give.attr = FALSE)
#> List of 1
#> $ Oceania:List of 4
#> ..$ Australia and New Zealand:List of 6
#> .. ..$ Australia : num 9.32
#> .. ..$ Christmas Island : num 0
#> .. ..$ Cocos (Keeling) Islands : num 0
#> .. .. [list output truncated]
#> ..$ Melanesia :List of 5
#> .. ..$ Fiji : num 24.4
#> .. ..$ New Caledonia : num 4.03
#> .. ..$ Papua New Guinea: num 50.3
#> .. .. [list output truncated]
#> ..$ Micronesia :List of 8
#> .. ..$ Guam : num 3.03
#> .. ..$ Kiribati : num 45.4
#> .. ..$ Marshall Islands : num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]Special arguments .xname, .xpos, .xparents and .xsiblings
With base rapply(), the f function only has access to the content of the list element under evaluation, and there is no convenient way to access its name or location in the nested list from inside the f function. To overcome this limitation, rrapply() defines the special arguments .xname, .xpos, .xparents and .xsiblings inside the f and condition functions (in addition to the principal function argument):
.xnameevaluates to the name of the list element;.xposevaluates to the position of the element in the nested list structured as an integer vector;.xparentsevaluates to a vector of parent list names in the path to the current list element;.xsiblingsevaluates to the parent list containing the current list element and its direct siblings.
Using the .xname and .xpos arguments, we can transform or filter list elements based on their names and/or positions in the nested list:
## apply f based on element's name
rrapply(
renewable_oceania,
condition = \(x) !is.na(x),
f = \(x, .xname) sprintf("Renewable energy in %s: %.2f%%", .xname, x),
how = "flatten"
) |>
head(n = 5)
#> Australia
#> "Renewable energy in Australia: 9.32%"
#> New Zealand
#> "Renewable energy in New Zealand: 32.76%"
#> Fiji
#> "Renewable energy in Fiji: 24.36%"
#> New Caledonia
#> "Renewable energy in New Caledonia: 4.03%"
#> Papua New Guinea
#> "Renewable energy in Papua New Guinea: 50.34%"
## filter elements by name
rrapply(
renewable_energy_by_country,
condition = \(x, .xname) .xname %in% c("Belgium", "Netherlands", "Luxembourg"),
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ World:List of 1
#> ..$ Europe:List of 1
#> .. ..$ Western Europe:List of 3
#> .. .. ..$ Belgium : num 9.14
#> .. .. ..$ Luxembourg : num 13.5
#> .. .. ..$ Netherlands: num 5.78Knowing that Europe is located at renewable_energy_by_country[[c(1, 5)]], we can filter all European countries with a renewable energy share above 50% using the .xpos argument as follows,
## filter European countries > 50% using .xpos
rrapply(
renewable_energy_by_country,
condition = \(x, .xpos) identical(.xpos[1:2], c(1L, 5L)) && x > 50,
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ World:List of 1
#> ..$ Europe:List of 2
#> .. ..$ Northern Europe:List of 3
#> .. .. ..$ Iceland: num 78.1
#> .. .. ..$ Norway : num 59.5
#> .. .. ..$ Sweden : num 51.4
#> .. ..$ Western Europe :List of 1
#> .. .. ..$ Liechtenstein: num 62.9This can be done more conveniently using the .xparents argument, which this does not require looking up the location of Europe in the nested list,
## filter European countries > 50% using .xparents
rrapply(
renewable_energy_by_country,
condition = function(x, .xparents) "Europe" %in% .xparents && x > 50,
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ World:List of 1
#> ..$ Europe:List of 2
#> .. ..$ Northern Europe:List of 3
#> .. .. ..$ Iceland: num 78.1
#> .. .. ..$ Norway : num 59.5
#> .. .. ..$ Sweden : num 51.4
#> .. ..$ Western Europe :List of 1
#> .. .. ..$ Liechtenstein: num 62.9Using the .xpos argument, we can quickly look up the position of a specific element in the nested list,
## return position of Sweden in list
rrapply(
renewable_energy_by_country,
condition = \(x, .xname) .xname == "Sweden",
f = \(x, .xpos) .xpos,
how = "flatten"
)
#> $Sweden
#> [1] 1 5 2 14Using the .xsiblings argument, we can look up the direct neighbors of an element in the nested list,
## look up neighbors of Sweden in list
rrapply(
renewable_energy_by_country,
condition = \(x, .xsiblings) "Sweden" %in% names(.xsiblings),
how = "flatten"
) |>
head(n = 10)
#> Aland Islands Denmark Estonia Faroe Islands Finland
#> NA 33.06 26.55 4.24 42.03
#> Iceland Ireland Isle of Man Latvia Lithuania
#> 78.07 8.65 4.30 38.48 31.42We can also use the .xpos argument to determine the maximum depth of the list or the length of the longest sublist as follows,
## maximum list depth
rrapply(
renewable_energy_by_country,
f = \(x, .xpos) length(.xpos),
how = "unlist"
) |>
max()
#> [1] 5
## longest sublist length
rrapply(
renewable_energy_by_country,
f = \(x, .xpos) max(.xpos),
how = "unlist"
) |>
max()
#> [1] 28When unnesting nested lists with how = "bind", the .xname, .xpos or .xparents arguments can be useful to decide which list elements to include in the unnested data.frame:
## filter elements and unnest list
rrapply(
pokedex,
condition = \(x, .xpos, .xname) length(.xpos) < 4 & .xname %in% c("num", "name", "type"),
how = "bind"
) |>
head()
#> num name type
#> 1 001 Bulbasaur Grass, Poison
#> 2 002 Ivysaur Grass, Poison
#> 3 003 Venusaur Grass, Poison
#> 4 004 Charmander Fire
#> 5 005 Charmeleon Fire
#> 6 006 Charizard Fire, FlyingModifying list elements
By default, both base rapply() and rrapply() recurse into any list-like element. Setting classes = "list" in rrapply() overrides this behavior and applies the f function to any list element (i.e. a sublist) that satisfies the condition argument. If the condition is not satisfied for a list element, rrapply() recurses further into the sublist, applies f to the elements that satisfy condition and so on. The use of classes = "list" signals the rrapply() function not to descend into list objects by default. For this reason this behavior can only be triggered via the classes argument and not through the use of e.g. condition = is.list.
The mode classes = "list" can be useful to e.g. collapse sublists or calculate summary statistics across elements in a nested list:
## calculate mean value of Europe
rrapply(
renewable_energy_by_country,
condition = \(x, .xname) .xname == "Europe",
f = \(x) mean(unlist(x), na.rm = TRUE),
classes = "list",
how = "flatten"
)
#> Europe
#> 22.36565Note that the principal argument in the f function now evaluates to a list. For this reason, we first have to unlist the sublist before calculating the mean.
To calculate the mean renewable energy shares for each continent, we can make use of the fact that the .xpos vector of each continent has length (i.e. depth) 2:
## calculate mean value for each continent
## (Antartica's value is missing)
rrapply(
renewable_energy_by_country,
condition = \(x, .xpos) length(.xpos) == 2,
f = \(x) mean(unlist(x), na.rm = TRUE),
classes = "list"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ World:List of 6
#> ..$ Africa : num 54.3
#> ..$ Americas : num 18.2
#> ..$ Antarctica: logi NA
#> ..$ Asia : num 17.9
#> ..$ Europe : num 22.4
#> ..$ Oceania : num 17.8Remark: if classes = "list", the f function is only applied to the (non-terminal) list elements. To apply f to both terminal and non-terminal elements in the nested list, we can include additional classes, such as classes = c("list", "numeric", "character"). To apply f to any terminal and non-terminal element in the nested list, we can even combine classes = c("list", "ANY"). To illustrate, we search across all list elements for the country or region with M49-code "155":
## filter country or region by M49-code
rrapply(
renewable_energy_by_country,
condition = \(x) attr(x, "M49-code") == "155",
f = \(x, .xname) .xname,
classes = c("list", "ANY"),
how = "unlist"
)
#> World.Europe.Western Europe
#> "Western Europe"As a more complex example, we unnest the Pokémon evolutions in pokedex into a wide data.frame by returning the sublists with Pokémon evolutions as character vectors:
## simplify pokemon evolutions to character vectors
rrapply(
pokedex,
condition = \(x, .xname) .xname %in% c("name", "next_evolution", "prev_evolution"),
f = \(x) if(is.list(x)) sapply(x, `[[`, "name") else x,
classes = c("list", "character"),
how = "bind"
) |>
head(n = 9)
#> name next_evolution prev_evolution
#> 1 Bulbasaur Ivysaur, Venusaur NA
#> 2 Ivysaur Venusaur Bulbasaur
#> 3 Venusaur NA Bulbasaur, Ivysaur
#> 4 Charmander Charmeleon, Charizard NA
#> 5 Charmeleon Charizard Charmander
#> 6 Charizard NA Charmander, Charmeleon
#> 7 Squirtle Wartortle, Blastoise NA
#> 8 Wartortle Blastoise Squirtle
#> 9 Blastoise NA Squirtle, WartortleHint: as data.frames are also list-like objects, rrapply() applies f to individual data.frame columns by default. Set classes = "data.frame" to avoid this behavior and apply the f and condition functions to complete data.frame objects instead of individual data.frame columns.
## create a nested list of data.frames
oceania_df <- rrapply(
renewable_oceania,
condition = \(x, .xpos) length(.xpos) == 2,
f = \(x) data.frame(name = names(x), value = unlist(x)),
classes = "list",
how = "replace"
)
## this does not work!
tryCatch({
rrapply(
oceania_df,
f = function(x) subset(x, !is.na(value)), ## filter NA-rows of data.frame
how = "replace"
)
}, error = function(error) error$message)
#> [1] "object 'value' not found"
## this does work
rrapply(
oceania_df,
f = function(x) subset(x, !is.na(value)),
classes = "data.frame",
how = "replace"
)[[1]][1:2]
#> $`Australia and New Zealand`
#> name value
#> Australia Australia 9.32
#> New Zealand New Zealand 32.76
#>
#> $Melanesia
#> name value
#> Fiji Fiji 24.36
#> New Caledonia New Caledonia 4.03
#> Papua New Guinea Papua New Guinea 50.34
#> Solomon Islands Solomon Islands 65.73
#> Vanuatu Vanuatu 33.67Recursive list updating
how = "recurse"
If classes = "list" and how = "recurse", rrapply() applies the f function to any list element that satisfies the condition argument, but recurses further into any updated list element after application of f. This can be useful to e.g. recursively update the class or other attributes of all elements in a nested list:
## recursively remove all list attributes
rrapply(
renewable_oceania,
f = \(x) c(x),
classes = c("list", "ANY"),
how = "recurse"
) |>
str(list.len = 3, give.attr = TRUE)
#> List of 1
#> $ Oceania:List of 4
#> ..$ Australia and New Zealand:List of 6
#> .. ..$ Australia : num 9.32
#> .. ..$ Christmas Island : logi NA
#> .. ..$ Cocos (Keeling) Islands : logi NA
#> .. .. [list output truncated]
#> ..$ Melanesia :List of 5
#> .. ..$ Fiji : num 24.4
#> .. ..$ New Caledonia : num 4.03
#> .. ..$ Papua New Guinea: num 50.3
#> .. .. [list output truncated]
#> ..$ Micronesia :List of 8
#> .. ..$ Guam : num 3.03
#> .. ..$ Kiribati : num 45.4
#> .. ..$ Marshall Islands : num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]how = "names"
The option how = "names" is a special case of how = "recurse", where the value of f is used to replace the name of the evaluated list element instead of its content (as with all other how options). By default, how = "names" uses classes = c("list", "ANY") in order to allow updating of all names in the nested list.
## recursively replace all names by M49-codes
rrapply(
renewable_oceania,
f = \(x) attr(x, "M49-code"),
how = "names"
) |>
str(list.len = 3, give.attr = FALSE)
#> List of 1
#> $ 009:List of 4
#> ..$ 053:List of 6
#> .. ..$ 036: num 9.32
#> .. ..$ 162: logi NA
#> .. ..$ 166: logi NA
#> .. .. [list output truncated]
#> ..$ 054:List of 5
#> .. ..$ 242: num 24.4
#> .. ..$ 540: num 4.03
#> .. ..$ 598: num 50.3
#> .. .. [list output truncated]
#> ..$ 057:List of 8
#> .. ..$ 316: num 3.03
#> .. ..$ 296: num 45.4
#> .. ..$ 584: num 11.8
#> .. .. [list output truncated]
#> .. [list output truncated]Conclusion
To conclude, we return to the list recursion exercise in the first section. Using rrapply(), one possible solution is to split the question into two steps as follows:
## look up position of Euler (Leonhard)
euler <- rrapply(
students,
condition = \(x, .xname) .xname == "Euler" && attr(x, "given") == "Leonhard",
f = \(x, .xpos) .xpos,
classes = "list",
how = "flatten"
)[["Euler"]]
## filter descendants of Euler (Leonhard) and replace missing values by zero
rrapply(
students,
condition = \(x, .xpos) identical(.xpos[seq_along(euler)], euler),
f = \(x) replace(x, is.na(x), 0),
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ Bernoulli:List of 1
#> ..$ Bernoulli:List of 1
#> .. ..$ Euler:List of 2
#> .. .. ..$ Euler : num 0
#> .. .. ..$ Lagrange:List of 3
#> .. .. .. ..$ Fourier: num 73788
#> .. .. .. ..$ Plana : num 0
#> .. .. .. ..$ Poisson: num 128235Knowing that Johann Euler is a descendant of Leonhard Euler, we can further simplify this into a single function call using the .xparents argument:
## filter descendants of Euler (Leonhard) and replace missing values by zero
rrapply(
students,
condition = \(x, .xparents) "Euler" %in% .xparents,
f = \(x) replace(x, is.na(x), 0),
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ Bernoulli:List of 1
#> ..$ Bernoulli:List of 1
#> .. ..$ Euler:List of 2
#> .. .. ..$ Euler : num 0
#> .. .. ..$ Lagrange:List of 3
#> .. .. .. ..$ Fourier: num 73788
#> .. .. .. ..$ Plana : num 0
#> .. .. .. ..$ Poisson: num 128235Or alternatively, we could first update the names of the elements in the nested list to include both first and last names and then prune the list in a second step:
## include first names in list element names
students_fullnames <- rrapply(
students,
f = \(x, .xname) paste(attr(x, "given"), .xname),
how = "names"
)
## filter descendants of Euler (Leonhard) and replace missing values by zero
rrapply(
students_fullnames,
condition = \(x, .xparents) "Leonhard Euler" %in% .xparents,
f = \(x) replace(x, is.na(x), 0),
how = "prune"
) |>
str(give.attr = FALSE)
#> List of 1
#> $ Jacob Bernoulli:List of 1
#> ..$ Johann Bernoulli:List of 1
#> .. ..$ Leonhard Euler:List of 2
#> .. .. ..$ Johann Euler : num 0
#> .. .. ..$ Joseph Lagrange:List of 3
#> .. .. .. ..$ Jean-Baptiste Fourier: num 73788
#> .. .. .. ..$ Giovanni Plana : num 0
#> .. .. .. ..$ Simeon Poisson : num 128235Additional details
The latest stable version of the rrapply-package is available on CRAN. Additional details and examples on how to use the rrapply() function can be found at https://jorischau.github.io/rrapply/ and a quick reference sheet can be downloaded from the github repository at https://github.com/JorisChau/rrapply/.
Session Info
sessionInfo()
#> R version 4.2.1 (2022-06-23)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 20.04.4 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] rrapply_1.2.5
#>
#> loaded via a namespace (and not attached):
#> [1] bookdown_0.27 digest_0.6.29 R6_2.5.1 jsonlite_1.8.0
#> [5] magrittr_2.0.3 evaluate_0.15 blogdown_1.10 stringi_1.7.8
#> [9] rlang_1.0.4 cli_3.3.0 rstudioapi_0.13 jquerylib_0.1.4
#> [13] bslib_0.3.1 rmarkdown_2.14 tools_4.2.1 stringr_1.4.0
#> [17] xfun_0.31 yaml_2.3.5 fastmap_1.1.0 compiler_4.2.1
#> [21] htmltools_0.5.2 knitr_1.39 sass_0.4.1