README.Rmd

---
output: github_document
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = ">",
  fig.path = "man/figures/README-"
)
```

```{r library, echo = FALSE}
library(packageRank)
```

## packageRank: compute, visualize and contextualize R package and application download counts

```{r plot_examples, fig.height = 6, fig.width = 6, echo = FALSE, fig.cap = " ", echo = FALSE, eval = FALSE}
plot(cranDownloads("cholera", "last-month"))
plot(packageRank("cholera"))
plot(packageLog("cholera"))
plot(cranDistribution("cholera"))
```

```{r plot_examples_code, fig.height = 6, fig.width = 6, fig.cap = " ", echo = FALSE}
par(mfrow = c(2, 2))
plot(cranDownloads("cholera", from = "2025-12-04", to = "2026-01-01"), package.version = TRUE)
plot(packageRank("cholera", "2026-01-01"))
plot(packageLog("cholera", "2026-01-01"))
plot(cranDistribution("cholera", "2026-01-01"))
par(mfrow = c(1, 1))
```

* Functions and Visualization
    * [`cranDownloads()`]
        + enhanced, user-friendly version of cranlogs::cran_downloads()
            + "spell check" or validate package names
            + two additional date formats: yyyy-mm and yyyy
            + shortcuts for `from =` and `to =` (use one without other)
            + check date validity
            + cumulative download counts (growth curves)
            + `pro.mode = TRUE` shortcut to cranlogs::cran_downloads()
            + nominal and cumulative count
        + [`plot(cranDownloads())`]
            + single date -> 1D dotchart (cross-sectional)
            + multiple dates -> 2D time series (longitudinal)
            + logarithm of counts, smoothers, 'ggplot2' confidence bands
            + annotations: package version, R version, ChatGPT release, weekends
            + unit of observation: "day" (default), "week", "month", or "year"
    * [`packageRank()`]
        + nominal counts v. rank and percentile rank
        + percentile rank: nonparametric measure of location in distribution
            + percent of packages with fewer downloads
        + [`plot(packageRank())`]
            + rank v. base10 logarithm of count 
            + annotated with nominal count and percentile rank
    * [`packageLog()`]
        + package download logs via Posit/RStudio CDN server
        * [`plot(packageLog())`]
            + time series (longitudinal) plot of log (24-hour period)
                + 1D plot of observed times with >= 1 download
                + 2D plot of observed times v. download count
                + time units: "second" (default), "minute", or "hour"
    * [`cranDistribution()`] 
        + summary of download distribution for all of CRAN via `package = NULL`
        + [`plot(cranDistribution())`]
            + histogram of base10 logarithm of count v. frequency
            + annotate histogram with specific package via `package` argument
    * [`packageHistory()`]
        + version release history
    * [reverse lookup]  
        + `queryCount()`, `queryPackage()`, `queryPercentile()`, and `queryRank()`
* [Inflation and Filters]
    * download counts are positively biased (at least traffic through Posit CDN)
        + software artifacts: downloads that are too small
        + behavioral artifacts: too many prior versions or effort to downloads all CRAN packages
    * available filters:
        + IP: campaigns from IP addresses with too many downloads
        + small: observations <= 1000 bytes 
        + size: observed downloads < actual package size
        + version: include only current version
* [Results Availability]
    * `cranDownloads()` (and 'cranlogs' package)
        + yesterday's results are usually posted today by 18:00 UTC
    * R and R package logs (and functions _other_ than `cranDownloads()`)
        + yesterday's results are usually posted today by 17:00 UTC 
    * `logInfo()` checks availability and status of logs and ‘cranlogs’
* [Data Fixes and Notes]
    1. jumbled logs at end of 2012 and first day of 2013 -- FIXED
        + three duplicates: Oct 07, Oct 08, and Oct 11
        + one mis-labeled: Oct 12 (actual) is found on Oct 14 (nominal)
        + 77 days (Oct 13 - Dec 28) offset by +3 days: e.g., Nov 28 is in Dec 01
        + 2 net effects: 
            + double download counts: Oct 06-08, Oct 11, Dec 27-28, and Jan 01
            + triple download counts on Dec 26
    2. R Windows download spikes (Nov 2022 - March 2023) -- NOTE
        + Sundays and Wednesdays from 2022-11-06 through 2023-03-19
    3. doubled or tripled R application downloads counts (2023) -- FIXED
        + Sep 13 - Oct 02
    4. seven lost logs (2025) -- FIXED
        + Aug 25-26; Aug 29 - Sep 02
        + message in the console 
        + graphical annotation in plots
        + smoothers ignore missing dates
* [et cetera]
    * Bioconductor: bioconductorDownloads() and bioconductorRank().
    * country code top-level domains in CRAN/package logs
    * use of memoization
    * internet connection time out problem

### getting started

To install ['packageRank'](https://cran.r-project.org/package=packageRank) from [CRAN](https://cran.r-project.org/):

```{r cran_install, eval = FALSE}
install.packages("packageRank")
```

To install the development version (GitHub):

```{r gh_install, eval = FALSE}
# You may need to install.packages("remotes").
remotes::install_github("lindbrook/packageRank", build_vignettes = TRUE)
```

------

### `cranDownloads()`

`cranDownloads()` essentially uses all the same arguments as `cranlogs::cran_downloads()`:

```{r cran_downloads, eval = FALSE}
cranlogs::cran_downloads(packages = "HistData")
```
```{r cran_downloads_code, echo = FALSE}
cranlogs::cran_downloads(packages = "HistData", from = "2020-05-01", to = "2020-05-01")
```

Other than the use of the singular for the 'package' argument, he difference is that `cranDownloads()` adds five features:

#### i) "spell check" for package names

```{r spell_check_fail, eval = FALSE}
cranDownloads(package = "GGplot2")
```
```
## Error in cranDownloads(package = "GGplot2") :
##   GGplot2: misspelled or not on CRAN.
```

<br/>
```{r spell_check_pass, eval = FALSE}
cranDownloads(package = "ggplot2")
```
```{r spell_check_pass_code, echo = FALSE}
cranDownloads(package = "ggplot2", from = "2020-05-01", to = "2020-05-01")
```

<br/>
Note that this also works for inactive or "retired" packages in the [Archive](https://CRAN.R-project.org/src/contrib/Archive/):

#### ii) additional date formats

With `cranlogs::cran_downloads()`, you specify a time frame using the `from` and `to` arguments. The downside is that you need to specify dates as "yyyy-mm-dd". For convenience's sake, `cranDownloads()` allows you to use "yyyy-mm" or yyyy ("yyyy" also works).

##### "yyyy-mm"


With `cranlogs::cran_downloads()`, if you want the download counts for ['HistData'](https://CRAN.R-project.org/package=HistData) for February 2020 you'd have to type out the whole date and remember that 2020 was a leap year:

```{r yyyy-mm_1, eval = FALSE}
cranlogs::cran_downloads(package = "HistData", from = "2020-02-01",
  to = "2020-02-29")
```

<br/>
With `cranDownloads()`, you can just specify the year and month:

```{r yyyy-mm_2, eval = FALSE}
cranDownloads(package = "HistData", from = "2020-02", to = "2020-02")
```

##### yyyy or "yyyy"

With `cranlogs::cran_downloads()`, if you want the download counts for ['rstan'](https://CRAN.R-project.org/package=rstan) for 2020 you'd type something like:

```{r yyyy_1, eval = FALSE}
cranlogs::cran_downloads(packages = "rstan", from = "2022-01-01",
  to = "2022-12-31")
```

<br/>
With `cranDownloads()`, you can use:

```{r yyyy_2, eval = FALSE}
cranDownloads(package = "rstan", from = 2020, to = 2020)
```

Note that "2020" will also work.


#### iii) shortcuts with `from = ` and `to = ` in `cranDownloads()`

These additional date formats also provide convenient shortcuts. Let's say you want the year-to-date download counts for ['rstan'](https://CRAN.R-project.org/package=rstan). With `cranlogs::cran_downloads()`, you'd type something like:

```{r yyyy_4, eval = FALSE}
cranlogs::cran_downloads(package = "rstan", from = "2023-01-01",
  to = Sys.Date() - 1)
```

<br/>
With `cranDownloads()`, you can just pass the current year to `from` argument:

```{r yyyy_5, eval = FALSE}
cranDownloads(package = "rstan", from = 2023)
```

If you wanted the entire download history, pass the current year to the `to` argument:

```{r yyyy_6, eval = FALSE}
cranDownloads(package = "rstan", to = 2026)
```

Note that the Posit/RStudio logs begin on 01 October 2012.

#### iv) check date validity

```{r check_date, eval = FALSE}
cranDownloads(package = "HistData", from = "2019-01-15", to = "2019-01-35")
```
```
## Error in resolveDate(to, type = "to") : Not a valid date.
```

#### v) cumulative count

By default, `cranDownloads()` also computes the cumulative download count. This is useful for plotting growth curves.

```{r cranDownloads_cumulative, eval = FALSE}
cranDownloads(package = "HistData", when = "last-week")
```

```{r cranDownloads_cumulative_code, echo = FALSE}
cranDownloads(package = "HistData", from = "2020-05-01", to = "2020-05-07")
```

#### pro.mode

Some of these features come at a cost: a one-time, per session download of additional data. While those data are cached via the 'memoise' package, this adds time the first time `cranDownloads()` is run. 

For faster results, you can bypass those features by setting `pro.mode = TRUE`. The downside is that you might see odd results like zero downloads for packages on dates before they were on CRAN or zero downloads for mis-spelled/non-existent packages. You'll also won't be able to use the `to` argument by itself. 

For example, 'packageRank' was first published on CRAN on 2019-05-16 - you can verify this via `packageHistory("packageRank")`. But if you use `cranlogs::cran_downloads()` or `cranDownloads(pro.mode = TRUE)` before that date, you'll see zero downloads for days before 2019-05-16:

```{r pro_mode_ex_ante}
cranDownloads("packageRank", from = "2019-05-10", to = "2019-05-16", pro.mode = TRUE)
```

This is particularly noticeable if you mis-spell or pass a "newer" package to cranDownloads().

```{r pro_mode_non_existent}
cranDownloads("vr", from = "2019-05-10", to = "2019-05-16", pro.mode = TRUE)
```

Finally, if you just use `to` without a value for `from`, you'll get an error:

```{r pro_mode_to, eval = FALSE}
cranDownloads(to = 2024, pro.mode = TRUE)
```
```
Error: You must also provide a date for "from".
```

<br/>

------

### `plot(cranDownloads())`

'packageRank' uses R's generic plot method:

```{r cranDownloads_viz1}
plot(cranDownloads(package = "HistData", from = "2019", to = "2019"))
```

If you pass a vector of package names for a single day, you'll get a dotchart:

```{r cranDownloads_viz2a}
plot(cranDownloads(package = c("ggplot2", "data.table", "Rcpp"),
  from = "2020-03-01", to = "2020-03-01"))
```

If you pass a vector package names for multiple days, you'll get a single graph with multiple time series plots using ['ggplot2'](https://CRAN.R-project.org/package=ggplot2) facets:

```{r cranDownloads_viz2}
plot(cranDownloads(package = c("ggplot2", "data.table", "Rcpp"),
  from = "2020", to = "2020-03-20"))
```
<br/>

To plot these data in a single plot frame, set `multi.plot = TRUE`:

```{r cranDownloads_viz3}
plot(cranDownloads(package = c("ggplot2", "data.table", "Rcpp"),
  from = "2020", to = "2020-03-20"), multi.plot = TRUE)
```

<br/>
To plot these data as separate plots, on the same scale, set `graphics = "base"`. You'll be prompted for each plot:

```{r cranDownloads_viz4, eval = FALSE}
# Code only. Graph not shown.
plot(cranDownloads(package = c("ggplot2", "data.table", "Rcpp"), from = "2020", 
  to = "2020-03-20"), graphics = "base")
```

To do the above using separate, independent scales, set `same.xy = FALSE`:

```{r cranDownloads_viz5, eval = FALSE}
# Code only. Graph not shown.
plot(cranDownloads(package = c("ggplot2", "data.table", "Rcpp"), from = "2020", 
  to = "2020-03-20"), graphics = "base", same.xy = FALSE)
```

#### `log.y = TRUE`

To use the base 10 logarithm of the download count in a plot, set `log.y = TRUE`:

```{r log_count}
plot(cranDownloads(package = "HistData", from = "2019", to = "2019"),
  log.y = TRUE)
```

Note that any zero counts will be replaced by ones so that the logarithm can be computed (This does not affect the data returned by `cranDownloads()`).

#### `package = NULL`


The default first argument of `cranDownloads()` is `package = NULL`. This computes the total number of package downloads from CRAN. To plot these data, use:

```{r null_packages}
plot(cranDownloads(from = 2019, to = 2019))
```

#### `package = "R"`

`cranDownloads(package = "R")` computes the total number of downloads of the R application by platfrom: "mac" = macOS, "src" = source, and "win" = Windows. Note that, as with `cranlogs::cran_downloads()`, you can only use "R" _or_ a vector of packages names, not both!. 

To plot these data:

```{r r_downloads}
plot(cranDownloads(package = "R", from = 2019, to = 2019))
```

If you want plot the total count of R downloads, set `r.total = TRUE`:

```{r r_total, eval = FALSE}
plot(cranDownloads(package = "R", from = 2019, to = 2019), r.total = TRUE)
```

#### `smooth = TRUE`

To add a smoother, use `smooth = TRUE`:

```{r lowess}
plot(cranDownloads(package = "rstan", from = "2019", to = "2019"),
  smooth = TRUE)
```

Note that loess is the default smoother, but with base graphics, lowess is used when there are 7 or fewer observations. To control the degree of smoothness, use the `span` argument (the default is span = 0.75) for loess and where applicable, use the `f` argument (the default is f = 2/3):

```{r span, eval = FALSE}
plot(cranDownloads(package = c("HistData", "rnaturalearth", "Zelig"),
  from = "2020", to = "2020-03-20"), smooth = TRUE, span = 0.75)

plot(cranDownloads(package = c("HistData", "rnaturalearth", "Zelig"),
  from = "2020", to = "2020-03-20"), smooth = TRUE, graphics = "ggplot2", 
  span = 0.33)
```

#### `se = TRUE`

With graphs that use 'ggplot2', `se = TRUE` will add confidence bands:

```{r ci}
plot(cranDownloads(package = c("HistData", "rnaturalearth", "Zelig"),
  from = "2020", to = "2020-03-20"), smooth = TRUE, se = TRUE)
```

#### `package.version = TRUE` or `package.version = "line"`

To annotate a graph with a package's release dates as ticks on the top axis, set `package.version = TRUE`:

```{r pkg_release_date}
plot(cranDownloads(package = "rstan", from = "2019", to = "2019"),
  package.version = TRUE, unit.observation = "week")
```

If you want a vertical line, set `package.version = "line"`

```{r vertical_date_line}
plot(cranDownloads(package = "rstan", from = "2019", to = "2019"),
  package.version = "line", unit.observation = "week")
```

#### `r.version = TRUE` or r.version = "line"

To annotate a graph with R release dates:

```{r r_release_date}
plot(cranDownloads(package = "rstan", from = "2019", to = "2019"),
  r.version = TRUE, unit.observation = "week")
```

If you want a vertical line, set `package.version = "line"`

#### `chatgpt = TRUE` or `chatgpt = "line"`

By default, graphs that include ChatGPT's release date, 2022-11-30, will be annotated with an axis tick and a vertical line:

```{r chatgpt_release_date}
plot(cranDownloads(package = "R", from = "2020-12", to = "2025-01"),
  r.total = TRUE, unit.observation = "week")
```

To exclude this, set plot.cranDownloads(chatgpt = FALSE) 

#### `weekend = TRUE`

With `unit.observation = "day"` and `graphics = "base"`, you can highlight weekends, as empty circles, by setting `weekend = TRUE`:

```{r r_weekend}
plot(cranDownloads(package = "rstan", from = "2024-06", to = "2024-06"), 
  weekend = TRUE)
```

#### `statistic = "cumulative"`

To plot growth curves using cumulative counts, set `statistic = "cumulative"`:

```{r cranDownloads_growth_curves}
plot(cranDownloads(package = c("ggplot2", "data.table", "Rcpp"), from = "2020", 
 to = "2020-03-20"), statistic = "cumulative", multi.plot = TRUE,
 points = FALSE)
```

#### unit of observation: "day", "week", "month", "year"

The default unit of observation for `cranDownloads()` is the day. The graph below plots the daily downloads for [‘cranlogs'](https://CRAN.R-project.org/package=cranlogs) from 01 January 2022 through 27 September 2023.

```{r day}
plot(cranDownloads(package = "cranlogs", from = 2022, to = "2023-09-27"))
```

To view the data from a less granular perspective, change plot.cranDownloads()'s `unit.observation` argument to "week", "month", or "year".


##### `unit.observation = "week"`

The graph below plots the data aggregated by week, which begin on Sunday.

```{r week}
plot(cranDownloads(package = "cranlogs", from = 2022, to = "2023-09-27"), 
  smooth = TRUE, unit.observation = "week")
```

Four things to note. 

First, if the first week (far left) is incomplete (the 'from' date is not a Sunday), that observation will be split in two: one point for the observed total on 'from' date (empty gray square) and another point for the *backdated* total (blue asterisk). The backdated observation simply completes the week by pushing the start date back to include the previous Sunday. 

In the example above, the nominal start date (01 January 2022) is pushed back to include data through the previous Sunday (26 December 2021). This is useful because when using a weekly unit of observation, the first "week" (far left) is often truncated. Consequently, you won't get the most representative picture of the data. Backdating aims to fix this. 

Second, if the last week (far right) is in-progress (the 'to' date is not a Saturday), that observation will be split in two: the observed total (empty gray square) and an estimated total based on the proportion of week completed (empty red circle). 

Third, smoothers only use complete observations. This includes backdated data but excludes in-progress and estimated data. 

Fourth, with the exception of first week's observed count, which is plotted at its nominal date, points on the x-axis are plotted on Sundays.


##### `unit.observation = "month"`

The graph below plots the data aggregated by month.

```{r month}
plot(cranDownloads(package = "cranlogs", from = 2022, to = "2023-09-27"), 
  smooth = TRUE, unit.observation = "month")
```

Three things to note. 

First, if the last/current month (far right) is still in-progress (it's not yet the end of the month), that observation will be split in two: one point for the in-progress total (empty black square), another for the estimated total (empty red circle). The estimate is based on the proportion of the month completed. In the example above, the 635 observed downloads from April 1 through April 15 translates into an estimate of 1,270 downloads for the entire month (30 / 15 * 635). 

Second, smoothers only use complete observations, not in-progress or estimated data. 

Third, all points are plotted along the x-axis at the first day of the month.

#### pro.mode

Perhaps the biggest downside of using `cranDownloads(pro.mode = TRUE)` is that you might draw mistaken inferences from plotting the data since it can add false zeroes to the data.

Using the example of 'packageRank', which was published on 2019-05-16:

```{r pro_mode_plot}
plot(cranDownloads("packageRank", from = "2019-05", to = "2019-05", 
  pro.mode = TRUE), smooth = TRUE)
```

```{r non_pro_mode_plot}
plot(cranDownloads("packageRank", from = "2019-05", to = "2019-05", 
  pro.mode = FALSE), smooth = TRUE)
```

------

### `packageRank()`

After spending some time with the _nominal_ download counts above, the "compared to what?" question will come to mind. For instance, consider the data for the 'cholera' package from the first week of March 2020:

```{r motivation, eval = FALSE}
plot(cranDownloads(package = "cholera", from = "2020-03-01", to = "2020-03-07"))
```

```{r motivation_code, echo = FALSE}
par(mar = c(5, 4, 4, 4))
plot(cranDownloads(package = "cholera", from = "2020-03-01", to = "2020-03-07"))
par(mar = c(5, 4, 4, 2))
```

Do Wednesday and Saturday reflect surges of interest in the package or surges of traffic to [CRAN](https://CRAN.R-project.org/)? To put it differently, how can we know if a given download count is typical or unusual? 

To answer these questions, we can start by looking at the total number of package downloads:

```{r motivation_cran}
plot(cranDownloads(from = "2020-03-01", to = "2020-03-07"))
```

Here we see that there's a big difference between the work week and the weekend. This seems to indicate that the download activity for ['cholera'](https://CRAN.R-project.org/package=cholera) on the weekend seems high. Moreover, the Wednesday peak for ['cholera'](https://CRAN.R-project.org/package=cholera) downloads seems higher than the mid-week peak of total downloads.

One way to better address these observations is to locate your package's download counts in the overall frequency distribution of download counts. 'cholera' allows you to do so via `cranDistribution()`. Below are the distributions of logarithm of download counts for Wednesday and Saturday. Each vertical segment (along the x-axis) represents a download count. The height of a segment represents that download count's frequency. The location of ['cholera'](https://CRAN.R-project.org/package=cholera) in the distribution is highlighted in red.

```{r cranDistribution_ex, echo = FALSE}
plot_package_distribution <- function(dat, xlim, ylim) {
  freq.dist <- dat$freq.dist
  freqtab <- dat$freqtab
  plot(freq.dist$count, freq.dist$frequency, type = "h", log = "x",
    xlab = "Count", ylab = "Frequency", xlim = xlim, ylim = ylim)
  axis(3, at = freqtab[1], cex.axis = 0.8, padj = 0.9, col.axis = "dodgerblue",
    col.ticks = "dodgerblue", labels = paste(names(freqtab[1]), "=",
    format(freqtab[1], big.mark = ",")))
  abline(v = freqtab[1], col = "dodgerblue", lty = "dotted")

  if (!is.null(dat$package)) {
    pkg.ct <- freqtab[names(freqtab) == dat$package]
    pkg.bin <- freqtab[freqtab == pkg.ct]
    axis(3, at = pkg.ct, labels = format(pkg.ct, big.mark = ","),
      cex.axis = 0.8, padj = 0.9, col.axis = "red", col.ticks = "red")
    abline(v = pkg.ct, col = grDevices::adjustcolor("red", alpha.f = 0.5))
    day <- weekdays(as.Date(dat$date), abbreviate = TRUE)
    title(paste0(dat$package, " @ ", dat$date, " (", day, ")"))
  } else title(paste("Distribution of Package Download Counts:", dat$date))
}

distn.data <- lapply(c("2020-03-04", "2020-03-07"), function(date) cranDistribution(package = "cholera", date = date))

xlim <- range(lapply(distn.data, function(x) x$freq.dist$count))
ylim <- range(lapply(distn.data, function(x) x$freq.dist$frequency))
```

```{r cranDistribution_ex_wed, eval = FALSE}
plot(cranDistribution(package = "cholera", date = "2020-03-04"))
```
```{r cranDistribution_ex_wed_code, echo = FALSE}
plot_package_distribution(distn.data[[1]], xlim, ylim)
```

```{r cranDistribution_ex_sat, eval = FALSE}
plot(cranDistribution(package = "cholera", date = "2020-03-07"))
```
```{r cranDistribution_ex_sat_code, echo = FALSE}
plot_package_distribution(distn.data[[2]], xlim, ylim)
```

While these plots give us a better picture of where ['cholera'](https://CRAN.R-project.org/package=cholera) is located, comparisons between Wednesday and Saturday are still impressionistic: all we can confidently say is that the download counts for both days were greater than the mode.

To facilitate interpretation and comparison, I use the _percentile rank_ of a download count instead of the simple nominal download count. This nonparametric statistic tells you the percentage of packages that had fewer downloads. In other words, it gives you the location of your package relative to the locations of all other packages. More importantly, by rescaling download counts to lie on the bounded interval between 0 and 100, percentile ranks make it easier to compare packages within and across distributions.

This function returns a package's nominal count, rank, and percentile rank for a given day (default is "today" or last available).

For example, we can compare Wednesday ("2020-03-04") to Saturday ("2020-03-07"):

```{r packageRank1}
packageRank(package = "cholera", date = "2020-03-04")
```

On Wednesday, we can see that ['cholera'](https://CRAN.R-project.org/package=cholera) had 38 downloads, came in 5,788th place out of the 18,038 different packages downloaded, and earned a spot in the 68th percentile.

```{r packageRank2}
packageRank(package = "cholera", date = "2020-03-07")
```

On Saturday, we can see that ['cholera'](https://CRAN.R-project.org/package=cholera) had 29 downloads, came in 3,189st place out of the 15,950 different packages downloaded, and earned a spot in the 80th percentile.

So contrary to what the nominal counts tell us, one could say that the interest in ['cholera'](https://CRAN.R-project.org/package=cholera) was actually greater on Saturday than on Wednesday.

#### computing percentile rank

To compute percentile ranks, I do the following. For each package, I tabulate the number of downloads and then compute the percentage of packages with fewer downloads. Here are the details using ['cholera'](https://CRAN.R-project.org/package=cholera) from Wednesday as an example:

```{r percentile}
pkg.rank <- packageRank(package = "cholera", date = "2020-03-04")

downloads <- pkg.rank$cran.data$count
names(downloads) <- pkg.rank$cran.data$package

round(100 * mean(downloads < downloads["cholera"]), 1)
```

To put it differently:

```{r percentile2}
(pkgs.with.fewer.downloads <- sum(downloads < downloads["cholera"]))

(tot.pkgs <- length(downloads))

round(100 * pkgs.with.fewer.downloads / tot.pkgs, 1)
```

#### competition v. nominal ranks

Note that, by default, `packageRank()` computes the [competition rank](https://en.wikipedia.org/wiki/Ranking#Standard_competition_ranking_(%221224%22_ranking)) (i.e., "1224"). [Nominal or ordinal ranking](https://en.wikipedia.org/wiki/Ranking#Ordinal_ranking_(%221234%22_ranking)) (i.e., "1234" ranking) is available by setting `packageRank(rank.ties = FALSE)`.

### `plot(packageRank())`

To visualize the results for `packageRank()` for Wednesday and Sunday, use `plot()`.

```{r packageRank_plot_wed, eval = FALSE}
plot(packageRank(packages = "cholera", date = "2020-03-04"))
```

```{r packageRank_data, echo = FALSE}
dat <- lapply(c("2020-03-04", "2020-03-07"), function(x) {
  packageRank("cholera", date = x)
})

freqtab1 <- dat[[1]]$cran.data$count
freqtab2 <- dat[[2]]$cran.data$count
xlim <- range(seq_along(freqtab1), seq_along(freqtab2))
ylim <- range(c(freqtab1), c(freqtab2))
```

```{r packageRank_plot_code_wed, echo = FALSE}

freqtab <- dat[[1]]$cran.data$count
names(freqtab) <- dat[[1]]$cran.data$package
package.data <- dat[[1]]$package.data
pkg <- dat[[1]]$package
date <- dat[[1]]$date
y.max <- freqtab[1]
q <- stats::quantile(freqtab)[2:4]

iqr <- vapply(c("75%", "50%", "25%"), function(id) {
  dat <- which(freqtab > q[[id]])
  dat[length(dat)]
}, numeric(1L))

plot(seq_along(freqtab), c(freqtab), type = "l", xlab = "Rank",
  ylab = "log10(Count)", log = "y", xlim = xlim, ylim = ylim)
abline(v = iqr, col = "black", lty = "dotted")
iqr.labels <- c("75th", "50th", "25th")
invisible(lapply(seq_along(iqr), function(i) {
  text(iqr[[i]], y.max / 2, labels = iqr.labels[i], cex = 0.75)
}))
abline(v = which(names(freqtab) == pkg), col = "red")
abline(h = freqtab[pkg], col = "red")
pct <- package.data[package.data$package == pkg, "percentile"]
pct.label <- paste0(round(pct, 2), "%")
axis(3, at = which(names(freqtab) == pkg), padj = 0.9, col.axis = "red",
  col.ticks = "red", labels = pct.label, cex.axis = 0.8)
axis(4, at = freqtab[pkg], col.axis = "red", col.ticks = "red",
  cex.axis = 0.8, labels = format(freqtab[pkg], big.mark = ","))
points(which(names(freqtab) == pkg), freqtab[pkg], col = "red")
points(which(names(freqtab) == names(freqtab[1])), y.max,
  col = "dodgerblue")
text(which(names(freqtab) == names(freqtab[1])), y.max, pos = 4,
  labels = paste(names(freqtab[1]), "=", format(freqtab[1],
  big.mark = ",")), cex = 0.8, col = "dodgerblue")
text(max(xlim), max(ylim),
  labels = paste("Tot = ", format(sum(freqtab), big.mark = ",")), cex = 0.8,
  col = "dodgerblue", pos = 2)
day <- weekdays(as.Date(date), abbreviate = TRUE)
title(main = paste0(pkg, " @ ", date, " (", day, ")"))
```

<br/>

```{r packageRank_plot_sat, eval = FALSE}
plot(packageRank(packages = "cholera", date = "2020-03-07"))
```

```{r packageRank_plot_code_sat, echo = FALSE}
freqtab <- dat[[2]]$cran.data$count
names(freqtab) <- dat[[2]]$cran.data$package
package.data <- dat[[2]]$package.data
pkg <- dat[[2]]$package
date <- dat[[2]]$date
y.max <- freqtab[1]
q <- stats::quantile(freqtab)[2:4]

iqr <- vapply(c("75%", "50%", "25%"), function(id) {
  dat <- which(freqtab > q[[id]])
  dat[length(dat)]
}, numeric(1L))

plot(seq_along(freqtab), c(freqtab), type = "l", xlab = "Rank",
  ylab = "log10(Count)", log = "y", xlim = xlim, ylim = ylim)
abline(v = iqr, col = "black", lty = "dotted")
iqr.labels <- c("75th", "50th", "25th")
invisible(lapply(seq_along(iqr), function(i) {
  text(iqr[[i]], y.max / 2, labels = iqr.labels[i], cex = 0.75)
}))
abline(v = which(names(freqtab) == pkg), col = "red")
abline(h = freqtab[pkg], col = "red")
pct <- package.data[package.data$package == pkg, "percentile"]
pct.label <- paste0(round(pct, 2), "%")
axis(3, at = which(names(freqtab) == pkg), padj = 0.9, col.axis = "red",
  col.ticks = "red", labels = pct.label, cex.axis = 0.8)
axis(4, at = freqtab[pkg], col.axis = "red", col.ticks = "red",
  cex.axis = 0.8, labels = format(freqtab[pkg], big.mark = ","))
points(which(names(freqtab) == pkg), freqtab[pkg], col = "red")
points(which(names(freqtab) == names(freqtab[1])), y.max,
  col = "dodgerblue")
text(which(names(freqtab) == names(freqtab[1])), y.max, pos = 4,
  labels = paste(names(freqtab[1]), "=", format(freqtab[1],
  big.mark = ",")), cex = 0.8, col = "dodgerblue")
text(max(xlim), max(ylim),
  labels = paste("Tot = ", format(sum(freqtab), big.mark = ",")), cex = 0.8,
  col = "dodgerblue", pos = 2)
day <- weekdays(as.Date(date), abbreviate = TRUE)
title(main = paste0(pkg, " @ ", date, " (", day, ")"))
```

These graphs above, which are customized here to be on the same scale, plot the _rank order_ of packages' download counts (x-axis) against the logarithm of those counts (y-axis). It then highlights (in red) a package's position in the distribution along with its percentile rank and download count. In the background, the 75th, 50th and 25th percentiles are plotted as dotted vertical lines. The package with the most downloads, ['magrittr'](https://CRAN.R-project.org/package=magrittr) in both cases, is at top left (in blue). The total number of downloads is at the top right (in blue).

------

### `packageLog()`

This function returns the download log(s) for selected package(s) for a given day (the default is "today" or last available).

```{r packageLog, eval = FALSE}
packageLog(package = "packageRank")
```

```{r packageLog_code, echo = FALSE}
packageLog(package = "packageRank", date = "2026-01-01")
```

The logs record the "date", "time", "size" (in bytes), "r_version" (R version), "r_arch" (computer architecture: x86_64 = Intel, aarch64 = Apple Silicon, etc.), "r_os" (operating system: linux-gnu, darwin20, mingw32, etc.), "package", "version" (package version), "country" (top level country code domain) and "ip_id" (anonymized IP address).

If you see information for "r_version", "r_arch", "r_os", the client is the RStudio IDE. If those fields are NA, the client is something else (including Positron apparently). 

### `plot(packageLog())`

Plotting the logs will give you time series plots. If you set `type = "1D" `(1-dimensional), which is the default, you'll get a horizontal dot plot that shows the times when at least one download is observed. If you set `type = "2D"` (2-dimensional) you'll get a time series graph that plots time versus count. There are three time units: "second" (default), "minute", and "hour". 

The plot below contrasts the data with time unit "second" to the same data aggregated by time unit "hour":

```{r log_unit_obs, echo = TRUE, eval = FALSE, fig.height = 6, fig.width = 6}
plot(packageLog(package = "cranlogs", date = "2026-01-01"), type = "1D", 
  unit.observation = "second")
plot(packageLog(package = "cranlogs", date = "2026-01-01"), type = "1D", 
  unit.observation = "hour")
```

```{r log_unit_obs_code, echo = FALSE, eval = TRUE, fig.height = 8, fig.width = 6}
par(mfrow = c(2, 1))
plot(packageLog(package = "cranlogs", date = "2026-01-01"), type = "1D",
  unit.observation = "second")
plot(packageLog(package = "cranlogs", date = "2026-01-01"), type = "1D",
  unit.observation = "hour")
par(mfrow = c(1, 1))
```

By default, your local time is appended to the top side of the graph (side = 3). You can override this by either setting `local.timezone = FALSE` to or by using a time zone from OlsonNames(), e.g., `local.timezone = "Australia/Sydney"`:

```{r time_type, echo = TRUE, eval = FALSE, fig.height = 6, fig.width = 6}
plot(packageLog(package = "HistData", date = "2026-01-01"), type = "2D", 
  unit.observation = "hour")
plot(packageLog(package = "HistData", date = "2026-01-01"), type = "2D", 
  unit.observation = "hour", local.timezone = "Australia/Sydney")
```

```{r time_type_code, echo = FALSE, eval = TRUE, fig.height = 8, fig.width = 6}
par(mfrow = c(2, 1))
plot(packageLog(package = "HistData", date = "2026-01-01"), type = "2D",
  unit.observation = "hour")
plot(packageLog(package = "HistData", date = "2026-01-01"), type = "2D",
  unit.observation = "hour", local.timezone = "Australia/Sydney")
par(mfrow = c(1, 1))
```

------

### `packageHistory()`

This function returns a package's release history.

```{r, eval = TRUE}
packageHistory(package = "cholera")
```

------

### `cranDistribution()`

This function computes the frequency distribution of downloads and returns summary statistics and the top-N packages.

```{r cranDistribution, echo = TRUE, eval = FALSE}
cranDistribution(package = NULL)
```

```{r cranDistribution_code, echo = FALSE, eval = TRUE}
cranDistribution(package = NULL, date = "2026-01-01")
```

If you pass a package to the function, e.g., `cranDistribution(package = "packageRank")`, data for that package will be appended: 


```{r cranDistribution_pkg, echo = TRUE, eval = FALSE}
cranDistribution(package = "packageRank")
```

```{r cranDistribution_pkg_code, echo = FALSE, eval = TRUE}
cranDistribution(package = "packageRank", date = "2026-01-01")
```

### `plot(cranDistribution())`

If you plot the above functions, you'll get a histogram of the overall distribution of download counts (base 10 logarithm):

```{r plot_cranDistribution, echo = TRUE, eval = FALSE}
plot(cranDistribution(package = NULL))
```

```{r plot_cranDistribution_code, echo = FALSE, eval = TRUE}
plot(cranDistribution(package = NULL, date = "2026-01-01"))
```

If you pass a package to the function, its location in the distribution will be annotated:

```{r pkg_plot_cranDistribution, echo = TRUE, eval = FALSE}
plot(cranDistribution(package = "packageRank"))
```
```{r pkg_plot_cranDistribution_code, echo = FALSE}
plot(cranDistribution(package = "packageRank", date = "2026-01-01"))
```

------

### reverse lookup

To do a reverse lookup (e.g., find packages with a given download count or percentile rank), use `queryCount()`, `queryPackage()`, `queryPercentile()` or `queryRank()`.
 

```{r q_count, eval = FALSE}
queryCount(count = 100)
```
```{r q_count_code, echo = FALSE}
queryCount(count = 100, date = "2026-01-01")
```

```{r q_pkg, eval = FALSE}
queryPackage(package = "cholera")
```
```{r q_pkg_code, echo = FALSE}
queryPackage(package = "cholera", date = "2026-01-01")
```

```{r q_pct, eval = FALSE}
head(queryPercentile(percentile = 99))
```
```{r q_pct_code, echo = FALSE}
head(queryPercentile(percentile = 99, date = "2026-01-01"))
```

Note that due to the discrete nature of counts, your choice of percentile may not be available. For details, see this [note](https://github.com/lindbrook/packageRank/blob/master/docs/queryPercentile_note.md) in the package's GitHub repository.


```{r q_rank, eval = FALSE}
queryRank(rank = 9)
```
```{r q_rank_code, echo = FALSE}
queryRank(rank = 9, date = "2026-01-01")
```
------

### Inflation and Filters

['cranlogs'](https://CRAN.R-project.org/package=cranlogs) computes package downloads by counting log entries. While straightforward, this approach can run into problems. Putting aside package dependencies (the effect of packages that use packages), what I have in mind here are two types of "invalid" log entries. 

The first are software artifacts. These are entries that are smaller, often orders of magnitude smaller, than the package's binary or source file. The second are behavioral artifact. The main culprit here appears to stem from efforts to download all of the packages on [CRAN](https://cran.r-project.org/). In both cases, simple nominal counts will give you an inflated sense of the degree of interest in your package. 

For what it's worth, an early examination of inflation is included as part of this [R-hub blog post](https://blog.r-hub.io/2020/05/11/packagerank-intro/#inflationary-bias-of-download-counts).

#### software artifacts

When looking at package's download logs, the first thing you'll often see are wrongly sized log entries. They come in two flavors: 1) "small" entries approximately 500 bytes in size and 2) "medium" entries (i.e., "small" <= "medium" <= full download). "Small" entries manifest themselves as standalone entries, as paired with a full download, or as part of a triplet with a "medium" and a full download. "Medium" entries manifest themselves as either a standalone entry or as part of a triplet.

The example below illustrates a triplet:

```{r triplet}
packageLog(date = "2020-07-01")$cholera[4:6, -(4:6)]
```

The "medium" entry is the first observation (99,622 bytes). The full download is the second entry (4,161,948 bytes). The "small" entry is the last observation (536 bytes). At a minimum, what makes a triplet (or a pair) is that all members share system configuration (e.g. IP address, etc.) and have identical or adjacent time stamps.

To deal with "small" log entries, I filter out observations smaller than 1,000 bytes (the smallest package on [CRAN](https://cran.r-project.org/) appears to be ['LifeInsuranceContracts'](https://cran.r-project.org/package=LifeInsuranceContracts), whose source file weighs in at 1,100 bytes). "Medium" entries are harder to handle. I remove them using a function that looks up a package's actual size.

#### behavioral artifacts

The other pattern you'll often see when looking at package download logs is the presence of "too many" prior versions. While there are legitimate reasons for downloading past versions (e.g., research, container-based software distribution, etc.), I'd argue that such patterns are "fingerprints" of efforts to download [CRAN](https://cran.r-project.org/). While there is nothing inherently problematic about this (other than infrastructure costs), it does inflate your package download count. When your package is downloaded as part of such efforts, those downloads are more a reflection of an interest in [CRAN](https://cran.r-project.org/) itself (a collection of packages) rather than of an interest in your package _per se_.  

To illustrate, consider the following example:

```{r, sequence_ex, eval = FALSE}
packageLog(package = "cholera", date = "2020-07-31")[8:14, -(4:6)]
```
```
>              date     time    size package version country ip_id
> 132509 2020-07-31 21:03:06 3797776 cholera   0.2.1      US    14
> 132106 2020-07-31 21:03:07 4285678 cholera   0.4.0      US    14
> 132347 2020-07-31 21:03:07 4109051 cholera   0.3.0      US    14
> 133198 2020-07-31 21:03:08 3766514 cholera   0.5.0      US    14
> 132630 2020-07-31 21:03:09 3764848 cholera   0.5.1      US    14
> 133078 2020-07-31 21:03:11 4275831 cholera   0.6.0      US    14
> 132644 2020-07-31 21:03:12 4284609 cholera   0.6.5      US    14
```

Here, we see that seven different versions of the package were downloaded as a sequential bloc. A little digging shows that on that date there were seven extant versions of 'cholera':

```{r, cholera_history, eval = FALSE}
packageHistory(package = "cholera")
```
```
>   Package Version       Date Repository
> 1 cholera   0.2.1 2017-08-10    Archive
> 2 cholera   0.3.0 2018-01-26    Archive
> 3 cholera   0.4.0 2018-04-01    Archive
> 4 cholera   0.5.0 2018-07-16    Archive
> 5 cholera   0.5.1 2018-08-15    Archive
> 6 cholera   0.6.0 2019-03-08    Archive
> 7 cholera   0.6.5 2019-06-11    Archive
> 8 cholera   0.7.0 2019-08-28       CRAN
```

And such, it may be useful to exclude such entries. To do so, I filter out these entries in two ways. The first identify IP addresses that download "too many" packages and then filter out _campaigns_, large blocs of downloads that occur in (nearly) alphabetical order. The second looks for campaigns not associated with "greedy" IP addresses and filters out sequences of past versions downloaded in a narrowly defined time window.

#### example usage

To get an idea of how inflated your package's download count may be, use `filteredDownloads()`. Below are the results for 'ggplot2' for 15 September 2021.

```{r, filteredDownloads}
filteredDownloads(package = "ggplot2", date = "2021-09-15")
```

While there were 113,842 nominal downloads, applying all the filters reduced that number to 111,662, an inflation of 1.95%.

There are 5 filters. You can control them using the following arguments (listed in order of application):

* `ip.filter`: removes campaigns of "greedy" IP addresses.
* `small.filter`: removes entries smaller than 1,000 bytes.
* `sequence.filter`: removes blocs of past versions.
* `size.filter`: removes entries smaller than a package's binary or source file.
* `version.filter`: include only the most recent package version.

For `filteredDownloads()`, they are all on by default. For `packageLog()` and `packageRank()`, they are off by default. To apply them, simply set the argument for the filter you want to TRUE:

```{r, small_filter, eval = FALSE}
packageRank(package = "cholera", small.filter = TRUE)
```

Alternatively, for `packageLog()` and `packageRank()` you can simply set `all.filters = TRUE`.

```{r, all_filters, eval = FALSE}
packageRank(package = "cholera", all.filters = TRUE)
```

Note that the `all.filters = TRUE` is contextual. Depending on the function used, you'll either get the CRAN-specific or the package-specific set of filters. The former sets `ip.filter = TRUE` and `size.filter = TRUE`; it works independently of packages at the level of the entire log. The latter sets sequence.filter = TRUE` and `size.filter TRUE`; it relies on package specific information (e.g., size of source or binary file).

Ideally, we'd like to use both sets. However, the package-specific set is computationally expensive because they need to be applied individually to all packages in the log, which can involve tens of thousands of packages. While not unfeasible, currently this takes a long time. For this reason, when `all.filters = TRUE`, `packageRank()`, `ipPackage()`, `countryPackage()`, `countryDistribution()` and `cranDistribution()` use only CRAN specific filters while `packageLog()`, `packageCountry()`, and `filteredDownloads()` use both [CRAN](https://cran.r-project.org/) and package specific filters.

------

### Results Availability

To understand when results become available, you need to know that [‘packageRank'](https://CRAN.R-project.org/package=packageRank) has two upstream, online dependencies. The first is Posit/RStudio's [CRAN package download logs](http://cran-logs.rstudio.com/). These logs record traffic that passes through the [“0-Cloud”](https://CLOUD.R-project.org/) mirror, which is currently sponsored by Posit. The second is Gábor Csárdi's [‘cranlogs'](https://CRAN.R-project.org/package=cranlogs) R package, which uses the Posit/RStudio logs to compute the download counts of both R packages and the R application itself.

The [CRAN package download logs](http://cran-logs.rstudio.com/) for the _previous_ day are typically posted by 17:00 UTC. The results for [‘cranlogs'](https://CRAN.R-project.org/package=cranlogs) usually become available soon thereafter.

#### Why aren't today's logs and results available?

Occasionally, problems with "today's" data can arise due to problems with one or both of the upstream dependencies (illustrated below).

```
CRAN Download Logs --> 'cranlogs' --> 'packageRank'
```

If there's a problem with the [logs](http://cran-logs.rstudio.com/) (e.g., they're not posted on time), both [‘cranlogs'](https://CRAN.R-project.org/package=cranlogs) and [‘packageRank'](https://CRAN.R-project.org/package=packageRank) will be affected. If this happens, you'll see things like an unexpected zero count(s) for your package(s) (actually, you'll see a zero download count for both your package and for all of [CRAN](https://cran.r-project.org/)), data from "yesterday", or a "Log is not (yet) on the server" error message.

```
'cranlogs' --> packageRank::cranDownloads()
```

If there's a problem with [‘cranlogs'](https://CRAN.R-project.org/package=cranlogs) but not with the [logs](http://cran-logs.rstudio.com/), only `cranDownalods()` will be affected. In that case, you might get a warning that only "previous" results will be used. All other [‘packageRank'](https://CRAN.R-project.org/package=packageRank) functions should work since they either directly access the logs or use some other data source. Usually, these errors resolve themselves the next time the underlying scripts are run (tomorrow, if not sooner). 

#### `logInfo()`

To check the status of the download logs and 'cranlogs', use `logInfo()`. This function checks whether 1) "today's" log is posted on Posit/RStudio's server and 2) "today's" results have been computed by 'cranlogs'. 

```{r logInfo0, eval = FALSE}
logInfo()
```
```
$`Today's log/result`
[1] "2023-02-01"

$`Today's log posted?`
[1] "Yes"

$`Today's results on 'cranlogs'?`
[1] "No"

$status
[1] "Today's log is typically posted by 01 Feb 09:00 PST -- 01 Feb 17:00 UTC."
```

#### time zones

Because you're typically interested in _today's_ log file, another thing that affects availability is your time zone. For example, let's say that it's 09:01 on 01 January 2021 and you want to compute the percentile rank for ['ergm'](https://CRAN.R-project.org/package=ergm) for the last day of 2020. You might be tempted to use the following:

```{r timezone, eval = FALSE}
packageRank(package = "ergm")
```

However, depending on _where_ you make this request, you may not get the data you expect. In Honolulu, USA, you will. In Sydney, Australia you won't. The reason is that you've forgotten a key piece of trivia: Posit/RStudio typically posts _yesterday's_ log around 17:00 UTC the following day.

The expression works in Honolulu because 09:01 HST on 01 January 2021 is 19:01 UTC 01 January 2021. So the log you want has been available for 2 hours. The expression fails in Sydney because 09:01 AEDT on 01 January 2021 is 31 December 2020 22:01 UTC. The log you want won't actually be available for another 19 hours.

To make life a little easier, ['packageRank'](https://CRAN.R-project.org/package=packageRank) does two things. First, when the log for the date you want is not available (due to time zone rather than server issues), you'll just get the last available log. If you specified a date in the future, you'll either get an error message or a warning with an estimate of when the log you want should be available.

Using the Sydney example and the expression above, you'd get the results for 30 December 2020:

```{r sydney, eval = FALSE}
packageRank(package = "ergm")
```
```{r sydney_code, echo = FALSE}
packageRank(package = "ergm", date = "2020-12-30")
```

If you had specified the date, you'd get an additional warning:

```{r sydneyB, eval = FALSE}
packageRank(package = "ergm", date = "2021-01-01")
```
```{r sydney_codeB, echo = FALSE}
packageRank(package = "ergm", date = "2020-12-30")
```
```
Warning message:
2020-12-31 log arrives in ~19 hours at 02 Jan 04:00 AEDT. Using previous!
```

Keep in mind that 17:00 UTC is not a hard deadline. Barring server issues, the logs are usually posted a little _before_ that time. I don't know when the script starts but the posting time seems to be a function of the number of entries: closer to 17:00 UTC when there are more entries (e.g., weekdays); earlier than 17:00 UTC when there are fewer entries (e.g., weekends). Again, barring server issues, the 'cranlogs' results are usually available _before_ 18:00 UTC.

Here's what you might see using the Honolulu example:

```{r logInfo, eval = FALSE}
logInfo(details = TRUE)
```
```
$`Today's log/result`
[1] "2020-12-31"

$`Today's log posted?`
[1] "Yes"

$`Today's results on 'cranlogs'?`
[1] "Yes"

$`Available log/result`
[1] "Posit/RStudio (2020-12-31); 'cranlogs' (2020-12-31)."

$`Current date-time`
[1] "01 Jan 09:01 HST -- 01 Jan 19:01 UTC"

$status
[1] "Everything OK."
```

The function uses your local time zone, which depends on R's ability to compute your local time and time zone (e.g., `Sys.time()` and `Sys.timezone()`). My understanding is that there may be operating system or platform specific issues that could undermine this.


------

### Data Fixes and Notes

There are three data errors and one data issue to note. I've patched the errors.

#### 1) jumbled logs at end of 2012 and first day of 2013 - Error

The logs collected between late 2012 and the beginning of 2013 are a bit jumbled. 

To understand the problem, we need to be know that the Posit/RStudio download logs are stored as separate files with a name/URL that embeds the log's date:

```
http://cran-logs.rstudio.com/2022/2022-01-01.csv.gz
```

For the logs in question, this convention was broken in three ways: i) some logs are effectively duplicated (same log, different names), ii) at least one mislabeled log and iii) the logs from 13 October through 28 December are offset by +3 days (e.g., the file with the name/URL "2012-12-01" contains the log for "2012-11-28"). As a result, we get erroneous download counts and actually lose the last three logs of 2012. Details are available online in this [note](https://github.com/lindbrook/packageRank/blob/master/docs/logs.md) in the package's GitHub repository.

Unsurprisingly, this affects download counts. 

Functions that rely on `cranlogs::cran_download()` (e.g., ['packageRank::cranDownloads()'](https://github.com/lindbrook/packageRank/blob/master/R/cranDownloads.R), ['adjustedcranlogs'](https://CRAN.R-project.org/package=adjustedcranlogs) and ['dlstats'](https://CRAN.R-project.org/package=dlstats)) are susceptible to the first error - duplicate names. My understanding is that this is because ['cranlogs'](https://CRAN.R-project.org/package=cranlogs) uses the date in the log rather than the date in the filename/URL to retrieve logs. To put it differently, ['cranlogs'](https://CRAN.R-project.org/package=cranlogs) can't detect multiple instances of logs with the same date. I found 3 logs with duplicate filename/URLs, and 5 additional instances of overcounting (including one of tripling). ['fixCranlogs()'](https://github.com/lindbrook/packageRank/blob/master/R/fixCranlogs.R) addresses this overcounting by recomputing the download counts using the correct log(s) when any of the eight problematic dates are requested. Details about the 8 days and `fixCranlogs()` can be found [here](https://github.com/lindbrook/packageRank/blob/master/docs/logs.md).

Functions that access logs via their filename/URL, e.g., `packageRank()` and `packageLog()`, are affected by the second and third defects - mislabeled and offset logs. [`fixDate_2012()`](https://github.com/lindbrook/packageRank/blob/master/R/fixDate_2012.R) addresses this by re-mapping problematic logs so that you get the log you expect.

#### 2) Windows R Application download spikes (Nov 2022 - March 2023) - Note

Typically, the pattern of R application downloads is a series of weekday peaks and weekends troughs. You can see this in the graph below, which plots the January 2022 data broken down by platform (Mac, Source, and Windows) and weekday/weekend (filled v. empty circles):

```{r R_jan2022}
plot(cranDownloads(package = "R", from = "2022-01", to = "2022-01"), 
  r.version = TRUE, weekend = TRUE)
```

However, between November 2022 and March 2023, this pattern was broken. On Sundays (06 November 2022 - 19 March 2023) and Wednesdays (18 January 2023 - 15 March 2023), there were noticeable, repeated, orders-of-magnitude spikes in the daily downloads of just the Windows version of R. 

```{r sundays, eval = TRUE, echo = FALSE}
plot(cranDownloads("R", from = "2022-10-06", to = "2023-04-14"))
axis(3, at = as.Date("2022-11-06"), labels = "2022-11-06", cex.axis = 2/3, 
  padj = 0.9)
axis(3, at = as.Date("2023-03-19"), labels = "2023-03-19", cex.axis = 2/3, 
  padj = 0.9)
abline(v = as.Date("2022-11-06"), col = "gray", lty = "dotted")
abline(v = as.Date("2023-03-19"), col = "gray", lty = "dotted")
```

These spikes appear to be real patterns in the data and not coding errors. Detailed, visual evidence for this can be found online in this [note](https://github.com/lindbrook/packageRank/blob/master/docs/windows_spikes.md) in the package's GitHub respository.

#### 3) doubled (or tripled) R application downloads counts (2023) - Error

From 2023-09-13 through 2023-10-02, the download counts for the R application returned by `cranlogs::cran_downloads(package = "R")`, is, with two exceptions, twice the count you'd get when looking at the actual log(s). The two exceptions are: 1) 2023-09-28 where the counts are identical but for a "rounding error" possibly due to NAs and 2) 2023-09-30 where there is actually a three-fold difference. 

Here are the relevant ratios of counts comparing ['cranlogs'](https://CRAN.R-project.org/package=cranlogs) results with counts based on the underlying logs:

```
    2023-09-12 2023-09-13 2023-09-14 2023-09-15 2023-09-16 2023-09-17 2023-09-18 2023-09-19
osx          1          2          2          2          2          2          2          2
src          1          2          2          2          2          2          2          2
win          1          2          2          2          2          2          2          2
    2023-09-20 2023-09-21 2023-09-22 2023-09-23 2023-09-24 2023-09-25 2023-09-26 2023-09-27
osx          2          2          2          2          2          2          2          2
src          2          2          2          2          2          2          2          2
win          2          2          2          2          2          2          2          2
    2023-09-28 2023-09-29 2023-09-30 2023-10-01 2023-10-02 2023-10-03
osx   1.000000          2          3          2          2          1
src   1.000801          2          3          2          2          1
win   1.000000          2          3          2          2          1
```

Details and code for replication can be found in issue [#69](https://github.com/r-hub/cranlogs/issues/69). [`fixRCranlogs()`](https://github.com/lindbrook/packageRank/blob/master/R/fixRCranlogs.R) corrects the problem. Note that there was a similar issue for package download counts around the same period but that is now fixed in ['cranlogs'](https://CRAN.R-project.org/package=cranlogs). For details, see issue [#68](https://github.com/r-hub/cranlogs/issues/68)

#### 4) seven lost logs (2025) - Error

In 2025, 7 logs, 8/25-8/26 and 8/29-9/02, appear to be lost. For what it's worth, both gaps were preceded by two unusually large number of downloads: Sun 8/24 (14,521,256) and Wed 8/27 & Thu 8/28 (16,860,505 and 16,477,023). These outliers are approximately twice the size of "typical" download counts (see graph below).

As a "fix", the missing dates (see cholera::missing.date), `cranDownloads()` does the following. First, when a missing date is included it prints a message in the console. Second, when plotting two gray polygons are added to the graph to highlight those dates. They are labeled with a "⌀" (empty set) on the top axis. Third, smoothers ignore the missing data.

The graph below, which plots the _total_ number of downloads recorded by the Posit/RStudio mirror from Sat 7/05 through Sun 9/14, shows the magnitude of the outliers and the two graphical fixes (open circles are weekends).

```{r missing}
plot(cranDownloads(from = "2025-07-05", to = "2025-09-10"), smooth = TRUE, 
  points = TRUE, weekend = TRUE)
```

------

### et cetera

This section describes some additional issues that may be of interest.

#### Bioconductor

Note that the "raw" Bioconductor package download are already aggregated by month.

#### country codes (top level domains)

While the IP addresses in the [Posit/RStudio logs](http://cran-logs.rstudio.com/) are anonymized, `packageCountry()` and `countryPackage()` the logs include ISO country codes or top level domains (e.g., AT, JP, US). 

Note that coverage extends to only about 85% of observations (approximately 15% country codes are NA), and  that there seems to be a a couple of typos for country codes: "A1" (A + number one) and "A2" (A + number 2). According to [Posit/RStudio's documentation](http://cran-logs.rstudio.com/), this coding was done using MaxMind's free database, which no longer seems to be available and may be a bit out of date.

#### memoization

To avoid the bottleneck of downloading multiple log files, `packageRank()` is currently limited to individual calendar dates. To reduce the bottleneck of re-downloading logs, which can approach 100 MB, ['packageRank'](https://CRAN.R-project.org/package=packageRank) makes use of memoization via the ['memoise'](https://CRAN.R-project.org/package=memoise) package.

Here's relevant code:

```{r memoization, eval = FALSE}
fetchLog <- function(url) data.table::fread(url)

mfetchLog <- memoise::memoise(fetchLog)

if (RCurl::url.exists(url)) {
  cran_log <- mfetchLog(url)
}

# Note that data.table::fread() relies on R.utils::decompressFile().
```

This means that logs are cached; logs that have already been downloaded in your current R session will not be downloaded again.

#### timeout

With R 4.0.3, the timeout value for internet connections became more explicit. Here are the relevant details from that release's ["New features"](https://cran.r-project.org/doc/manuals/r-release/NEWS.html):

```
The default value for options("timeout") can be set from environment variable
R_DEFAULT_INTERNET_TIMEOUT, still defaulting to 60 (seconds) if that is not set
or invalid.
```

This change can affect functions that download logs. This is especially true over slower internet connections or when you're dealing with large log files. To fix this, `fetchCranLog()` will, if needed, temporarily set the timeout to 600 seconds.