Pipeline runs seem slower as of recent version? [general]

[MilesMcBain]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I upgraded from targets 1.8.0 to 1.11.4. - Yes I know I was a bit behind, and while I like the new style of console output, I do feel like my pipelines are taking longer to complete.

Check out this report:

> targets::tar_make()
here() starts at C:/Users/msmcba/repos/popsas_verification
[conflicted] Will prefer dplyr::filter over any other package.
+ popsas_r_results dispatched
✔ popsas_r_results completed [656ms, 128.82 kB]
+ results_comparison dispatched
✔ results_comparison completed [0ms, 244.49 kB]
✔ ended pipeline [6.2s, 2 completed, 4 skipped]
There were 14 warnings (use warnings() to see them)

Target processing took 656ms out of the 6.2s of pipeline time. I've noticed that the pipeline runs seem to hang at the end with a step that says: → de-duplicating _targets/meta/meta for a few seconds... much more than would seem reasonable for a simple de-dupe.

Anyone else feeling things are a bit less snappy?

[wlandau]

Thanks for reporting this. I'm surprised, especially given all the optimization that has happened since version 1.8.0. What kinds of pipelines do you see slowness? Is it just large pipelines, or small ones too?

I've noticed that the pipeline runs seem to hang at the end with a step that says: → de-duplicating _targets/meta/meta for a few seconds... much more than would seem reasonable for a simple de-dupe.

How large is that metadata?

Is it possible you're running into #1480? Looks like Rdatatable/data.table#3509 is still open at the time I'm writing this.

If you have a reprex and/or profiling data, I can dive further.

[MilesMcBain]

Thanks for the reply Will. I don't think #1480 is related to my case.

I can dig a bit further on my end. My suspicion is that this may be something related to my work environment, especially the Windows aspect.

[wlandau]

Thanks. Over the years, I've heard similar reports of slowness that turned out to come from security tools and network drives.