Example reports.

[rhcu]

I've added reports for https://www.mozilla.org/, https://www.python.org/, and https://stackoverflow.com/.
Some results:
Almost in all reports, in the file mozilla-central/widget/GfxInfoX11.cpp line 212 - mIsIntel = true;, is uncovered.
Another examples of uncovered files are mozilla_LockFTLibrary and mozilla_UnlockFTLibrary in Mozilla-central/gfx/2d/Factory.cpp both in Mozilla and Python reports.
Mostly, they are looking same with small differences. The average number of uncovered lines is ~200. Seems that Stackoverflow report has fewer lines uncovered because actions were more simple (were no input types: only links and buttons).
Something that I've noticed: in these reports only C++ files are uncovered, while all uncovered files from Stackoverflow are included into another 2 reports.

example_reports.zip

[marco-c]

A couple comments:

1. Maybe we should remove the lines which have "0" coverage? They are lines which were covered in both the crawler run and the tests, so we don't really care about them. We are already cleaning files which only contain 0 since Clean coverage if has no problems #70, but maybe we should also remove the "0" lines in general, so we have something cleaner to look at. Or maybe not, I'm undecided.
2. If JS is missing maybe there's a problem with the JS parsing. Could you check if the JS files are also missing if you don't perform the diff?

Almost in all reports, in the file mozilla-central/widget/GfxInfoX11.cpp line 212 - mIsIntel = true;, is uncovered.

This is interesting, maybe we are never testing this configuration.

Another examples of uncovered files are mozilla_LockFTLibrary and mozilla_UnlockFTLibrary in Mozilla-central/gfx/2d/Factory.cpp both in Mozilla and Python reports.

I wonder why we see this but we don't see the files which call this function (https://dxr.mozilla.org/mozilla-central/source/gfx/skia/skia/src/ports/SkFontHost_cairo.cpp#763 or https://dxr.mozilla.org/mozilla-central/source/gfx/cairo/cairo/src/cairo-ft-font.c#1533). Maybe we should keep the full reports too, to analyze these cases, at least until we are sure everything is correct.

[rhcu]

Maybe we should remove the lines which have "0" coverage?

Probably, we can use '--css-file' option of genhtml with own css file where lines with 0 coverage won't be colored?

If JS is missing maybe there's a problem with the JS parsing. Could you check if the JS files are also missing if you don't perform the diff?

Turned out that there are no JS records before diffing too. I was more optimistic and thought that they are all tested. :) I'll try to find a problem in the code that causes this.

Maybe we should keep the full reports too, to analyze these cases, at least until we are sure everything is correct.

I will add generate_html option for the full report. It is stored as JSON file now.

[rhcu]

Error: 'dict' object has no attribute 'iteritems'. Couldn't find source info for resource://gre/modules/GMPExtractorWorker.js, removing record
With this, all files from jsvm_dir are removed. It turned out that lcov_rewriter.py uses iteritems, while in Python 3 it was replaced with just items. Probably, this is the problem.

[marco-c]

With this, all files from jsvm_dir are removed. It turned out that lcov_rewriter.py uses iteritems, while in Python 3 it was replaced with just items. Probably, this is the problem.

Ah, we should fix this in lcov_rewriter.py. We can replace iteritems with viewitems (from future.utils import viewitems).

[marco-c]

Probably, we can use '--css-file' option of genhtml with own css file where lines with 0 coverage won't be colored?

Maybe this is the best option, so we still show the 0s but we don't attract the attention to them.

[rhcu]

We can replace iteritems with viewitems (from future.utils import viewitems).

I've made this change and script doesn't produce errors now; however, output.json doesn't contain any JS file anyway, while jsvm_output_file.info has some of those. E.g, there is toolkit/components/osfile/osfile.jsm file in jsvm_output_file.info, but in output.json there are only .cpp files in this folder.

Probably, there is also an error in how I am passing JS coverage .info file to grcov command? https://github.com/marco-c/coverage-crawler/blob/f164a6de4a961277cb4006c6939290526bf5c955/crawler.py#L268

[marco-c]

(discussion about JS moved to #98 (comment))

[rhcu]

Some more results from the fixed crawler:

209/8837 (uncovered lines/covered lines)- Python & Wikipedia
420/33058 - Mozilla (worked mostly on Instagram website after interacting with a link leading to it)
213/9453 - Rust

Reports from Python & Wikipedia and Rust look very similar.

Examples from coverage information:

Not covered lines in all reports:

1. /mozilla-central/gfx/2d/Factory.cpp
   

2. /mozilla-central/gfx/thebes - 3 files:
   *gfxPlatform
   
   *gfxPlatformFonts
   
   *gfxPlatformGtk
   

3. The maximum number of uncovered lines I've noticed so far is 111 in /mozilla-central/toolkit/xre/nsNativeAppSupportUnix.cpp

4. Actually, no uncovered JS files. Probably, they are all well tested since they are present in output.json

5. I used 30 actions per each page instead of 20, and, as a result, there are more uncovered files. I've downloaded all artifacts on July, 1.

Only in Mozilla (mostly, Instagram) report:

1. Weird case - function is present as executed but all lines are covered (file /mozilla-central/dom/bindings/BindingUtils.h)
   
2. Only here I noticed network files present: DNS, HTTP. As well as cache and databases. Probably, this is because of Instagram specific functionality.

*Relevant to DBs (file /mozilla-central/dom/indexedDB/ActorsParent.cpp; only 2 lines):

*Relevant to Cache (10 lines; file /mozilla-central/netwerk/cache2/CacheFileIOManager.cpp):

All reports with steps taken while crawling:
reports.zip
The Full Report in JSON format:
FullReport.zip

[gmierz]

Nice work Assiya! The file you showed below is a header file (.h) which have a few problems in their coverage. One problem is that the number of lines shown in a covered file for a header often doesn't match from one run to the next. There are probably more bugs, and this could be one of them.

[marco-c]

Actually, no uncovered JS files. Probably, they are all well tested since they are present in output.json

We should investigate this more, it seems strange.

Nice work Assiya! The file you showed below is a header file (.h) which have a few problems in their coverage. One problem is that the number of lines shown in a covered file for a header often doesn't match from one run to the next. There are probably more bugs, and this could be one of them.

The problem here is likely that the function is a templated function, so basically there are two functions here sharing the same lines. Maybe in one case both were executed and in the other case only one was executed.
It's a side effect of the diffing, rather than a problem in the coverage data.

*Relevant to Cache (10 lines; file /mozilla-central/netwerk/cache2/CacheFileIOManager.cpp):

We can probably try to write a GTest for this one. And maybe some test for the gfx/ cases too, but let's start with this one which might be easily feasible.

[marco-c]

Actually, no uncovered JS files. Probably, they are all well tested since they are present in output.json

We should investigate this more, it seems strange.

@rhcu sent me a list of the JS files covered in the run, and the list is pretty short. There must be something else wrong going on.
I'm filing a separate issue to investigate this.

[marco-c]

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1476544 for the SmartCacheSize function in CacheFileIOManager.cpp.

[marco-c]

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1477111 for the atom code in gfxPlatformGtk.