Investigate if we can share parts of the crawler code with the autowebcompat project

[marco-c]

The autowebcompat's crawler is at https://github.com/marco-c/autowebcompat/blob/master/collect.py.

For this issue, we should:

1. investigate whether it's feasible to write a single crawler that can be used for both projects (or does one project require something that really doesn't work for the other?);
2. create a crawler library and make both this project and autowebcompat use it.

If we find that we can't share the entire crawler, we can probably at least share parts of it (we could create a crawler_utils library).

[rhcu]

Hi @marco-c
From the first glance, many parts look similar. At least, how crawler goes through children links.

I think that we cannot use the single crawler because while we need to collect information only about Firefox browser, autowebcompat collects information from both Chrome and Firefox, also screenshots are added.

However, we can create a library with the main functionality of crawler.py to use it in autowebcompat. Or, can we just import similar functions from one file to another?

Also, I am not sure but is it possible to use downloaded builds for interaction with Firefox and Chrome in autowebcompat project?

P.S, I guess that we need to add this line utils.mkdir('data') to crawler.py because now it crushes if there is no such directory.