No interface to specify the Chrome Path cause selenium problem

[honorinli151]

In the Webrowser.py by default, the parameter chromeDriverPath=None, and then in the package Webcrawler.py, by default we use chrome, but no path specification possibility of chrome is in the API of Class Crawler. So still chromeDriverPath=None. And in the function initSeleniumDriver(self, retry=True),

if self.chromeDriverPath is not None:
                    params["executable_path"] = self.chromeDriverPath
                self.driver = webdriver.Chrome(**params)

With chromeDriverPath=None, we will never get params["executable_path"] = self.chromeDriverPath, so we will always get errror while using chrome with Class Crawler

selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

[hayj]

Thank you for the issue, I fixed it, you have to pass Browser params through browserParams dict in the Crawler init. Try this:

from webcrawler.crawler import *

def crawlingCallback(data, browser=None):
    print(data)

crawler = Crawler(["https://github.com/hayj/WebCrawler"],
                  crawlingCallback=crawlingCallback,
                  proxies=getProxiesTest(),
                  browserCount=1,
                  browserParams={"chromeDriverPath": "/home/hayj/Programs/browserdrivers/chromedriver"})
crawler.start()

[honorinli151]

The new commit files has a problem, in the build you have imported hjwebbrowser.utils, but in this file, you have imported a package called error404detctor which is not included in the folder wm-dist.

[honorinli151]

• And, more important is that, what is exactly the addToTheDatabaserawl and shouldn't it be addToTheDatabasecrawl with a 'c'?

• And for the data mentioned ahead, its assignment status which is object <REQUEST_STATUS.success: 1> is not encodable with mongodb, you need to convert it to be encodable like convert it to string.

• And, I did not find the parameter about stopping the crawler by number of pages crawled, perhaps, it is just because I am a bit tired, you have mentioned it somewhere but I did not see.

I think that should be all about my test with this package, I wish.

[hayj]

I fixed the README

If you store data in a Mongo database, you can use databasetools.mongo.dictToMongoStorable:

from databasetools.mongo import *
data = dictToMongoStorable(data)

I didn't implemented a counter yet, but you can inc your own counter when you call crawler.put("...") and stop it at a given threshold.

[honorinli151]

Thanks, but I do not think your inside counter proposition about the put will work, cause in your function htmlCallback in crawler.py, while this function handles the process and it automatically (no possibility to count the pages crawled), so it seems that I have to redefine the htmlCallback to set up the counter, but it seems too complexe. Any suggestions? Or I have missed something?

[honorinli151]

Anyway I am trying with the check of nums of sites in MongoDB.

[hayj]

I think you can try something like this (with a simple counter in the main callback):

counter = 0
def crawlingCallback(data, browser=None):
    global counter
    global logger
    global crawler
    status = data["status"]
    if status in [REQUEST_STATUS.success,
                  REQUEST_STATUS.timeoutWithContent,
                  REQUEST_STATUS.error404]:
        try:
            addToTheDatabase(data) # Your own function
        except Exception as e:
            logException(e, logger, location="crawlingCallback")
    counter += 1
    if counter >= 400:
        crawler.stop() # or os._exit(1)

[honorinli151]

I have used a Timer check every 60 seconds the MongoDB, like

def check_mongodb(crawler):
    client = MongoClient('mongodb://localhost:27017/')
    db = client.news_stock
    if db.error404.count() > 2:
        print("Over clapped!")
        crawler.stop()
t = threading.Timer(60, check_mongodb(crawler))
#t.cancel()

Anyway, it works but not sure about the efficiency.