The task has three parts -- data collection, data exploration / algorithm development, then finally predictive modeling.
We want you to query the wikipedia API and collect all of the articles under the following wikipedia categories:
We want your code to be modular enough that any valid category from Wikipedia can be queried by your code.
The results of the query should be written to PostgreSQL tables, page and category. You will also need to build some sort of reference between the pages and categories. Keep in mind that a page can have many categories and a category can have many pages so a straight foreign key arrangement will not work.
optional
Make it so that your code can be run via a python script e.g.
$ docker run -v `pwd`:/src python -m download #SOME_CATEGORY#optional
Make it so that your code can query nested sub-categories e.g.
$ docker run -v `pwd`:/src python -m download #SOME_CATEGORY# #NESTING_LEVEL#Use Latent Semantic Analysis to search your pages. Given a search query, find the top 5 related articles to the search query.
optional
Make it so that your code can be run via a python script e.g.
$ docker run -v `pwd`:/src python -m search #SOME_TERM#In this part, we want you to build a predictive model from the data you've just indexed. Specifically, when a new article from wikipedia comes along, we would like to be able to predict what category the article should fall into. We expect a training script of some sort that is runnable and will estimate a model.
optional
Make it so that your code can be run via a python script e.g.
$ docker run -v `pwd`:/src python -m trainFinally, you should be able to pass the url of a wikipedia page and it will generate a prediction for the best category for that page, along with a probability of that being the correct category.
optional
Make it so that your code can be run via a python script e.g.
$ docker run -v `pwd`:/src python -m predict #URL#You may use the include docker-compose.yml file to build your project.