Issue with metric

[shashankg7]

Hi,

Thanks for hosting this library of github. I am trying it on my custom dataset. It works fine for ranking, but when I pass relevance scores > 1 (like 3,2,1 levels), it gives MAP, NDCG as 1 always.

I am not sure what is wrong. Can you please help me out?

Thanks

[jattenberg]

Hi @shashankg7 I'm not actively maintaining this. Seems like you need a mapping from scores to rankings. I'm happy to take a pull request if you could build one, it should be straight forward, basically just sorting.

[shashankg7]

HI @jattenberg . That problem is resolved. I have a small doubt related to RankLib. It would be really useful if you could resolve my doubt.

My problem involved working with set of queries and set of relevant documents corresponding to each query i.e. I have dataset with only relevance level of 1 and no zeros.

For training I can randomly sample o(irrelevant) samples from my corpus, but for testing I will only have (q,d) pairs with +1 label.

Does it make sense to evaluate on such dataset?