Missing data

[DougBurke]

With the current data loaded, the Missions tab lists 389 papers for Chandra, but we know there are 600+ papers associated with Chandra. It may be that some do not have a direct data link - e.g. http://labs.adsabs.harvard.edu/ui/abs/2000A&A...359..489C is associated with Chandra but seems to have no direct obsid links - but this does not explain everything.

Digging into this, it looks like there is an issue with A&A papers: for example we have both

//ads.harvard.edu/sem/bib#2001A&A...380..251G
//ads.harvard.edu/sem/bib#2001A%26A...380..251G

(leading http: has been removed to avoid creating links to things that don't exist) so somewhere in the data ingest/creation there's a URIs are not being "normalised" in the same way.

[DougBurke]

Interestingly,

http://labs.adsabs.harvard.edu/semantic2/alpha/explorer/publications#fq=text%3A2000A%26A...359..489C&q=*%3A*

has CHANDRA listed under the Missions tab, but

http://localhost:3000/semantic2/alpha/explorer/publications#fq=text%3A2000A%26A...359..489C&q=_%3A_

does not (on both OS-X and Linux). So there's a possibility of a "recent" bug as well.

Note that

http://labs.adsabs.harvard.edu/semantic2/alpha/explorer/publications#fq=text%3A2001A%26A...380..251G&q=*%3A*

does list CHANDRA under the missions facet but has no data listed in its "Datasets" section.

[DougBurke]

& is a valid character for a URI fragment - as discussed in http://stackoverflow.com/questions/2849756/list-of-valid-characters-for-the-fragment-identifier-in-an-url - so I suggest that

a) file names remain % encoded
b) URI generation of the fragments are not % encoded

as this should lead to the smallest code churn.

[DougBurke]

Looks to be (partly/all?) down to Sherry's input data since I found

cat ../chandradata/Publications/2000A%26A...359..489C.xml
 <paper>
  <bibcode>2000A%26A...359..489C</bibcode>
  <classified_by>CDA</classified_by>
  <paper_type>science</paper_type>
  <flags>
      <data_use>indirect</data_use>
      <multi_observatory />
      <followup />
  </flags>
 </paper>

I am testing a fix to Chandra input (to remove the percent encoding in this case) but it'll take some time to fix. I think we should ask Sherry to use & rather than %26 for this case, since I think other Chandra data uses this encoding.

[DougBurke]

This change leads to having 633 papers loaded with Chandra data with 423 appearing in the Missions facet, which is an improvement. Need to determine whether the other 200 or so papers are expected to have no listing.

[DougBurke]

With just the Chandra data added to Sesame and Solr I get 633 papers and 423 papers listed under the Missions=Chandra selection.

If I then query sesame with

SELECT (COUNT(?s) as ?c) WHERE { 
  ?s adsobsv:datum_p "true" .
}

I get a value of 425 when I was naively expecting 423.I am assuming that adsobsv:datum_p maps to the data_use flag equalling "direct" in the Chandra publications XML.

Looks like there are 2 mystery papers in sesame since

SELECT (COUNT(?s) as ?c) WHERE { 
  ?s a fabio:JournalArticle .
}

returns 635 rather than 633. Presumably these 2 are the same as above.

Looking at the biblist files I see:

% wc -l sherry.p.a.biblist.txt
 632 sherry.p.a.biblist.txt
% wc -l sherry.p.a.biblist.txt.cut 
 630 sherry.p.a.biblist.txt.cut 
% wc -l hutoverlap.biblist.txt 
   4 hutoverlap.biblist.txt

so it's not clear why appsem shows 633 when I would expect 634 (as using the .cut file at present). Aha, it turns out that
there is a overlap between the two biblist files which would explain the 633.

% cat hutoverlap.biblist.txt sherry.p.a.biblist.txt.cut | sort | uniq -d 
2006ApJS..167..161K

This is not going to explain all the issues we are seeing and should go away when we go to loading all the Chandra data (and so remove the hut-separated processing steps).

[DougBurke]

Looking at the bibcodes in sherry.p.a.biblist.txt.cut and hutoverlap.biblist.txt, I get

# papers = 633
# with data = 423
# with data = 210

where "with data" means the chandradata/Publications/.xml file has at least one paper/data/obsid element. So it does look like the appsem view of 633 papers, only 423 of them being associated with Chandra is "correct" for the current code.

There's still the issue of the mystery extra bibcodes in sesame from the ones in the biblist files.

[DougBurke]

The two extra papers in sesame compared to appsem/AstroExplorer turn out to be the ones that are cut because of simbad issues:

% cat sherry.p.a.biblist.txt sherry.p.a.biblist.txt.cut | sort | uniq -u
2003ApJ...595..727I
2004ApJ...612..729H

and from a comparison of all the papers in sesame with those in union of sherry.p.a.biblist.txt.cut and hutoverlap.biblist.txt, the two extra are:

http://ads.harvard.edu/sem/bib#9ef9d845-b2c7-48d4-9dd0-411ccd0a43df -> 2003ApJ...595..727I -> http://ads.harvard.edu/sem/bib#2003ApJ...595..727I
http://ads.harvard.edu/sem/bib#3a2d6fef-169e-4816-89f1-598923bb6416 -> 2004ApJ...612..729H -> http://ads.harvard.edu/sem/bib#2004ApJ...612..729H

[DougBurke]

Note there are three HUT papers with no associated data sets. If you load in just HUT then you get 152 papers but only 149 papers listed under missions/obs types/data prod types/wavelengths. Running a sparql query:

res = sesame.makeQuery("SELECT ?s ?o { ?s a fabio:JournalArticle . OPTIONAL { ?s adsbase:aboutScienceProduct ?o . } }")

lists the following papers with no science products:

http://ads.harvard.edu/sem/bib#1992AAS...18110208L
http://ads.harvard.edu/sem/bib#1994ASPC...54..281K
http://ads.harvard.edu/sem/bib#2000ApJ...537..977D

Looking at the ADS results for these papers it does seem that this is correct - the papers mention HUT but it looks like we do not know what obsids - since for instance

http://adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2000ApJ...537..977D&link_type=DATA

gives the line "MAST References (HST, HUT, EUVE, IUE)" which is a link to

http://archive.stsci.edu/mastbibref.php?bibcode=2000ApJ...537..977D

which shows data for EUVE, HST and IUE but not HUT. The other two papers have no "data links" section in their ADS record.

So this is another example of the Chandra "data_use=indirect" flag.