Update ConsumedConditions scripts and add 2024 MC and DATA

[francescobrivio]

PR Description

This PR updates the ConditionsConsumed sub-package:

• Updated README with minimal set of instructions to run the script
• Added the 2024 MC and DATA cmsDrivers
  • These are taken from @atpathak's 2024_MC_data branch with two minor modifications:
    • Use data from Run2024G instead of Run2024D
    • Set the --beamspot parameter to be --beamspot DBrealistic, as done in all CMS MC campaigns starting from 2024
• Added all the txt files generated from running getAlCaCondInGT.py
• A few updates in ConditionsConsumed/getAlCaCondInGT.py to:
  • Improve readability and the command line options
  • Fix checkTagInFile method to avoid clashing cases where part of the tag name is included in multiple Records.
    • This was causing an issue especially with BeamSpotObjectsRcd and SimBeamSpotObjectsRcd, where the former was incorrectly marked as consumed in "GEN" step (while it's obviously not!).
  • Fix printAllRow method to avoid including in the output txt files 2 lines from the cmsDriver logs were the string / was present (Filter efficiency (taking into account weights)= (5) / (5) = 1.000e+00 +- 0.000e+00) which have nothing to do with the conditions: this was solved by looking in the log files for lines containing the string frontier instead.

PR Validation

The updated scripts were used to update the 2024 twiki: https://twiki.cern.ch/twiki/bin/view/CMS/AlCaDBHLT2024

[mmusich]

Maybe I am missing something... but:

cmsrel CMSSW_14_0_19
cd CMSSW_14_0_19/src/
cmsenv
git clone https://github.com/cms-AlCaDB/AlCaTools.git
git remote add francescobrivio https://github.com/francescobrivio/AlCaTools.git
git fetch francescobrivio
git cherry-pick dbe9182748eb574c2a00615ddc4ff875143ebe93
cd ConditionsConsumed/
./runCMSDrivers_mc2024.sh

head -70 output_step7_AODSIM.log

gives:

----- Begin Fatal Exception 03-Dec-2024 17:48:27 CET-----------------------
An exception of category 'NoProductResolverException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'AODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'AODSIMoutput'
   [3] Calling method for module TrackMVAClassifierPrompt/'initialStepClassifier1'
Exception Message:
No data of type "GBRForest" with label "MVASelectorIter0_13TeV" in record "GBRWrapperRcd"
 Please add an ESSource or ESProducer to your job which can deliver this data.
----- End Fatal Exception -------------------------------------------------

[francescobrivio]

Ciao @mmusich indeed you are right, I hadn't noticed this crash hidden in the very long printout.

The driver for step-7 is:

cmsDriver.py step7 --conditions auto:phase1_2024_realistic \
  -n 5 \
  --era Run3_2024 \
  --geometry DB:Extended \
  -s RAW2DIGI,L1Reco,RECO,RECOSIM \
  --datatier AODSIM \
  --eventcontent AODSIM \
  --filein file:output_step6_HLT.root \
  --fileout output_step7_AODSIM.root \
  --customise_commands='process.GlobalTag.DumpStat =cms.untracked.bool(True)' \
  --outputCommands "keep *, drop *_mix_*_*" |& tee output_step7_AODSIM.log

which at (a very) first glance looks correct.

The crash message seems to indicate a possible mismatch between CMSSW and the GT...I'll investigate further tomorrow.
Or maybe, since the command really comes from @atpathak, he or the other AlCa conveners @perrotta @consuegs have already an idea?

[mmusich]

The crash message seems to indicate a possible mismatch between CMSSW and the GT.

this error message is weird, because in principle that record label is used exclusively for the phase-0 tracking and indeed in cmssw the label is switched to MVASelectorInitialStep_Phase1:

https://github.com/cms-sw/cmssw/blob/15e4fe71930ab92d58e8721e73c802ba92262a0b/RecoTracker/IterativeTracking/python/InitialStep_cff.py#L344-L347

so that label should not be looked at all (in normal conditions, and indeed I've never seen this failure in production mode)
It looks as if the era modifier is not applied correctly in the cmsDriver.py.

I can only speculate that --outputCommands "keep * (which is not usually how the event content is formed) triggers the execution of some piece of code unprotected by the phase1 modifier.

[francescobrivio]

I have tried to run the same step7-cmsDriver with:

1. No keep/drop statements, it crashes with:

   ----- Begin Fatal Exception 04-Dec-2024 09:36:18 CET-----------------------
   An exception of category 'ProductNotFound' occurred while
      [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
      [1] Running path 'AODSIMoutput_step'
      [2] Prefetching for module PoolOutputModule/'AODSIMoutput'
      [3] Calling method for module MuonSimClassifier/'muonSimClassifier'
   Exception Message:
   Principal::getByToken: Found zero products matching all criteria
   Looking for type: std::vector<TrackingParticle>
   Looking for module label: mix
   Looking for productInstanceName: MergedTrackTruth
   
      Additional Info:
         [a] If you wish to continue processing events after a ProductNotFound exception,
   add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.
   ----- End Fatal Exception -------------------------------------------------
   

2. Only using --outputCommands "drop *_mix_*_*", crashes with same error
3. Using --era Run3_2023, it crashes with:

   ----- Begin Fatal Exception 04-Dec-2024 09:39:19 CET-----------------------
   An exception of category 'NoProductResolverException' occurred while
      [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
      [1] Running path 'AODSIMoutput_step'
      [2] Prefetching for module PoolOutputModule/'AODSIMoutput'
      [3] Calling method for module TrackMVAClassifierPrompt/'initialStepClassifier1'
   Exception Message:
   No data of type "GBRForest" with label "MVASelectorIter0_13TeV" in record "GBRWrapperRcd"
    Please add an ESSource or ESProducer to your job which can deliver this data.
   ----- End Fatal Exception -------------------------------------------------
   

[atpathak]

Yes, indeed, I also came across with the same fatal exception, I discussed with @mmusich regarding that in MM, and since then I didn't find any other good solutions yet.

When I ran first time, I also got the exception with Looking for module label: mix, and that's a reason I put this in
--outputCommands "drop *_mix_*_*", but after then I also somehow haven't noticed the MVASelectorIter0_13TeV exception.

Interestingly, we are using CMSSW_14_0_19 and 2024 MC GT is 'phase1_2024_realistic':'140X_mcRun3_2024_realistic_v26', and indeed I don't find any "GBRForest" with label "MVASelectorIter0_13TeV" in record "GBRWrapperRcd" in https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/140X_mcRun3_2024_realistic_v26. (if you search with this name GBRWrapperRcd)

Yes, indeed, for some reason, it is trying to access a phase 0 pixel tracking record.
At the moment, this step 7 is not working with current CMS release and 2024 MC GT. I will let you know if I get anything. @perrotta Please let me know if you have any suggestions. Thank you!