Renewal of payload proxies

[arrabito]

Dear all,

Could you please tell me which mechanism is responsible for the renewal of payload proxies, is it the ProxyRenewalAgent?

At one site payload proxies get expired after 12 hours :

PEM file has expired /tmp/tmpnmrlzdpo is not valid after 2025-10-29 21:43:00

while at another site they don't.

So I guess that there is some different configuration at the 2 sites.

Looking at the pilot logs I found that at one site (where proxies get expired) there is the following variable set:

MYPROXY_SERVER

while at the other is not. Could it be the cause of the renewal problem?

As an additional information, but I don't know if it's relevant, we have the following configuration for the ProxyRenewalAgent:

ProxyRenewalAgent
        {
          PollingTime = 900
          #Email to use as a sender for the expiration reminder
          MailFrom = [proxymanager@diracgrid.org](mailto:proxymanager@diracgrid.org)
          #Renew, or not, the proxies stored on the MyProxy service
          UseMyProxy = False
          MinimumLifeTime = 3600
          RenewedLifeTime = 54000
        }

Thank you for your help,

Luisa

[fstagni]

Hi,
the ProxyRenewalAgent, despite its name, does not renew proxies (name is something historical), just purge old ones.
MYPROXY_SERVER is another historical thing that is completely unsupported by now, so do not look at it.

The ComputingElement asks for proxy renewal, with these settings:

        self.minProxyTime = gConfig.getValue("/Registry/MinProxyLifeTime", 10800)  # secs
        self.defaultProxyTime = gConfig.getValue("/Registry/DefaultProxyLifeTime", 43200)  # secs
        self.proxyCheckPeriod = gConfig.getValue("/Registry/ProxyCheckingPeriod", 3600)  # secs

Modify at will.

[arrabito]

Hi @fstagni

Thank you for the explanations. I will try to change this value /Registry/DefaultProxyLifeTime.

But I don't really understand how at one site there are successfull jobs lasting more than 12 hours while at another I see the PEM expired message after 12 hours....

Any idea?

[fstagni]

Are these HtCondorCE or ARC CE sites? Or SSH?

[arrabito]

They are both HtCondorCE.

Then we also have ARC CE and SSH sites. Is there anything to know about the last 2?

[fstagni]

Actually, IIUC the MYPROXY_SERVER identifies the myproxy server used for proxy renewal in the HTCondor world. That might have settings such that expires the proxy after 12 hours.
ARC and HTCondor will try to manage your proxies, for SSH instead it's all internal to DIRAC.

[arrabito]

I see. So let's see if my change in /Registry/DefaultProxyLifeTime is enough. I will let you know.

[arrabito]

Hi @fstagni I've set:

/Registry/DefaultProxyLifeTime = 86400

but I still see proxy generated with 12 hours of lifetime, see below:

2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Requesting proxy', 'for /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr@ctao_prod
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Found Job LogLevel JDL parameter with value: INFO
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Submitting JobWrapper Job21375244 to PoolCE
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Job being submitted (DIRAC JobID: 21375244; Task ID: 0)
2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/Task-0 INFO: Payload proxy information:
subject      : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr/CN=8401460194/CN=6460439606/CN=3911717654/CN=8853180364
issuer       : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr/CN=8401460194/CN=6460439606/CN=3911717654
identity     : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr
timeleft     : 11:59:58
DIRAC group  : ctao_prod
path         : /tmp/tmpl18bwej3
username     : arrabito
properties   : NormalUser, PrivateLimitedDelegation, ProductionManagement, JobAdministrator
VOMS         : True
VOMS fqan    : ['/ctao/Role=production']

Making again jobs fail.

Any idea?

Thank you.

[fstagni]

Hi, few questions:

• This is DIRAC v8 or v9? There have been changes between the 2 versions in this respect.
• what do you see in the log of the pilots going to the other site (where it does not fail?)
• in the jobAgent I see self.defaultProxyLength = 86400 * 5 which is 5 days, so there's something wrong.

[arrabito]

Hi @fstagni

* This is DIRAC v8 or v9? There have been changes between the 2 versions in this respect.

Sorry, I've forgot to specify that it's v8.0.71

* what do you see in the log of the pilots going to the other site (where it does not fail?)

I see the same duration of 12 hours proxy:

2025-11-19 11:58:46 UTC WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Requesting proxy', 'for /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr@ctao_prod
....
timeleft     : 11:59:58

but when jobs have a duration of more than 12 hours, I don't see the error : "Proxy has expired" as at PIC, and they finish well.

* in the jobAgent I see `self.defaultProxyLength = 86400 * 5` which is 5 days, so there's something wrong.

that's strange indeed

[fstagni]

Would you by chance be able to try with v9?

[aldbr]

Hi @arrabito, here we are talking about the user proxy downloaded by the JobAgent once it fetches a job, aren't we?

I see that the user proxy could be renewed in ComputingElement._monitorProxy() IIUC:

DIRAC/src/DIRAC/Resources/Computing/ComputingElement.py

Lines 359 to 461 in 106cca5

	def _monitorProxy(self, payloadProxy=None):
	"""Base class for the monitor and update of the payload proxy, to be used in
	derived classes for the basic renewal of the proxy, if further actions are
	necessary they should be implemented there
	:param str payloadProxy: location of the payload proxy file
	:returns: S_OK(filename)/S_ERROR
	"""
	if not payloadProxy:
	return S_ERROR("No payload proxy")
	# This will get the pilot proxy
	ret = getProxyInfo()
	if not ret["OK"]:
	pilotProxy = None
	else:
	pilotProxy = ret["Value"]["path"]
	self.log.notice("Pilot Proxy:", pilotProxy)
	retVal = getProxyInfo(payloadProxy)
	if not retVal["OK"]:
	self.log.error("Could not get payload proxy info", retVal)
	return retVal
	self.log.verbose(f"Payload Proxy information:\n{formatProxyInfoAsString(retVal['Value'])}")
	payloadProxyDict = retVal["Value"]
	payloadSecs = payloadProxyDict["chain"].getRemainingSecs()["Value"]
	if payloadSecs > self.minProxyTime:
	self.log.verbose("No need to renew payload Proxy")
	return S_OK()
	# if there is no pilot proxy, assume there is a certificate and try a renewal
	if not pilotProxy:
	self.log.info("Using default credentials to get a new payload Proxy")
	return gProxyManager.renewProxy(
	proxyToBeRenewed=payloadProxy,
	minLifeTime=self.minProxyTime,
	newProxyLifeTime=self.defaultProxyTime,
	proxyToConnect=pilotProxy,
	)
	# if there is pilot proxy
	retVal = getProxyInfo(pilotProxy)
	if not retVal["OK"]:
	return retVal
	pilotProxyDict = retVal["Value"]
	if "groupProperties" not in pilotProxyDict:
	self.log.error("Invalid Pilot Proxy", "Group has no properties defined")
	return S_ERROR("Proxy has no group properties defined")
	pilotProps = pilotProxyDict["groupProperties"]
	# if running with a pilot proxy, use it to renew the proxy of the payload
	if Properties.PILOT in pilotProps or Properties.GENERIC_PILOT in pilotProps:
	self.log.info("Using Pilot credentials to get a new payload Proxy")
	return gProxyManager.renewProxy(
	proxyToBeRenewed=payloadProxy,
	minLifeTime=self.minProxyTime,
	newProxyLifeTime=self.defaultProxyTime,
	proxyToConnect=pilotProxy,
	)
	# if we are running with other type of proxy check if they are for the same user and group
	# and copy the pilot proxy if necessary
	self.log.info("Trying to copy pilot Proxy to get a new payload Proxy")
	pilotProxySecs = pilotProxyDict["chain"].getRemainingSecs()["Value"]
	if pilotProxySecs <= payloadSecs:
	errorStr = "Pilot Proxy is not longer than payload Proxy"
	self.log.error(errorStr)
	return S_ERROR(f"Can not renew by copy: {errorStr}")
	# check if both proxies belong to the same user and group
	pilotDN = pilotProxyDict["chain"].getIssuerCert()["Value"].getSubjectDN()["Value"]
	retVal = pilotProxyDict["chain"].getDIRACGroup()
	if not retVal["OK"]:
	return retVal
	pilotGroup = retVal["Value"]
	payloadDN = payloadProxyDict["chain"].getIssuerCert()["Value"].getSubjectDN()["Value"]
	retVal = payloadProxyDict["chain"].getDIRACGroup()
	if not retVal["OK"]:
	return retVal
	payloadGroup = retVal["Value"]
	if pilotDN != payloadDN or pilotGroup != payloadGroup:
	errorStr = "Pilot Proxy and payload Proxy do not have same DN and Group"
	self.log.error(errorStr)
	return S_ERROR(f"Can not renew by copy: {errorStr}")
	if pilotProxyDict.get("hasVOMS", False):
	return pilotProxyDict["chain"].dumpAllToFile(payloadProxy)
	attribute = Registry.getVOMSAttributeForGroup(payloadGroup)
	vo = Registry.getVOMSVOForGroup(payloadGroup)
	retVal = VOMS().setVOMSAttributes(pilotProxyDict["chain"], attribute=attribute, vo=vo)
	if not retVal["OK"]:
	return retVal
	chain = retVal["Value"]
	return chain.dumpAllToFile(payloadProxy)

Can you spot the logs in your pilot outputs to compare them? Such as:

- Pilot Proxy:
- No need to renew payload Proxy
- Using default credentials to get a new payload Proxy

[arrabito]

Hi @fstagni
I've just tried with v9.0.9 with our certification instance.
I've tried very short testing jobs and I've set the following configuration in the CS:

Registry/DefaultProxyLifeTime = 86400

In the log I cannot see any of the strings :

- Pilot Proxy:
- No need to renew payload Proxy
- Using default credentials to get a new payload Proxy

I attach here the full log.
pilot_pic.txt
Thank you for your help.

Luisa

[arrabito]

As an additional information, the server version is a little behind the pilot version.
The pilot version is v9.0.9 while the server one is v9.0.4, but I guess that here what matters is the pilot version.
Anyway, I'm also going to update the server to v9.0.9.

[arrabito]

Hi @arrabito, here we are talking about the user proxy downloaded by the JobAgent once it fetches a job, aren't we?

yes

[arrabito]

I've forgot to mention that for the last testing job, as it was very short I don't know if it had worked well with a long job.
However, I still see:

timeleft : 11:59:57

in the Payload proxy information, despite the setting:

Registry/DefaultProxyLifeTime = 86400

but I don't know if this is expected.

2025-12-02T07:55:18,168009Z WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Submitting JobWrapper Job98673 to InProcessCE
2025-12-02T07:55:19,143247Z WorkloadManagement/JobAgent/InProcess INFO: Payload proxy information:
subject      : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr/CN=1182140267/CN=7693509471/CN=2001281570/CN=3223501301
issuer       : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr/CN=1182140267/CN=7693509471/CN=2001281570
identity     : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr
**timeleft     : 11:59:57**
DIRAC group  : ctao-prod
DiracX       : True
path         : /tmp/tmpyofiki5_
username     : arrabito
properties   : NormalUser, PrivateLimitedDelegation, ProductionManagement, JobAdministrator
VOMS         : True
VOMS fqan    : ['/ctao/Role=production']

[arrabito]

Hi @fstagni @aldbr, after some investigations with the help of @aldbr we have discovered that there is a limit of 12 hours for the proxy duration imposed by our IAM server. That's why despite what we set for:

Registry/DefaultProxyLifeTime

the max duration is 12 hours.

Nevertheless the payload proxy should have been renewed by ComputingElement._monitorProxy() anyway.

Now the strange thing is that I've submitted another set of jobs, after changing :

LocalCEType=Pool
to
LocalCEType=InProcess

and only when using LocalCEType=InProcess I see in the logs the renewal as effective :

2025-12-02 22:22:24 UTC WorkloadManagement/JobAgent/InProcess INFO: Using Pilot credentials to get a new payload Proxy

and every 20 minutes (which corresponds to our ProxyCheckingPeriod):

NOTICE: Pilot Proxy: ....

As PoolCE uses in turn InProcessCE, I don't really understand the reason why the proxy renewal didn't work in that case.

Any idea?

[arrabito]

Just to let you know that we managed to get an extension in the proxy duration from our IAM server.
Now that the limitation is much higher (7 days) I confirm that the :

Registry/DefaultProxyLifeTime

configuration is effective and visible in the pilot log:

2025-12-05T14:13:38,436720Z WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Requesting proxy', 'for /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr@ctao-prod
2025-12-05T14:13:39,141986Z WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Found Job LogLevel JDL parameter with value INFO
2025-12-05T14:13:39,605135Z WorkloadManagement/JobAgent/WorkloadManagement/JobAgent INFO: Submitting JobWrapper Job98683 to InProcessCE
2025-12-05T14:13:40,630642Z WorkloadManagement/JobAgent/InProcess INFO: Payload proxy information:
subject      : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr/CN=1182140267/CN=5929220001/CN=898456257/CN=183174522
issuer       : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr/CN=1182140267/CN=5929220001/CN=898456257
identity     : /DC=org/DC=terena/DC=tcs/C=FR/O=Centre national de la recherche scientifique/CN=ARRABITO Luisa luisa.arrabito.5@cnrs.fr
timeleft     : 41:59:58
DIRAC group  : ctao-prod

Apart from the fact that I don't understand why the I cannot see logs concerning proxy renewal when using LocalCEType=Pool, I would consider this discussion as solved.

Thanks for your help.