Mogi example main program fails

[EJFielding]

I was also unable to run the main example code for the Mogi model:
models/mogi/examples] fielding% mogi altar: could not resolve 'mogi' into a component that implements protocol 'altar.models'

[aivazis]

Eric, Both of these are the same error, caused by the fact that the legacy makefile doesn’t build the new library. I will fix asap.

…

-- Michael

On Nov 20, 2018, at 7:59 AM, Eric Fielding ***@***.***> wrote: I was also unable to run the main example code for the Mogi model: models/mogi/examples] fielding% mogi altar: could not resolve 'mogi' into a component that implements protocol 'altar.models' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[EJFielding]

Michael, Thanks for the explanation. I was not sure if it was the same cause or not. ++Eric From: Michael Aivazis <notifications@github.com> Reply-To: AlTarFramework/altar <reply@reply.github.com> Date: Tuesday, November 20, 2018 at 8:21 AM To: AlTarFramework/altar <altar@noreply.github.com> Cc: "Eric J. Fielding" <Eric.J.Fielding@jpl.nasa.gov>, Author <author@noreply.github.com> Subject: Re: [AlTarFramework/altar] Mogi example main program fails (#7) Eric, Both of these are the same error, caused by the fact that the legacy makefile doesn’t build the new library. I will fix asap.

-- Michael

On Nov 20, 2018, at 7:59 AM, Eric Fielding ***@***.***> wrote: I was also unable to run the main example code for the Mogi model: models/mogi/examples] fielding% mogi altar: could not resolve 'mogi' into a component that implements protocol 'altar.models' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#7 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AG6uSBw8D5fxtYCa3nymaudpmYmILsBmks5uxCwKgaJpZM4YraGz>.

[EJFielding]

I am now getting a different error when I try to run the Mogi example main model:

models/mogi/examples] fielding% mogi
altar: unable to find support for <fast> mode
altar: file='/Users/fielding/Documents/progs/Caltech/AlTar-2.0/models/mogi/mogi/Mogi.py', line=146, function='initialize': firewall breached; aborting...

[aivazis]

Eric, It looks like the C++/CUDA implementations are not available in your installation. What di you use to build altar? As a check, edit mogi.pfg and switch the “mode” to “native”; this will fall back to the pyre python implementation. Verify that altar behaves. — Michael

…

On Feb 22, 2019, at 11:45 PM, Eric Fielding ***@***.***> wrote: I am now getting a different error when I try to run the Mogi example main model: models/mogi/examples] fielding% mogi altar: unable to find support for <fast> mode altar: file='/Users/fielding/Documents/progs/Caltech/AlTar-2.0/models/mogi/mogi/Mogi.py', line=146, function='initialize': firewall breached; aborting... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGCCyYewY_LV9eZ5868NYNsCTlgSk1Wjks5vQPGzgaJpZM4YraGz>.

[EJFielding]

I am still using the old "make" method to build AlTar 2.0.

I changed the mogi.pfg file to mode "native" and it is running successfully now.

[aivazis]

Ok. I’ll check why make didn’t build the extensions. You are on a mac, right? — Michael

…

On Feb 23, 2019, at 9:20 AM, Eric Fielding ***@***.***> wrote: I am still using the old "make" method to build AlTar 2.0. I changed the mogi.pfg file to mode "native" and it is running successfully now. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGCCyRI5rh8eTOOarpsiKZ7SwznLlnCCks5vQXhGgaJpZM4YraGz>.

[EJFielding]

Yes, MacBook with MacPorts.

[EJFielding]

I have updated my Pyre and AlTar installations with git pull. They seem to install successfully. Now when I try to run "mogi" in the examples directory, I get a new error:
Traceback (most recent call last): File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/builds/bin/mogi", line 32, in <module> status = app.run() File "/Users/fielding/Documents/progs/Caltech/extra_aivazis/pyre/products/packages/pyre/shells/Application.py", line 197, in run return self.shell.launch(self, *args, **kwds) File "/Users/fielding/Documents/progs/Caltech/extra_aivazis/pyre/products/packages/pyre/shells/Script.py", line 49, in launch status = application.main(*args, **kwds) File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/altar/shells/Application.py", line 53, in main self.controller.initialize(application=self) File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/altar/bayesian/Annealer.py", line 58, in initialize self.sampler.initialize(application=application) File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/altar/bayesian/Metropolis.py", line 54, in initialize self.uniform = altar.pdf.uniform_pos(support=(0,1), rng=rng) File "/Users/fielding/Documents/progs/Caltech/extra_aivazis/pyre/products/packages/gsl/pdf.py", line 116, in __init__ super().__init__(**kwds) TypeError: object.__init__() takes exactly one argument (the instance to initialize)

I get the same error whether I have the mode "native" or "fast" in the mogi.pfg file.

[lijun99]

HI Eric, The altar.pdf.uniform_pos(support=(0,1), rng=rng) doesn't need support. Please change it to self.uniform = altar.pdf.uniform_pos(rng=rng) in altar/bayesian/Metropolis.py. I'll ask Michael to update the repo. Best, Lijun

…

On Fri, May 3, 2019 at 4:25 PM Eric Fielding ***@***.***> wrote: I have updated my Pyre and AlTar installations with git pull. They seem to install successfully. Now when I try to run "mogi" in the examples directory, I get a new error: Traceback (most recent call last): File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/builds/bin/mogi", line 32, in <module> status = app.run() File "/Users/fielding/Documents/progs/Caltech/extra_aivazis/pyre/products/packages/pyre/shells/Application.py", line 197, in run return self.shell.launch(self, *args, **kwds) File "/Users/fielding/Documents/progs/Caltech/extra_aivazis/pyre/products/packages/pyre/shells/Script.py", line 49, in launch status = application.main(*args, **kwds) File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/altar/shells/Application.py", line 53, in main self.controller.initialize(application=self) File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/altar/bayesian/Annealer.py", line 58, in initialize self.sampler.initialize(application=application) File "/Users/fielding/Documents/progs/Caltech/AlTar-2.0/altar/altar/bayesian/Metropolis.py", line 54, in initialize self.uniform = altar.pdf.uniform_pos(support=(0,1), rng=rng) File "/Users/fielding/Documents/progs/Caltech/extra_aivazis/pyre/products/packages/gsl/pdf.py", line 116, in __init__ super().__init__(**kwds) TypeError: object.__init__() takes exactly one argument (the instance to initialize) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJHRSGP67KAVKV375NZNEK3PTTCX3ANCNFSM4GFNUGZQ> .

[EJFielding]

Thanks Lijun.

It runs after I made that change, but the results are all "nan".

altar: step
  β: 0
  θ: (4096 samples) x (6 parameters)
  Σ: 6 x 6
    [[   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00
         +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00
         +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00
         +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00
         +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00
         +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00   +0.0000e+00 ]]
 ** ** ** GSL error: endpoints do not enclose a minimum

altar: step
  β: 0.999999999850374
  θ: (4096 samples) x (6 parameters)
  Σ: 6 x 6
    [[          +nan          +nan          +nan          +nan          +nan          +nan
                +nan          +nan          +nan          +nan          +nan          +nan
                +nan          +nan          +nan          +nan          +nan          +nan
                +nan          +nan          +nan          +nan          +nan          +nan
                +nan          +nan          +nan          +nan          +nan          +nan
                +nan          +nan          +nan          +nan          +nan          +nan ]]

[lijun99]

Hi Eric and Michael, I have encountered the same problem. It seems to be the problem with mogi: if you run other models, such as linear, it should work well. I can take a look. Michael, could you please check it as well? Best, Lijun

…

On Fri, May 3, 2019 at 4:59 PM Eric Fielding ***@***.***> wrote: Thanks Lijun. It runs after I made that change, but the results are all "nan". altar: step β: 0 θ: (4096 samples) x (6 parameters) Σ: 6 x 6 [[ +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 ]] ** ** ** GSL error: endpoints do not enclose a minimum altar: step β: 0.999999999850374 θ: (4096 samples) x (6 parameters) Σ: 6 x 6 [[ +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan ]] — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJHRSGIJGZNPWDEAV4RETHDPTTGV7ANCNFSM4GFNUGZQ> .

[lijun99]

Hi Eric and Michael, I think the problem is, after using the right form of likelihood norm**2/2 (or larger negative values), the framework doesn't handle it well. To make it work, some of components need to be updated (or brought back from altar-1,1), 1. the dbeta solver with iterative grid-based searching method, instead of gsl solver; 2. the posterior needs to recomputed after beta update. and possibly, 3. condition the covariance matrix, which is not simply symmetrize the matrix, but makes some parameters less singular if they don't get updated. (may not to be critical if parameters are well defined). I think mogi presents an interesting example for testing altar framework: some parameters have large search ranges while others are small; the rate of invalid proposals are extremely high (> 99%) in the first few beta steps. Can someone give me a concrete example as well as the desired results so that I can make sure the framework works well, not converging to some local maximum. I also noticed that Grace made some changes in her pull request; are they relevant? These updates were already included in my cuda branch, https://github.com/lijun99/altar/tree/cuda. I will try to prepare a pull request version once I make sure the mogi results are correct. Best, Lijun

…

On Fri, May 3, 2019 at 5:29 PM Lijun Zhu ***@***.***> wrote: Hi Eric and Michael, I have encountered the same problem. It seems to be the problem with mogi: if you run other models, such as linear, it should work well. I can take a look. Michael, could you please check it as well? Best, Lijun On Fri, May 3, 2019 at 4:59 PM Eric Fielding ***@***.***> wrote: > Thanks Lijun. > > It runs after I made that change, but the results are all "nan". > > altar: step > > β: 0 > > θ: (4096 samples) x (6 parameters) > > Σ: 6 x 6 > > [[ +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 > > +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 > > +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 > > +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 > > +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 > > +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 +0.0000e+00 ]] > > ** ** ** GSL error: endpoints do not enclose a minimum > > > > altar: step > > β: 0.999999999850374 > > θ: (4096 samples) x (6 parameters) > > Σ: 6 x 6 > > [[ +nan +nan +nan +nan +nan +nan > > +nan +nan +nan +nan +nan +nan > > +nan +nan +nan +nan +nan +nan > > +nan +nan +nan +nan +nan +nan > > +nan +nan +nan +nan +nan +nan > > +nan +nan +nan +nan +nan +nan ]] > > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#7 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AJHRSGIJGZNPWDEAV4RETHDPTTGV7ANCNFSM4GFNUGZQ> > . >

[EJFielding]

Hi Lijun,
The input data for the Mogi model is generated by the program in the directory below, so you can easily check whether the inversion is getting the synthetic model correct.
Best,
++Eric

[lijun99]

(Updated the method to pull from another repo. -Lijun) git remote add develop https://github.com/lijun99/altar.git git fetch develop pr git checkout pr Hi Eric, Thanks! I used the generator but didn't pay attention to what's in there. It seems that dV = altar.properties.float(default=1.e10), not 10e10, judging from the search range (9,11) in mogi.pfg. Here are the results I got parameters (mean, sd): (0.006367653705854481, 3.9929836489301134) (0.07916527850719468, 4.548599168862911) (0.0009419295831507296, 0.056472701051412146) (7.252818172805752e-05, 0.05711650471564437) (10.000018077782386, 0.0011839893649379205) (3000.0094113483433, 6.044272715864086) which are positions x2, offsets x2, dv, depth. (The order of parameters may vary during simulation, since there is randomness how parameters are added to psets.) So it agrees with the sample generator. It turns out that conditioning the covariance matrix (to ensure positive definiteness) is bad for mogi, because the search ranges (or variances) for different parameters differ a lot. I disabled it for now; but we need to think about how to do it properly in the future. So, I have prepared a pull request version and submitted to github. If you or someone else are interested in testing the modified version before Michael merges it to master, you could pull my branch at first git remote add develop https://github.com/lijun99/altar.git git fetch develop pr git checkout pr Then make altar as usual. Best, Lijun

…

On Sun, May 5, 2019 at 5:34 PM Eric Fielding ***@***.***> wrote: Hi Lijun, The input data for the Mogi model is generated by the program in the directory below, so you can easily check whether the inversion is getting the synthetic model correct. Best, ++Eric — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJHRSGIG6XAXVLIQZ4I4ZH3PT54HDANCNFSM4GFNUGZQ> .

[EJFielding]

Lijun,

The mogi version in your "pr" branch works for me and gives good results, but only in the slow "native" mode.

@aivazis Are you still planning to fix the AlTar2 build system to work with the old "mm" and "make" or do we all need to switch to the new "mm"?