Issue with algorithm fges-mb

[Zarmas]

Hello,

I tried using the fges-mb algorithm, in an attempt to work around the issue I had when running the fges algorithm. The system I use has 14 threads and 128gb RAM. The data I enter are approximately 20000 continuous variables and 3000 samples. One of the commands I used was the following:

java -Xmx128G -jar ~/bin/causal-cmd/causal-cmd-1.4.1-SNAPSHOT-jar-with-dependencies.jar --data-type continuous --default --delimiter tab --json-graph --algorithm fges-mb --dataset dataset.txt --target targetname --score sem-bic-score --penaltyDiscount 10.0 --maxDegree 100

As with fges, the largest data size it could handle was 2000 variables with 100 samples, printing this additional message: "heuristicSpeedup = false".
The --parallelized option was not available for this algorithm, but it could run parallel when using the --default switch. The issue with that was that I could not change the other options for the algorithm, even if I specified them in the command (e.g. in the log it is "verbose: yes" even if I specify "--verbose no" in the command). Is there a way to get parallel execution while being able to choose the other algorithm options?

After trying to run the fges-mb algorithm with a larger data size I got errors. When used a 2000 variable-2000sample dataset, I got "Exception in thread "main" java.util.ConcurrentModificationException".
When I used an even larger dataset(i.e. 10000variables-3000samples) I got the following error line: Exception in thread "main" java.lang.NullPointerException. After displaying the errors, the executable exits.

I attach the full error messages to this ticket. I tried running the fges-mb algorithm with multiple different variables as target. Is there a way to get around these errors?

One other issue I had was when I was using the fges-mb algorithm on the whole dataset, it took too long to even display the message, approximately one hour, meaning it takes too long and repeating it for every variable will take years.

All the tests were done on the same 14 thread and 128gb RAM system, but the errors on the fges-mb algorithm were also presented on a different system with less ram and fewer threads. On the paper accompanying the caucal-cmd executable(Ramsey Et al. 2017), it is stated that it can be used for a million variables and more, but it seems impossible to use it on more than 2000 variables, on an above average system.

Thank you in advance,
George
ConcurrentModificationException.txt
NullPointerException.txt

[jdramsey]

Hmmm... the paper you cite was for an extremely sparse model with average degree 2; for denser problems it will be slower. But can I ask right away which version of causal-cmd you are using? The latest one is in Maven Central, here: https://s01.oss.sonatype.org/content/repositories/releases/io/github/cmu-phil/causal-cmd/1.4.1/

…

On Mon, Aug 1, 2022 at 9:20 AM gZarmalis ***@***.***> wrote: Hello, I tried using the fges-mb algorithm, in an attempt to work around the issue I had when running the fges algorithm. The system I use has 14 threads and 128gb RAM. The data I enter are approximately 20000 continuous variables and 3000 samples. One of the commands I used was the following: java -Xmx128G -jar ~/bin/causal-cmd/causal-cmd-1.4.1-SNAPSHOT-jar-with-dependencies.jar --data-type continuous --default --delimiter tab --json-graph --algorithm fges-mb --dataset dataset.txt --target targetname --score sem-bic-score --penaltyDiscount 10.0 --maxDegree 100 As with fges, the largest data size it could handle was 2000 variables with 100 samples, printing this additional message: "heuristicSpeedup = false". The --parallelized option was not available for this algorithm, but it could run parallel when using the --default switch. The issue with that was that I could not change the other options for the algorithm, even if I specified them in the command (e.g. in the log it is "verbose: yes" even if I specify "--verbose no" in the command). Is there a way to get parallel execution while being able to choose the other algorithm options? After trying to run the fges-mb algorithm with a larger data size I got errors. When used a 2000 variable-2000sample dataset, I got "Exception in thread "main" java.util.ConcurrentModificationException". When I used an even larger dataset(i.e. 10000variables-3000samples) I got the following error line: Exception in thread "main" java.lang.NullPointerException. After displaying the errors, the executable exits. I attach the full error messages to this ticket. I tried running the fges-mb algorithm with multiple different variables as target. Is there a way to get around these errors? One other issue I had was when I was using the fges-mb algorithm on the whole dataset, it took too long to even display the message, approximately one hour, meaning it takes too long and repeating it for every variable will take years. All the tests were done on the same 14 thread and 128gb RAM system, but the errors on the fges-mb algorithm were also presented on a different system with less ram and fewer threads. On the paper accompanying the caucal-cmd executable(Ramsey Et al. 2017), it is stated that it can be used for a million variables and more, but it seems impossible to use it on more than 2000 variables, on an above average system. Thank you in advance, George ConcurrentModificationException.txt <https://github.com/bd2kccd/causal-cmd/files/9234203/ConcurrentModificationException.txt> NullPointerException.txt <https://github.com/bd2kccd/causal-cmd/files/9234204/NullPointerException.txt> — Reply to this email directly, view it on GitHub <#74>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACLFSRYQ6MQMJUYTDW4CGH3VW7FLNANCNFSM55HRKRFA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jdramsey]

Also, have you increased the heap size for Java by including a flag like -Xmx10g for e.g. 10 gigs? On Mon, Aug 1, 2022 at 12:35 PM Joseph Ramsey ***@***.***> wrote:

…

Hmmm... the paper you cite was for an extremely sparse model with average degree 2; for denser problems it will be slower. But can I ask right away which version of causal-cmd you are using? The latest one is in Maven Central, here: https://s01.oss.sonatype.org/content/repositories/releases/io/github/cmu-phil/causal-cmd/1.4.1/ On Mon, Aug 1, 2022 at 9:20 AM gZarmalis ***@***.***> wrote: > Hello, > > I tried using the fges-mb algorithm, in an attempt to work around the > issue I had when running the fges algorithm. The system I use has 14 > threads and 128gb RAM. The data I enter are approximately 20000 continuous > variables and 3000 samples. One of the commands I used was the following: > > java -Xmx128G -jar > ~/bin/causal-cmd/causal-cmd-1.4.1-SNAPSHOT-jar-with-dependencies.jar > --data-type continuous --default --delimiter tab --json-graph --algorithm > fges-mb --dataset dataset.txt --target targetname --score sem-bic-score > --penaltyDiscount 10.0 --maxDegree 100 > > As with fges, the largest data size it could handle was 2000 variables > with 100 samples, printing this additional message: "heuristicSpeedup = > false". > The --parallelized option was not available for this algorithm, but it > could run parallel when using the --default switch. The issue with that was > that I could not change the other options for the algorithm, even if I > specified them in the command (e.g. in the log it is "verbose: yes" even if > I specify "--verbose no" in the command). Is there a way to get parallel > execution while being able to choose the other algorithm options? > > After trying to run the fges-mb algorithm with a larger data size I got > errors. When used a 2000 variable-2000sample dataset, I got "Exception in > thread "main" java.util.ConcurrentModificationException". > When I used an even larger dataset(i.e. 10000variables-3000samples) I got > the following error line: Exception in thread "main" > java.lang.NullPointerException. After displaying the errors, the executable > exits. > > I attach the full error messages to this ticket. I tried running the > fges-mb algorithm with multiple different variables as target. Is there a > way to get around these errors? > > One other issue I had was when I was using the fges-mb algorithm on the > whole dataset, it took too long to even display the message, approximately > one hour, meaning it takes too long and repeating it for every variable > will take years. > > All the tests were done on the same 14 thread and 128gb RAM system, but > the errors on the fges-mb algorithm were also presented on a different > system with less ram and fewer threads. On the paper accompanying the > caucal-cmd executable(Ramsey Et al. 2017), it is stated that it can be used > for a million variables and more, but it seems impossible to use it on more > than 2000 variables, on an above average system. > > Thank you in advance, > George > ConcurrentModificationException.txt > <https://github.com/bd2kccd/causal-cmd/files/9234203/ConcurrentModificationException.txt> > NullPointerException.txt > <https://github.com/bd2kccd/causal-cmd/files/9234204/NullPointerException.txt> > > — > Reply to this email directly, view it on GitHub > <#74>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACLFSRYQ6MQMJUYTDW4CGH3VW7FLNANCNFSM55HRKRFA> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >

[Zarmas]

Yes, I am using the latest version and I have increased the heap size. You can see the entire command I use on the first message of the post.

[jdramsey]

Hmmmm.... 20,000 variables should not be a problem, unless the data loader can't load it (@kvb2univpitt ?). The execution time may be due to the density of the graph. Let me try a 20,000 variable problem with N = 3000 in the Tetrad GUI...

[jdramsey]

Wait, is this continuous or discrete or mixed?

[Zarmas]

Continuous.

[jdramsey]

Oh wow, this is breaking the GUI. Hmm.. let me think how I did that before for the million variable problem. I was using different data structures at the time, a different simulator ('large scale simulator", which we still have), and a different covariance matrix class (which calculated covariances on the fly instead of storing them in a covariance matrix). The data loading is @kvb2univpitt's, so hopefully he can comment on that. I suppose I could give you a version that uses the covariance matrix on the fly. For 3000 samples it might be a bit slow but it should finish for you, depending on the density of your true model.

[jdramsey]

I'm attending UAI virtually in the Netherlands since 3 AM this morning so am a bit sleep deprived, but if I can catch some sleep maybe I could work on this for you--that is, the parts that I can.

[jdramsey]

Wait, you should be able to tell if the data loading is successful. Is it?

[Zarmas]

This is the last message on the .txt:
Start search: Tue, August 02, 2022 02:28:44 PM

[kvb2univpitt]

Sorry for joining in the conversation late. If the search has started, that means the data has already been read in. So, it seems like the data loading part is fine. Based on the attached error files, it seems to break when running FGES. The reason why you're getting ConcurrentModificationException is because the data structure, HashMap in this case, is being modified by one thread and being read by another thread. @jdramsey It seems like parallelization might not be working correctly.

[jdramsey]

@Zarmas Sorry just coming back to looking at these causal-cmd issues now. I will try this again tomorrow.

[jdramsey]

@Zarmas Actually we were having some concurrency problems with FGES in another project and I think I just fixed them there. I'm thinking I just fixed them for your application as well. I looked at your error messages and think I've addressed those problem. (Due to concurrency, some nodes were being removed and then "removed" again, causing issues.)

We're going to do a new release soon of Tetrad and causal-cmd; hopefully these problems will go away.

[Zarmas]

@jdramsey Hello again, I am still getting the "ConcurrentModificationException" error message when using the fges-mb algorithm. The full error message is shown in the original message of this post. I have also encountered a problem when using the fges algorithm and posted it on a closed relative to fges issue, but can't re-open it since I wasn't the one that closed it.

[jdramsey]

Huh, this method that's throwing the exception is in line 408 in the current version 7.2.2 code, not line 647 as in the exception message--do you know which version you're using currently? public boolean isAdjacentTo(Node node1, Node node2) { if (node1 == null || node2 == null || this.edgeLists.get(node1) == null || this.edgeLists.get(node2) == null) { return false; } // Trying to fix a concurency problem. for (Edge edge : this.edgeLists.get(node1)) { if (Edges.traverse(node1, edge) == node2 && Edges.traverse(node2, edge) == node1) { return true; } } return false; }

…

On Wed, Mar 1, 2023 at 2:03 PM gZarmalis ***@***.***> wrote: @jdramsey <https://github.com/jdramsey> Hello again, I am still getting the "ConcurrentModificationException" error message when using the fges-mb algorithm. The full error message is shown in the original message of this post. I have also encountered a problem when using the fges algorithm and posted it on a closed relative to fges issue, but can't re-open it since I wasn't the one that closed it. — Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACLFSR72V2Z7BF7LN2MIYXDWZ6MOJANCNFSM55HRKRFA> . You are receiving this because you were mentioned.Message ID: ***@***.***>