Issue with fges algorithm

[Zarmas]

Hello,

I am trying to make a causal inference study, and I am using the following command:

java -Xmx128G -jar causal-cmd-1.4.1-SNAPSHOT-jar-with-dependencies.jar --data-type continuous --delimiter tab --parallelized --json-graph --algorithm fges --dataset dataset.txt --score sem-bic-score --penaltyDiscount 10.0 --maxDegree 20

on a 14 thread and 128gb RAM system. The full dataset I enter are approximately 20000 continuous variables and 3000 samples.

The process gets killed after running for approximately 14 minutes. Even in smaller samples the process gets killed. From the top command it seems that memory is close to 100% before it stops running. On the log file, the last entry is the following:

Initializing effect edges: 21000

Tried dividing the set and the smallest data size it could run was 2000 variables and 100 samples. On the paper accompanying the caucal-cmd executable(Ramsey Et al. 2017), it is stated that it can be used for a million variables and more, but it seems impossible to use it on more than 2000 variables, on an above average system. What can I do in order to use it for the dataset I want to, with the system I currently use?

Thank you in advance,
George

[kvb2univpitt]

Hello. Can you please provide the log file? What is the error message?

[Zarmas]

There is no error message, the command line just prints "Killed" along with the command I used.

On the causal-cmd.log, the messages stop at the following:

2022-08-03 09:17:31 INFO edu.pitt.dbmi.causal.cmd.data.DataValidations:166 - Start data validation on file dataset.txt.
2022-08-03 09:17:50 INFO edu.pitt.dbmi.causal.cmd.data.DataValidations:166 - End data validation on file dataset.txt.
2022-08-03 09:17:50 INFO edu.pitt.dbmi.causal.cmd.data.DataValidations:109 - There are 3000 cases and 20000 variables.
2022-08-03 09:17:50 INFO edu.pitt.dbmi.causal.cmd.data.DataFiles:166 - Start reading in file dataset.txt.
2022-08-03 09:18:10 INFO edu.pitt.dbmi.causal.cmd.data.DataFiles:166 - Finished reading in file dataset.txt.
2022-08-03 09:18:10 INFO edu.pitt.dbmi.causal.cmd.data.DataFiles:166 - File dataset.txt contains 3000 cases, 20000 variables.

And on the fges_*.txt, the last messages is the following:
Start search: Wed, August 03, 2022 09:18:11 AM
Initializing effect edges: 1000
Initializing effect edges: 2000
Initializing effect edges: 3000
Initializing effect edges: 4000
Initializing effect edges: 5000

[jdramsey]

Is 'verbose' on? I.e. --verbose? No, I don't see it. I think it's off by default It's possible the algorithm is running at this point but not printing anything?

[jdramsey]

Also I discovered some issues in my previous pipeline for doing large models which I need to think about.

@kvb2univpitt Is --faithfulnessAssumed false by default? It would help if it were false here. (I think that's the flag).

[kvb2univpitt]

@jdramsey Sorry, I missed your question. Yes, faithfulnessAssumed is set to false by default.

[jdramsey]

@kvb2univpitt @Zarmas Hmmm...... OK I need to think more. BTW the laptop I'm currently using is just 16 GB so it's a little hard for me to comment well, but I'm hopefully getting a 64 GB laptop soon and I'll try it when I get it. Also there's another different sort of algorithm I just thought of yesterday that I'd like to try on the larger laptop that would definitely benefit from parallelization. It's based on this paper that we just got published in UAI 2022:

https://arxiv.org/abs/2206.05421

though it does assume that you're interested in a particular variable (or list of variables). Would that happen to be true in your case, @Zarmas ? Or did you need causal structure for all 20,000 variables?

[Zarmas]

I am need the causal structure for all the variables, or to run it individually for all 20000 variables, if it can run fast.

[jdramsey]

Interestingly, just to note, I recently went back to look at old algorithm of mine from years ago called MBFS; I just renamed it to PC-MB for clarify. It's very fast for finding the Markov blanket or causes and effects about a variables. In fact, if the model is not too dense it's very competitive.

If you don't mind my being curious, what sort of application do you have where you actually need to know the causal structure over all 20,000 nodes? I can't currently think of one, which is why I've been going back to thinking of these partial algorithms the last few days. But maybe you convince me that it's worthwhile finding the whole causal structure so that i will be motivated to work on the problem for you.

[jdramsey]

(I ask this despite having spent a lot of time trying to find the causal structure over many nodes like that.)

[jdramsey]

Actually strategies for doing 20000 genes individually in parallel are occurring to me, but if I'm going to do this work for you you should send me emails so we can work out the details. I'm thinking if you have a big machine it might be doable, using a couple of strategies.

[jdramsey]

@Zarmas I take it this is a stale issue, so I'll close it. If you're still interested, you can re-open the issue.

[jdramsey]

@Zarmas Another comment--what I realized was that this was a problem specifically for running parallelized FGES in a context where more than one copy of FGES were also being run in parallel--something we hadn't previously tried to do. But it came up again recently, so then I looked back at this issue and realized it was the same issue. :-D

[Zarmas]

Hello again, I tried using the fges algorithm in the new version, in a dataset that could previously run in the 1.4.1 version with no problems. The command I used was the following:

java -Xmx24G -jar causal-cmd-1.7.0-SNAPSHOT-jar-with-dependencies.jar --data-type continuous --delimiter tab --parallelized --json-graph --algorithm fges --dataset dataset.txt --score sem-bic-score --penaltyDiscount 10.0 --maxDegree 20

And this time I get a new error. I wonder why a dataset that was small enough to be processed in the old version can no longer be supported. The error message I got was the following:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
at java.base/java.lang.StringBuffer.append(StringBuffer.java:317)
at java.base/java.io.StringWriter.write(StringWriter.java:106)
at com.google.gson.stream.JsonWriter.newline(JsonWriter.java:654)
at com.google.gson.stream.JsonWriter.beforeName(JsonWriter.java:669)
at com.google.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:401)
at com.google.gson.stream.JsonWriter.value(JsonWriter.java:417)
at com.google.gson.internal.bind.TypeAdapters$EnumTypeAdapter.write(TypeAdapters.java:916)
at com.google.gson.internal.bind.TypeAdapters$EnumTypeAdapter.write(TypeAdapters.java:859)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:70)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:196)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:368)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:70)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:196)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:368)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:70)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:97)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:61)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:70)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:207)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:144)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:70)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:196)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:368)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:70)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:196)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:368)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:70)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:196)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:368)