sinkhorn_knopp_largedata should return dense matrix

[hsuknowledge]

When I tried ConstructDiffusionOperators() with 45k cells, first I encountered this error.

Error in diag(P) : no method for coercing this S4 class to a vector

From the source code, we see that base::diag() is called on a Matrix object which is returned by Doubly_stochastic() function in case matrix columns exceed 10k. If it was < 10k, it returns a dense matrix, otherwise it returns a sparse Matrix produced by sinkhorn_knopp_largedata().

LocalizedMarkerDetector/R/matrix_operation.R

Line 151 in c1f4c0d

sinkhorn_knopp_largedata = function(A, niter = 100, tol = 1e-8, verb = FALSE) {

At first I thought specifying Matrix::diag() to fix the error solves the problem, but then I found that the self-multiplication P <- P %*% P of the sparse Matrix takes too long after just 2 iterations. Instead, I think P should always be a dense matrix, so that its multiplication takes a fixed amount of time while taking advantage of OpenBLAS multi-threading.

Update:

I think I found another bug. In the tutorial, the next function fast_get_lmd() cannot take rho as a sparse Matrix. You need to coerce it to a dense matrix, otherwise this error happens.

Error: Not compatible with requested type: [type=S4; target=double].

[hsuknowledge]

Dense matrices consume insane amounts of memory, though. Since ConstructDiffusionOperators() saves every iteration in memory, and by tutorial I set max_time = 2^20 = 20 iterations, it blows up very quickly. My P is 45k x 45k, and at like iteration 10 or so, my main memory of 256GB is all consumed, and it's now setting aside several GBs into the swap (Optane 900P).

I think for caching intermediate steps you can convert to sparse Matrix first.

LocalizedMarkerDetector/R/construct_cell_graph.R

Line 328 in c1f4c0d

P_ls = c(P_ls,list(P))

P_ls = c(P_ls,list(as(P, "CsparseMatrix")))

Update:

Finished 20 iterations, now the object.size() of my P_ls is 229GB. Judging from the VisualizeDiffusion() plots, I find diffusion times greater than 2^6 to "diffuse" too broad for my dataset, and could be of little to no help I think. I didn't need to run that too deep.