Obscure error in hunt driver with solid coupling

[guillermogf]

Hi,

I've detected an error when running the hunt driver with solid coupling (tw != 0).

The error cause has proved difficult to isolate as small variations in the model seem to suppress it or trigger it. Most notably, running the same problem setup with a different number of parallel processes can trigger it. The error has been observed at different mesh sizes, Hartmann numbers and tw values. It has been reproduced not only with the mpi backend and petsc solver but also with the julia solver and sequential backend (as reported by @pmartorell).

Here is a running script with which I can reproduce the issue:

using SparseMatricesCSR
using GridapPETSc

solver = Dict(
    :solver => :petsc,
    :matrix_type    => SparseMatrixCSR{0,PetscScalar,PetscInt},
    :vector_type    => Vector{PetscScalar},
    :petsc_options  => "-snes_monitor -ksp_error_if_not_converged true \
                         -ksp_converged_reason -ksp_type preonly -pc_type lu \
                         -pc_factor_mat_solver_type mumps -mat_mumps_icntl_28 1 \
                         -mat_mumps_icntl_29 2 -mat_mumps_icntl_4 3 \
                         -mat_mumps_cntl_1 0.001",
    :niter          => 100,
    :rtol           => 1e-5,
)

hunt(;
  title="mwe",
  backend=:mpi,
  np=(4,4),
  nc=(20,20),
  B=(0.0,50,0.0),
  L=1.0,
  σw1=0.0001,
  σw2=1000.0,
  tw=0.5,
  nsums=100,
  vtk=true,
  debug=false,
  BL_adapted=true,
  solver=solver,
  path="../data/"
  )

Notably either changing np=(4,4) to np=(2,2), or tw=0.5 to tw=0.1 suppresses the error. Toggling the BL_adapted flag also seems to affect whether the error is triggered.

Here follows the abriged output produced. Please find here the full output file. It's too long and a bit messy as it collects the error from several of the 16 mpi processes.

Abriged output

[...]
[7.467e+00 s in MAIN] pre_process
[3.753e+01 s in MAIN] fe_spaces
[2.904e+00 s in MAIN] solver_setup
[...]
┌ Error: 
│   exception =
│    AssertionError: A check failed
│    Stacktrace:
│      [1] macro expansion
│        @ ~/.julia/packages/Gridap/Ort08/src/Helpers/Macros.jl:61 [inlined]
│      [2] evaluate!(cache::Gridap.Arrays.CachedArray{Float64, 3, Array{Float64, 3}}, f::Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, a::Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, b::Array{Gridap.TensorValues.VectorValue{3, Float64}, 3})
│        @ Gridap.Fields ~/.julia/packages/Gridap/Ort08/src/Fields/FieldArrays.jl:539
│      [3] evaluate!(cache::Tuple{Gridap.Fields.MatrixBlock{Array{Float64, 3}}, Matrix{Gridap.Arrays.CachedArray{Float64, 3, Array{Float64, 3}}}}, k::Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, f::Gridap.Fields.VectorBlock{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}}, g::Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}})
│        @ Gridap.Fields ~/.julia/packages/Gridap/Ort08/src/Fields/ArrayBlocks.jl:904
│      [4] evaluate
│        @ ~/.julia/packages/Gridap/Ort08/src/Arrays/Maps.jl:87 [inlined]
│      [5] return_value
│        @ ~/.julia/packages/Gridap/Ort08/src/Fields/ArrayBlocks.jl:779 [inlined]
│      [6] return_type
│        @ ~/.julia/packages/Gridap/Ort08/src/Arrays/Maps.jl:62 [inlined]
│      [7] lazy_map(::Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, ::FillArrays.Fill{Gridap.Fields.VectorBlock{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, ::Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(+)}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(GridapMHD.conv)}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.PosNegReindex{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, Vector{Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Float64}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Arrays.Table{Int32, Vector{Int32}, Vector{Int32}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.TransposeMap, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Gridap.Fields.TransposeFieldIndices{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BlockMap{1}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.VectorBlock{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, 1, Tuple{Base.OneTo{Int64}}}, Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{typeof(Gridap.Fields.pinvJt)}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Base.OneTo{Int64}}}}}}}}}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(GridapMHD.conv)}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}}, 1, Tuple{FillArrays.Fill{Gridap.Fields.MatrixBlock{Gridap.Fields.TransposeFieldIndices{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, Gridap.TensorValues.VectorValue{3, Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{typeof(Gridap.Fields.pinvJt)}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}}}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.PosNegReindex{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, Vector{Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Float64}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Arrays.Table{Int32, Vector{Int32}, Vector{Int32}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Base.OneTo{Int64}}}}}}}}}}})
│        @ Gridap.Arrays ~/.julia/packages/Gridap/Ort08/src/Arrays/LazyArrays.jl:57
│      [8] evaluate!(cache::Nothing, f::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, x::Gridap.CellData.CellPoint{Gridap.CellData.ReferenceDomain, FillArrays.Fill{Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}, FillArrays.Fill{Matrix{Float64}, 1, Tuple{Base.OneTo{Int64}}}}}, Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}})
│        @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/CellFields.jl:539
│      [9] evaluate
│        @ ~/.julia/packages/Gridap/Ort08/src/Arrays/Maps.jl:87 [inlined]
│     [10] (::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain})(x::Gridap.CellData.CellPoint{Gridap.CellData.ReferenceDomain, FillArrays.Fill{Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}, FillArrays.Fill{Matrix{Float64}, 1, Tuple{Base.OneTo{Int64}}}}}, Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}})
│        @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/CellFields.jl:388
│     [11] integrate(f::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, quad::Gridap.CellData.CellQuadrature{Gridap.CellData.ReferenceDomain, Gridap.CellData.PhysicalDomain})
│        @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/CellQuadratures.jl:154
│     [12] integrate(f::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, b::Gridap.CellData.GenericMeasure)
│        @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/DomainContributions.jl:149
│     [13] #296
│        @ ~/.julia/packages/GridapDistributed/VR0ki/src/CellData.jl:234 [inlined]
│     [14] map(::GridapDistributed.var"#296#297", ::PartitionedArrays.MPIArray{Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, 1}, ::PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1})
│        @ PartitionedArrays ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:229
│     [15] integrate(f::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, b::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}})
│        @ GridapDistributed ~/.julia/packages/GridapDistributed/VR0ki/src/CellData.jl:233
│     [16] *(a::Gridap.CellData.Integrand, b::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}})
│        @ GridapDistributed ~/.julia/packages/GridapDistributed/VR0ki/src/CellData.jl:254
│     [17] dc_mhd_u_u(u::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.FESpaces.SingleFieldFEFunction{Gridap.CellData.GenericCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, PartitionedArrays.MPIArray{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, du::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, v_u::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, α::Float64, dΩ::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}})
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:291
│     [18] dc_mhd(x::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, dx::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, dy::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, α::Float64, dΩ::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}})
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:289
│     [19] (::GridapMHD.var"#dc#38"{Float64, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}})(x::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, dx::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, dy::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing})
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:61
│     [20] (::GridapMHD.var"#jac#40"{GridapMHD.var"#dc#38"{Float64, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}}, GridapMHD.var"#a#35"{Vector{Any}, Vector{Any}, Vector{Any}, GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.CellData.GenericCellField{Gridap.CellData.ReferenceDomain}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Arrays.IdentityVector{Int64}}, 1, Tuple{Base.OneTo{Int64}}}, Int64, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}, Dict{Symbol, Any}, Float64, Gridap.TensorValues.VectorValue{3, Float64}, Float64, Float64, Float64, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}}, Dict{Symbol, Any}})(x::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, dx::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, dy::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing})
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:75
│     [21] residual_and_jacobian(op::Gridap.FESpaces.FEOperatorFromWeakForm, uh::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}})
│        @ Gridap.FESpaces ~/.julia/packages/Gridap/Ort08/src/FESpaces/FEOperatorsFromWeakForm.jl:100
│     [22] residual_and_jacobian(op::Gridap.FESpaces.AlgebraicOpFromFEOp, x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64})
│        @ Gridap.FESpaces ~/.julia/packages/Gridap/Ort08/src/FESpaces/FEOperators.jl:194
│     [23] _setup_cache(x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}, nls::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.AlgebraicOpFromFEOp)
│        @ GridapPETSc ~/.julia/packages/GridapPETSc/MNuuG/src/PETScNonlinearSolvers.jl:129
│     [24] solve!(x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}, nls::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.AlgebraicOpFromFEOp, #unused#::Nothing)
│        @ GridapPETSc ~/.julia/packages/GridapPETSc/MNuuG/src/PETScNonlinearSolvers.jl:168
│     [25] solve!(x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}, nls::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.AlgebraicOpFromFEOp)
│        @ Gridap.Algebra ~/.julia/packages/Gridap/Ort08/src/Algebra/NonlinearSolvers.jl:22
│     [26] solve!(u::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, solver::Gridap.FESpaces.NonlinearFESolver, feop::Gridap.FESpaces.FEOperatorFromWeakForm, cache::Nothing)
│        @ Gridap.FESpaces ~/.julia/packages/Gridap/Ort08/src/FESpaces/FESolvers.jl:163
│     [27] solve!
│        @ ~/.julia/packages/Gridap/Ort08/src/FESpaces/FESolvers.jl:14 [inlined]
│     [28] solve!
│        @ ~/.julia/packages/Gridap/Ort08/src/FESpaces/FESolvers.jl:18 [inlined]
│     [29] _solve
│        @ ~/software/GridapMHD/GridapMHD.jl/src/Main.jl:329 [inlined]
│     [30] _solve(xh::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, solver::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.FEOperatorFromWeakForm, params::Dict{Symbol, Any})
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Main.jl:327
│     [31] main(_params::Dict{Symbol, Any}; output::Dict{Symbol, Any})
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Main.jl:145
│     [32] (::GridapMHD.var"#65#66"{Dict{Symbol, Any}})()
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:161
│     [33] with(f::GridapMHD.var"#65#66"{Dict{Symbol, Any}}; kwargs::Base.Pairs{Symbol, Vector{SubString{String}}, Tuple{Symbol}, NamedTuple{(:args,), Tuple{Vector{SubString{String}}}}})
│        @ GridapPETSc ~/.julia/packages/GridapPETSc/MNuuG/src/Environment.jl:38
│     [34] _hunt(; distribute::PartitionedArrays.var"#88#89"{MPI.Comm, Bool}, rank_partition::Tuple{Int64, Int64}, nc::Tuple{Int64, Int64}, ν::Float64, ρ::Float64, σ::Float64, B::Tuple{Float64, Int64, Float64}, f::Tuple{Float64, Float64, Float64}, ζ::Float64, L::Float64, u0::Float64, B0::Float64, σw1::Float64, σw2::Float64, tw::Float64, nsums::Int64, vtk::Bool, title::String, path::String, debug::Bool, res_assemble::Bool, jac_assemble::Bool, solve::Bool, solver::Dict{Symbol, Any}, verbose::Bool, BL_adapted::Bool, kmap_x::Int64, kmap_y::Int64, ranks_per_level::Nothing)
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:160
│     [35] _hunt
│        @ ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:41 [inlined]
│     [36] #59
│        @ ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:23 [inlined]
│     [37] with_mpi(f::GridapMHD.var"#59#62"{Tuple{Int64, Int64}, String, Base.Pairs{Symbol, Any, NTuple{11, Symbol}, NamedTuple{(:nc, :B, :L, :σw1, :σw2, :tw, :nsums, :vtk, :debug, :BL_adapted, :solver), Tuple{Tuple{Int64, Int64}, Tuple{Float64, Int64, Float64}, Float64, Float64, Float64, Float64, Int64, Bool, Bool, Bool, Dict{Symbol, Any}}}}, String}; comm::MPI.Comm, duplicate_comm::Bool)
│        @ PartitionedArrays ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:73
│     [38] with_mpi
│        @ ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:64 [inlined]
│     [39] hunt(; backend::Symbol, np::Tuple{Int64, Int64}, title::String, nruns::Int64, path::String, kwargs::Base.Pairs{Symbol, Any, NTuple{11, Symbol}, NamedTuple{(:nc, :B, :L, :σw1, :σw2, :tw, :nsums, :vtk, :debug, :BL_adapted, :solver), Tuple{Tuple{Int64, Int64}, Tuple{Float64, Int64, Float64}, Float64, Float64, Float64, Float64, Int64, Bool, Bool, Bool, Dict{Symbol, Any}}}})
│        @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:22
│     [40] top-level scope
│        @ none:19
│     [41] eval
│        @ ./boot.jl:370 [inlined]
│     [42] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:280
│     [43] _start()
│        @ Base ./client.jl:522
└ @ PartitionedArrays ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:75
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
[...]

Environment used:

• julia v1.9.4.
• Current GridapMHD.jl master branch (commit cf73820).
• OpenMPI v4.1.5.
• PETSc v3.15.4.

I will report back should I find simpler or further examples to reproduce the issue.

[pmartorell]

Thanks for reporting @guillermogf ,

you forgot to mention that the error did not reproduce in 9ea6729 for the following configuration. However, this example is too large for debugging.

using GridapMHD: hunt
using SparseMatricesCSR
using GridapPETSc

solver = Dict(
    :solver => :petsc,
    :matrix_type    => SparseMatrixCSR{0,PetscScalar,PetscInt},
    :vector_type    => Vector{PetscScalar},
    :petsc_options  => "-snes_monitor -ksp_error_if_not_converged true \
                         -ksp_converged_reason -ksp_type preonly -pc_type lu \
                         -pc_factor_mat_solver_type mumps -mat_mumps_icntl_28 1 \
                         -mat_mumps_icntl_29 2 -mat_mumps_icntl_4 3 \
                         -mat_mumps_cntl_1 0.001",
    :niter          => 100,
    :rtol           => 1e-5,
)

hunt(;
  title="hunt_solidcoupling_Ha1000_120x120",
  backend=:mpi,
  np=(4,4),
  nc=(120,120),
  B=(0.0,1000,0.0),
  L=1.0,
  σw1=0.0001,
  σw2=1000.0,
  tw=0.1,
  nsums=100,
  vtk=true,
  debug=false,
  BL_adapted=true,
  solver=solver,
  path="../data/"
  )

[pmartorell]

FYI @principejavier