[Issue]: Issues with Rocm/flang using 'reduction' with an array

[Yan-44]

Problem Description

Hi,
I am new on Rocm/flang github and I hope this is the correct place to post Rocm/flang issues. The simple code below reproduces issues that I experience with a very large Fortran code that I have developed when attempting to offload it:

1. Compilation (/opt/rocm-6.1.0/llvm/bin/flang -O2 -fopenmp test31b_reduction.f90 --offload-arch=gfx1030 -o test31b) yields the following error message:

   F90-F-0000-Internal compiler error. gen_sptr(): unexpected storage type 6 (test31b_reduction.f90: 23)

2. However, multi-threaded compilation (/opt/rocm-6.1.0/llvm/bin/flang -O2 -fopenmp test31b_reduction.f90 -o test31b) and execution work fine:

   toto(1) = 1000., correct answer is: toto(1) = 1000
   tata = 5.000000000000284 , correct answer is: tata = 5.000000000000284

3. Furthermore, upon commenting any occurrence of 'reduction(max: tata)', compilation (/opt/rocm-6.1.0/llvm/bin/flang -O2 -fopenmp test31b_reduction.f90 --offload-arch=gfx1030 -o test31b) works fine but execution yields the following error:

   0: ALLOCATE: array already allocated
   

4. Last, as expected, when commenting any occurrence of 'reduction', compilation (/opt/rocm-6.1.0/llvm/bin/flang -O2 -fopenmp test31b_reduction.f90 --offload-arch=gfx1030 -o test31b) works fine but execution yields erroneous results:

   toto(1) = 18., correct answer is: toto(1) = 1000
   tata = 6.3291139240506278E-002 , correct answer is: tata = 5.000000000000284

Could anyone help me fix issue 1?

Thanks for your help.

  Program test31b_reduction
  implicit none

  integer, parameter :: N=1000
  integer :: i
  double precision,dimension(:),allocatable :: toto
  double precision :: tata

  allocate(toto(2))
  toto=0._8
  tata=0._8  

  !$OMP TARGET map(toto,tata)
  !$OMP TEAMS reduction(+: toto) reduction(max: tata)
  !$OMP DISTRIBUTE PARALLEL DO reduction(+: toto) reduction(max: tata)
  do i = 1, N
     toto(1)=toto(1)+1._8
     if (tata<abs(1._8/(128.2_8-real(i,8)))) tata=abs(1._8/(128.2_8-real(i,8)))
  end do
  !$OMP END DISTRIBUTE PARALLEL DO
  !$OMP END TEAMS
  !$OMP END TARGET
  
  print*,'toto(1) = ',toto(1),', correct answer is: toto(1) = ',N
  print*,'tata = ',tata, ', correct answer is: tata = ',abs(1._8/(128.2_8-real(128,8)))
  deallocate(toto)

  end program test31b_reduction

Operating System

Ubuntu 20.04.6 LTS (Focal Fossa)

CPU

Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz

GPU

AMD Radeon PRO W6800 (amdgcn-amd-amdhsa--gfx1030)

ROCm Version

ROCm 6.1.0

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[ronlieb]

seems to work with rocm 6.3

[Yan-44]

Hi,
Thanks, @ronlieb for your answer.
I have checked rocm 6.3 prerequisites and observed that Ubuntu 20.04.6 LTS (Focal Fossa) was not among the supported operational systems. Unfortunately, the decision to upgrade the workstation I am using to (at least) Ubuntu 22.04.5 lies with the computer service of my lab, and for a number of reasons this will take days/weeks.
Hence, I won't be able to confirm rapidly that my code works well with Rocm 6.3. Shall I close the issue or is it OK if it remains open?
Best regards

[ronlieb]

your choice, leave it open if you wish.

[Yan-44]

Hi again @ronlieb (and others who might be interested by this issue),

My workstation having been upgraded to Ubuntu 24.04.2 LTS (Noble Numbat), I have installed Rocm-6.3.2 following the instructions stated at https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html.

Then I have checked with a simple program that offloading to my RADEON PRO W6800 was operational, and it was fine (simple program running with 360 teams and 256 threads as expected).

Finally, I have tried to compile the 'Program test31b_reduction' (see above) with the following instructions :
/opt/rocm-6.3.2/llvm/bin/flang test31b_reduction.f90 -O2 -fopenmp --offload-arch=gfx1030 -o test31b

and the same error message as before popped up at compile time:
F90-F-0000-Internal compiler error. gen_sptr(): unexpected storage type 6 (test31b_reduction.f90: 23)
F90/x86-64 Flang - 1.5 2017-05-01: compilation aborted

@ronlieb , can you clarify the system configuration and Rocm 6.3 version you used when you observed that the compilation worked on January 8th? Any hint to sort this issue out? Should I move this issue to the main forum (https://github.com/flang-compiler/flang)?

Thanks for your kind answers.

[ppanchad-amd]

Hi @Yan-44. Can you please check if your issue has been resolved with the latest ROCm 6.4.1? Thanks!

[Yan-44]

Hi @ppanchad-amd ,
I have just installed ROCm 6.4.1 and run my simple program above. I come up with exactly the same issues. My understanding is that dynamically allocated tables cannot appear in a 'reduction' clause (which is a problem for my code). But if I define 'toto' statically, then the code compiles BUT the result is erroneous :

yannick@FNAN-20-036:~/lbm/tests$ /opt/rocm-6.4.1/lib/llvm/bin/flang -fopenmp -O2 test31b_reduction.f90 --offload-arch=gfx1030 -o test31b
yannick@FNAN-20-036:~/lbm/tests$ ./test31b

toto(1) = 0.000000000000000 , correct answer is: toto(1) = 1000 <-- calculated 'toto(1)' is erroneous
tata = 5.000000000000284 , correct answer is: tata = 5.000000000000284 <-- calculated 'tata' is correct

Thanks for helping.

[ppanchad-amd]

Hi @Yan-44. I have created an internal ticket to assist with your issue. Thanks!

[tcgu-amd]

Hi @Yan-44, sorry to hear that the issue is still not fixed for you. I was able to reproduce your issue on ROCm 6.4 with gfx1100 as well. Further investigation will be held to see what's going on. Thanks!

[tcgu-amd]

Update: @Yan-44 this is actually a known issue internally. We currently don't support target reduction that are type array. Sorry about that...

[Yan-44]

Hi @tcgu-amd and @ppanchad-amd ,
This is actually not true. From my own experience with Rocm/flang, target reduction with a STATIC table that is ALONE works well, for example 'test31c_reduction' below compiles and runs well (results are correct):

`Program test31c_reduction
implicit none

integer, parameter :: N=1000
integer :: i
double precision,dimension(2) :: toto

toto=0._8

!$OMP TARGET map(toto)
!$OMP TEAMS reduction(+: toto)
!$OMP DISTRIBUTE PARALLEL DO reduction(+: toto)
do i = 1, N
toto(1)=toto(1)+real(i,8)
end do
!$OMP END DISTRIBUTE PARALLEL DO
!$OMP END TEAMS
!$OMP END TARGET

print*,'toto(1) = ',toto(1),', correct answer is: toto(1) = ',N*(N+1)/2

end program test31c_reduction`

Now, the following two situations yield issues:

1. If I make table 'toto' ALLOCATABLE in this program, then the code compiles well but delivers the message '0: ALLOCATE: array already allocated' at runtime.
2. If I define a new variable 'tata' and want to add a 'reduction(max:tata)' to the existing 'reduction(+:toto)' (such as in my initial program), then the program compiles well too but the reduction on 'toto(1)' yields erroneous results.

Note that ALLOCATABLE tables are also a problem with '!$OMP ATOMIC' directives, and I read from OPENMP reference guide that Fortran OPENMP does not support allocatables... for some obscure reasons (obscure to me, at least, because C OPENMP does!).

I hope this helps orientate investigations in order to fix these issues.

Regards,

P.S.: do you want me to close this issue or shall I leave it open until it is fixed?

[tcgu-amd]

Hi @tcgu-amd and @ppanchad-amd , This is actually not true. From my own experience with Rocm/flang, target reduction with a STATIC table that is ALONE works well, for example 'test31c_reduction' below compiles and runs well (results are correct):

`Program test31c_reduction implicit none

integer, parameter :: N=1000 integer :: i double precision,dimension(2) :: toto

toto=0._8

!$OMP TARGET map(toto) !$OMP TEAMS reduction(+: toto) !$OMP DISTRIBUTE PARALLEL DO reduction(+: toto) do i = 1, N toto(1)=toto(1)+real(i,8) end do !$OMP END DISTRIBUTE PARALLEL DO !$OMP END TEAMS !$OMP END TARGET

print*,'toto(1) = ',toto(1),', correct answer is: toto(1) = ',N*(N+1)/2

end program test31c_reduction`

Now, the following two situations yield issues:

1. If I make table 'toto' ALLOCATABLE in this program, then the code compiles well but delivers the message '0: ALLOCATE: array already allocated' at runtime.
2. If I define a new variable 'tata' and want to add a 'reduction(max:tata)' to the existing 'reduction(+:toto)' (such as in my initial program), then the program compiles well too but the reduction on 'toto(1)' yields erroneous results.

Note that ALLOCATABLE tables are also a problem with '!$OMP ATOMIC' directives, and I read from OPENMP reference guide that Fortran OPENMP does not support allocatables... for some obscure reasons (obscure to me, at least, because C OPENMP does!).

I hope this helps orientate investigations in order to fix these issues.

Regards,

P.S.: do you want me to close this issue or shall I leave it open until it is fixed?

Thanks for the clarification @Yan-44, that would definitely be useful for investigation. In this case, I think leaving it open would make the most sense. Thanks!