GPU hang error when run model

[helloworlde]

Hi, thanks so much for your documentation, I'm using AMD 8845HS with 780M GPU run deepseek-r1:1.5b by Ollama follow by your document. But there has GPU hang error after several rounds of conversation:

HW Exception by GPU node-1 (Agent handle: 0x7e6eb7d0bb40) reason :GPU Hang

• Hardware:

  • CPU: AMD 8845HS
  • GPU: 780M with 16GB VRAM
  • Memory: DDR5 5600Mhz 48G
  • OS: LXC Container in PVE 8.3

• docker-compose:

services:
  ollama:
    image: ollama/ollama:rocm
    container_name: ollama
    restart: unless-stopped
    devices:
      - "/dev/kfd"
      - "/dev/dri"
    volumes:
      - ./data:/root/.ollama
    environment:
      - OLLAMA_ORIGINS='chrome-extension://*,moz-extension://*'
      - HSA_OVERRIDE_GFX_VERSION=11.0.0
      - HCC_AMDGPU_TARGETS=gfx1103
      - OLLAMA_LLM_LIBRARY=rocm_v60002
      - OLLAMA_DEBUG=1 
    ports:
      - "11434:11434"  

• error message:

ollama  | time=2025-02-24T09:45:48.152Z level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-aabd4debf0c8f08881923f2c25fc0fdeed24435271c2b3e92c4af36704040dbc
ollama  | time=2025-02-24T09:45:48.152Z level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-aabd4debf0c8f08881923f2c25fc0fdeed24435271c2b3e92c4af36704040dbc
ollama  | llama_model_loader: loaded meta data with 26 key-value pairs and 339 tensors from /root/.ollama/models/blobs/sha256-aabd4debf0c8f08881923f2c25fc0fdeed24435271c2b3e92c4af36704040dbc (version GGUF V3 (latest))
ollama  | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
ollama  | llama_model_loader: - kv   0:                       general.architecture str              = qwen2
ollama  | llama_model_loader: - kv   1:                               general.type str              = model
ollama  | llama_model_loader: - kv   2:                               general.name str              = DeepSeek R1 Distill Qwen 1.5B
ollama  | llama_model_loader: - kv   3:                           general.basename str              = DeepSeek-R1-Distill-Qwen
ollama  | llama_model_loader: - kv   4:                         general.size_label str              = 1.5B
ollama  | llama_model_loader: - kv   5:                          qwen2.block_count u32              = 28
ollama  | llama_model_loader: - kv   6:                       qwen2.context_length u32              = 131072
ollama  | llama_model_loader: - kv   7:                     qwen2.embedding_length u32              = 1536
ollama  | llama_model_loader: - kv   8:                  qwen2.feed_forward_length u32              = 8960
ollama  | llama_model_loader: - kv   9:                 qwen2.attention.head_count u32              = 12
ollama  | llama_model_loader: - kv  10:              qwen2.attention.head_count_kv u32              = 2
ollama  | llama_model_loader: - kv  11:                       qwen2.rope.freq_base f32              = 10000.000000
ollama  | llama_model_loader: - kv  12:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
ollama  | llama_model_loader: - kv  13:                          general.file_type u32              = 15
ollama  | llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
ollama  | llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = qwen2
ollama  | llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
ollama  | llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
ollama  | llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
ollama  | llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151646
ollama  | llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151643
ollama  | llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 151643
ollama  | llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
ollama  | llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
ollama  | llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
ollama  | llama_model_loader: - kv  25:               general.quantization_version u32              = 2
ollama  | llama_model_loader: - type  f32:  141 tensors
ollama  | llama_model_loader: - type q4_K:  169 tensors
ollama  | llama_model_loader: - type q6_K:   29 tensors
ollama  | llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
ollama  | llm_load_vocab: special tokens cache size = 22
ollama  | llm_load_vocab: token to piece cache size = 0.9310 MB
ollama  | llm_load_print_meta: format           = GGUF V3 (latest)
ollama  | llm_load_print_meta: arch             = qwen2
ollama  | llm_load_print_meta: vocab type       = BPE
ollama  | llm_load_print_meta: n_vocab          = 151936
ollama  | llm_load_print_meta: n_merges         = 151387
ollama  | llm_load_print_meta: vocab_only       = 1
ollama  | llm_load_print_meta: model type       = ?B
ollama  | llm_load_print_meta: model ftype      = all F32
ollama  | llm_load_print_meta: model params     = 1.78 B
ollama  | llm_load_print_meta: model size       = 1.04 GiB (5.00 BPW) 
ollama  | llm_load_print_meta: general.name     = DeepSeek R1 Distill Qwen 1.5B
ollama  | llm_load_print_meta: BOS token        = 151646 '<｜begin▁of▁sentence｜>'
ollama  | llm_load_print_meta: EOS token        = 151643 '<｜end▁of▁sentence｜>'
ollama  | llm_load_print_meta: PAD token        = 151643 '<｜end▁of▁sentence｜>'
ollama  | llm_load_print_meta: LF token         = 148848 'ÄĬ'
ollama  | llm_load_print_meta: EOG token        = 151643 '<｜end▁of▁sentence｜>'
ollama  | llm_load_print_meta: max token length = 256
ollama  | llama_model_load: vocab only - skipping tensors
ollama  | time=2025-02-24T09:45:48.455Z level=DEBUG source=routes.go:1457 msg="chat request" images=0 prompt="You are a professional, authentic machine translation engine.\n\nYou are about to translate text from an article. Title: “OLLAMA_ORIGINS=chrome-extension://etc does not work · Issue #1686 · ollama/ollama”, Summary: {{imt_theme}}\n\nThis content may include the following terms {{imt_terms}}. Please handle these terms carefully.<｜User｜>; 把下一行文本作为纯文本输入，并将其翻译为简体中文，, if the text contains html tags, please consider after translate, where the tags should be in translated result, meanwhile keep the result fluently.仅输出翻译。如果某些内容无需翻译（如专有名词、代码等），则保持原文不变。不要解释，输入文本:\nOLLAMA_ORIGINS=chrome-extension://etc does not work #1686<｜Assistant｜>"
ollama  | time=2025-02-24T09:45:48.457Z level=DEBUG source=routes.go:1457 msg="chat request" images=0 prompt="You are a professional, authentic machine translation engine.\n\nYou are about to translate text from an article. Title: “OLLAMA_ORIGINS=chrome-extension://etc does not work · Issue #1686 · ollama/ollama”, Summary: {{imt_theme}}\n\nThis content may include the following terms {{imt_terms}}. Please handle these terms carefully.<｜User｜>; 把下一行文本作为纯文本输入，并将其翻译为简体中文，, if the text contains html tags, please consider after translate, where the tags should be in translated result, meanwhile keep the result fluently.仅输出翻译。如果某些内容无需翻译（如专有名词、代码等），则保持原文不变。不要解释，输入文本:\nOLLAMA_ORIGINS=chrome-extension://etc does not work · Issue #1686 · ollama/ollama<｜Assistant｜>"
ollama  | time=2025-02-24T09:45:48.458Z level=DEBUG source=cache.go:99 msg="loading cache slot" id=0 cache=0 prompt=174 used=0 remaining=174
// ... part of chat message
ollama  | time=2025-02-24T09:45:48.776Z level=DEBUG source=cache.go:99 msg="loading cache slot" id=1 cache=0 prompt=183 used=0 remaining=183
ollama  | time=2025-02-24T09:45:48.776Z level=DEBUG source=cache.go:99 msg="loading cache slot" id=2 cache=0 prompt=190 used=0 remaining=190
ollama  | time=2025-02-24T09:45:48.776Z level=DEBUG source=cache.go:99 msg="loading cache slot" id=3 cache=0 prompt=168 used=0 remaining=168
ollama  | HW Exception by GPU node-1 (Agent handle: 0x7e6eb7d0bb40) reason :GPU Hang
ollama  | time=2025-02-24T09:45:50.472Z level=DEBUG source=sched.go:407 msg="context for request finished"
ollama  | time=2025-02-24T09:45:50.472Z level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-aabd4debf0c8f08881923f2c25fc0fdeed24435271c2b3e92c4af36704040dbc refCount=14
ollama  | time=2025-02-24T09:45:50.472Z level=DEBUG source=sched.go:407 msg="context for request finished"
ollama  | time=2025-02-24T09:45:50.472Z level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-aabd4debf0c8f08881923f2c25fc0fdeed24435271c2b3e92c4af36704040dbc refCount=13
ollama  | time=2025-02-24T09:45:50.472Z level=DEBUG source=sched.go:407 msg="context for request finished"

• rocm

rocminfo              
ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.14
Runtime Ext Version:     1.6
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen 7 8845HS w/ Radeon 780M Graphics
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen 7 8845HS w/ Radeon 780M Graphics
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   5137                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            16                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Memory Properties:       
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    32638500(0x1f20624) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    32638500(0x1f20624) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    32638500(0x1f20624) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 4                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    32638500(0x1f20624) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx1103                            
  Uuid:                    GPU-XX                             
  Marketing Name:          AMD Radeon Graphics                
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      32(0x20) KB                        
    L2:                      2048(0x800) KB                     
  Chip ID:                 6400(0x1900)                       
  ASIC Revision:           12(0xc)                            
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   2700                               
  BDFID:                   50432                              
  Internal Node ID:        1                                  
  Compute Unit:            12                                 
  SIMDs per CU:            2                                  
  Shader Engines:          1                                  
  Shader Arrs. per Eng.:   2                                  
  WatchPts on Addr. Ranges:4                                  
  Coherent Host Access:    FALSE                              
  Memory Properties:       APU
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          32(0x20)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        32(0x20)                           
  Max Work-item Per CU:    1024(0x400)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Packet Processor uCode:: 40                                 
  SDMA engine uCode::      21                                 
  IOMMU Support::          None                               
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    16319248(0xf90310) KB              
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    16319248(0xf90310) KB              
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 3                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Recommended Granule:0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx1103         
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***        

• os

uname -a  
Linux dev 6.8.12-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-4 (2024-11-06T15:04Z) x86_64 x86_64 x86_64 GNU/Linux

Did you meet same issues or can you give me some message to fix this issue, thank you so much.

[alexhegit]

I do not meet this issue. Have you monitor the DDR memory usage? Any linux kernel log got by dmesg?

[helloworlde]

Thanks for responding. The memory of system is stable when request AI model, but there have some error in kernel log like this:

dmesg | grep -i -E "gpu|amdgpu|hang"
[235148.792892] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235148.793099] amdgpu 0000:c5:00.0: amdgpu: failed to remove hardware queue from MES, doorbell=0x1008
[235148.793101] amdgpu 0000:c5:00.0: amdgpu: MES might be in unrecoverable state, issue a GPU reset
[235148.793105] amdgpu 0000:c5:00.0: amdgpu: Failed to evict queue 4
[235148.793108] amdgpu: Failed to evict process queues
[235148.793120] amdgpu 0000:c5:00.0: amdgpu: GPU reset begin!
[235148.926482] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235148.926688] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.054604] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.054809] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.183002] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.183215] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.311418] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.311620] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.439907] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.440104] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.568246] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.568437] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.696813] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.697013] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.747602] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 3
[235149.747608] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 2
[235149.747609] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 1
[235149.747611] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 0
[235149.825441] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.825649] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.875188] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[235149.875394] [drm:amdgpu_mes_flush_shader_debugger [amdgpu]] *ERROR* failed to set_shader_debugger
[235149.953343] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235149.953703] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235149.955412] amdgpu 0000:c5:00.0: amdgpu: MODE2 reset
[235149.991836] amdgpu 0000:c5:00.0: amdgpu: GPU reset succeeded, trying to resume
[235149.992761] amdgpu 0000:c5:00.0: amdgpu: SMU is resuming...
[235149.993898] amdgpu 0000:c5:00.0: amdgpu: SMU is resumed successfully!
[235149.999364] amdgpu 0000:c5:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully.
[235149.999923] amdgpu 0000:c5:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[235149.999927] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[235149.999929] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[235149.999931] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[235149.999933] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[235149.999935] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[235149.999937] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[235149.999938] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[235149.999940] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[235149.999942] amdgpu 0000:c5:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[235149.999944] amdgpu 0000:c5:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[235149.999946] amdgpu 0000:c5:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8
[235149.999948] amdgpu 0000:c5:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0
[235150.002586] amdgpu 0000:c5:00.0: amdgpu: recover vram bo from shadow start
[235150.002590] amdgpu 0000:c5:00.0: amdgpu: recover vram bo from shadow done
[235150.002599] amdgpu 0000:c5:00.0: amdgpu: GPU reset(7) succeeded!
[235157.838444] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235157.838640] amdgpu 0000:c5:00.0: amdgpu: failed to remove hardware queue from MES, doorbell=0x1008
[235157.838642] amdgpu 0000:c5:00.0: amdgpu: MES might be in unrecoverable state, issue a GPU reset
[235157.838644] amdgpu 0000:c5:00.0: amdgpu: Failed to evict queue 4
[235157.838645] amdgpu: Failed to evict process queues
[235157.838666] amdgpu 0000:c5:00.0: amdgpu: GPU reset begin!
[235157.972078] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235157.972294] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.100426] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.100620] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.228094] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.228306] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.356394] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.356594] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.484807] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.485012] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.613389] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.613589] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.742008] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.742214] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.870714] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.870910] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235158.889737] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 3
[235158.889743] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 2
[235158.889745] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 1
[235158.889747] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 0
[235158.999077] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235158.999290] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235159.000817] amdgpu 0000:c5:00.0: amdgpu: MODE2 reset
[235159.028754] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[235159.030646] [drm:amdgpu_mes_flush_shader_debugger [amdgpu]] *ERROR* failed to set_shader_debugger
[235159.036300] amdgpu 0000:c5:00.0: amdgpu: GPU reset succeeded, trying to resume
[235159.037015] amdgpu 0000:c5:00.0: amdgpu: SMU is resuming...
[235159.039814] amdgpu 0000:c5:00.0: amdgpu: SMU is resumed successfully!
[235159.045249] amdgpu 0000:c5:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully.
[235159.045661] amdgpu 0000:c5:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[235159.045663] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[235159.045665] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[235159.045667] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[235159.045669] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[235159.045670] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[235159.045672] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[235159.045673] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[235159.045674] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[235159.045676] amdgpu 0000:c5:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[235159.045677] amdgpu 0000:c5:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[235159.045678] amdgpu 0000:c5:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8
[235159.045680] amdgpu 0000:c5:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0
[235159.047764] amdgpu 0000:c5:00.0: amdgpu: recover vram bo from shadow start
[235159.047766] amdgpu 0000:c5:00.0: amdgpu: recover vram bo from shadow done
[235159.047776] amdgpu 0000:c5:00.0: amdgpu: GPU reset(8) succeeded!
[235166.798676] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235166.798870] amdgpu 0000:c5:00.0: amdgpu: failed to remove hardware queue from MES, doorbell=0x1008
[235166.798873] amdgpu 0000:c5:00.0: amdgpu: MES might be in unrecoverable state, issue a GPU reset
[235166.798877] amdgpu 0000:c5:00.0: amdgpu: Failed to evict queue 4
[235166.798878] amdgpu: Failed to evict process queues
[235166.798913] amdgpu 0000:c5:00.0: amdgpu: GPU reset begin!
[235166.931752] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235166.931956] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.060300] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.060503] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.188848] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.189048] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.317492] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.317696] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.446113] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.446330] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.575045] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.575235] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.703606] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.703825] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.832094] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.832296] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235167.937463] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 3
[235167.937469] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 2
[235167.937470] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 1
[235167.937472] amdgpu 0000:c5:00.0: amdgpu: Failed to remove queue 0
[235167.960738] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[235167.960950] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[235168.058902] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[235168.059244] [drm:amdgpu_mes_flush_shader_debugger [amdgpu]] *ERROR* failed to set_shader_debugger
[235168.168455] [drm:gfx_v11_0_cp_gfx_enable.isra.0 [amdgpu]] *ERROR* failed to halt cp gfx
[235168.170020] amdgpu 0000:c5:00.0: amdgpu: MODE2 reset
[235168.206356] amdgpu 0000:c5:00.0: amdgpu: GPU reset succeeded, trying to resume
[235168.207202] amdgpu 0000:c5:00.0: amdgpu: SMU is resuming...
[235168.208948] amdgpu 0000:c5:00.0: amdgpu: SMU is resumed successfully!
[235168.214862] amdgpu 0000:c5:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully.
[235168.215258] amdgpu 0000:c5:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[235168.215260] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[235168.215261] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[235168.215262] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[235168.215264] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[235168.215265] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[235168.215266] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[235168.215267] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[235168.215268] amdgpu 0000:c5:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[235168.215270] amdgpu 0000:c5:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[235168.215271] amdgpu 0000:c5:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[235168.215272] amdgpu 0000:c5:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8
[235168.215273] amdgpu 0000:c5:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0
[235168.217014] amdgpu 0000:c5:00.0: amdgpu: recover vram bo from shadow start
[235168.217016] amdgpu 0000:c5:00.0: amdgpu: recover vram bo from shadow done
[235168.217027] amdgpu 0000:c5:00.0: amdgpu: GPU reset(9) succeeded!

Does that mean the ROCm driver couldn't work well with 780M?

[reywang18]

  - HSA_OVERRIDE_GFX_VERSION=11.0.0 or could be 11.0.2

Now ROCm has 7.2 version.
I think ROCm still has issue with 780M.