sync: rvck88: riscv: Add support for kernel mode vector#135
Conversation
mainline inclusion from mainline-v6.7-rc1 commit b18f7296fbfdb2ad0871f00f3042fc74663d52ac category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- For the sake of coherency, use local labels in assembly when applicable. This also avoid kprobes being confused when applying a kprobe since the size of function is computed by checking where the next visible symbol is located. This might end up in computing some function size to be way shorter than expected and thus failing to apply kprobes to the specified offset. Signed-off-by: Clément Léger <cleger@rivosinc.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Link: https://lore.kernel.org/r/20231024132655.730417-2-cleger@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7-rc1 commit 4cc0d8a3f109fbdd8100ed88fc9417203a5d5b4e category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- Some data were incorrectly annotated with SYM_FUNC_*() instead of SYM_DATA_*() ones. Use the correct ones. Signed-off-by: Clément Léger <cleger@rivosinc.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Link: https://lore.kernel.org/r/20231024132655.730417-4-cleger@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit ecd2ada8a5e0b464dab54f71d4ba7bbf5708711f category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Eric Biggers <ebiggers@google.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-2-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit 956895b9d8f74df015636288a81872c07c4fded3 category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- The goal of this patch is to provide full support of Vector in kernel softirq context. So that some of the crypto alogrithms won't need scalar fallbacks. By disabling bottom halves in active kernel-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector. So, softirq context is able to use Vector whenever it runs. After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context. Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT. Related-to: commit 696207d ("arm64/sve: Make kernel FPU protection RT friendly") Related-to: commit 66c3ec5 ("arm64: neon: Forbid when irqs are disabled") Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Eric Biggers <ebiggers@google.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-3-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit c5674d00cacdb1c47c72e19a552fbae401bc3532 category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen <hankuan.chen@sifive.com> Signed-off-by: Han-Kuan Chen <hankuan.chen@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-4-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit 7df56cbc27e4239807b5d8860f79a7350d63a741 category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Acked-by: Conor Dooley <conor.dooley@microchip.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-5-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit c2a658d419246108c9bf065ec347355de5ba8a05 category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- This patch utilizes Vector to perform copy_to_user/copy_from_user. If Vector is available and the size of copy is large enough for Vector to perform better than scalar, then direct the kernel to do Vector copies for userspace. Though the best programming practice for users is to reduce the copy, this provides a faster variant when copies are inevitable. The optimal size for using Vector, copy_to_user_thres, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it. The exception fixup code of the __asm_vector_usercopy must fallback to the scalar one because accessing user pages might fault, and must be sleepable. Current kernel-mode Vector does not allow tasks to be preemptible, so we must disactivate Vector and perform a scalar fallback in such case. The original implementation of Vector operations comes from https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel. Co-developed-by: Jerry Shih <jerry.shih@sifive.com> Signed-off-by: Jerry Shih <jerry.shih@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Suggested-by: Guo Ren <guoren@kernel.org> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-6-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit a93fdaf183125fea81f66b9bd756ef5a0c30859e category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- SR_SD summarizes the dirty status of FS/VS/XS. However, the current code structure does not fully utilize it because each extension specific code is divided into an individual segment. So remove the SR_SD check for now. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Song Shuai <songshuaishuai@tinylab.org> Reviewed-by: Guo Ren <guoren@kernel.org> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-7-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
…ore}() mainline inclusion from mainline-v6.7 commit d6c78f1ca3e8ec3fd1afa1bc567cdf083e7af9fe category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Acked-by: Conor Dooley <conor.dooley@microchip.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-8-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit 5b6048f2ff710196c85ce14373febe8be5115bbe category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- riscv_v_ctrl_set() should only touch bits within PR_RISCV_V_VSTATE_CTRL_MASK. So, use the mask when we really set task's vstate_ctrl. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-9-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit bd446f5df5afab212917f6732ba6442a5e8de85e category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-10-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit 2080ff9493072a94e42b1856d59f5f1bffb761b7 category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
mainline inclusion from mainline-v6.7 commit c27fa53b858b4ee6552a719aa599c250cf98a586 category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- The RISC-V Vector specification states in "Appendix D: Calling Convention for Vector State" [1] that "Executing a system call causes all caller-saved vector registers (v0-v31, vl, vtype) and vstart to become unspecified.". In the RISC-V kernel this is called "discarding the vstate". Returning from a signal handler via the rt_sigreturn() syscall, vector discard is also performed. However, this is not an issue since the vector state should be restored from the sigcontext, and therefore not care about the vector discard. The "live state" is the actual vector register in the running context, and the "vstate" is the vector state of the task. A dirty live state, means that the vstate and live state are not in synch. When vectorized user_from_copy() was introduced, an bug sneaked in at the restoration code, related to the discard of the live state. An example when this go wrong: 1. A userland application is executing vector code 2. The application receives a signal, and the signal handler is entered. 3. The application returns from the signal handler, using the rt_sigreturn() syscall. 4. The live vector state is discarded upon entering the rt_sigreturn(), and the live state is marked as "dirty", indicating that the live state need to be synchronized with the current vstate. 5. rt_sigreturn() restores the vstate, except the Vector registers, from the sigcontext 6. rt_sigreturn() restores the Vector registers, from the sigcontext, and now the vectorized user_from_copy() is used. The dirty live state from the discard is saved to the vstate, making the vstate corrupt. 7. rt_sigreturn() returns to the application, which crashes due to corrupted vstate. Note that the vectorized user_from_copy() is invoked depending on the value of CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD. Default is 768, which means that vlen has to be larger than 128b for this bug to trigger. The fix is simply to mark the live state as non-dirty/clean prior performing the vstate restore. Link: https://github.com/riscv/riscv-isa-manual/releases/download/riscv-isa-release-8abdb41-2024-03-26/unpriv-isa-asciidoc.pdf # [1] Reported-by: Charlie Jenkins <charlie@rivosinc.com> Reported-by: Vineet Gupta <vgupta@kernel.org> Fixes: c2a658d41924 ("riscv: lib: vectorize copy_to_user/copy_from_user") Signed-off-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Vineet Gupta <vineetg@rivosinc.com> Link: https://lore.kernel.org/r/20240403072638.567446-1-bjorn@kernel.org Cc: stable@vger.kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com> Signed-off-by: Gao Rui <gao.rui@zte.com.cn> Signed-off-by: Yanteng Si <si.yanteng@linux.dev> Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
riscv inclusion category: feature bugzilla: RVCK-Project/rvck#86 -------------------------------- Enable the following configs: CONFIG_RISCV_ISA_V The following configs have been merged in the previous PR and are now enabled together: CONFIG_SERIAL_EARLYCON_RISCV_SBI CONFIG_PARAVIRT CONFIG_PARAVIRT_TIME_ACCOUNTING Then update openeuler_defconfig: make save_oedefconfig make update_oedefconfig Build and boot testing passed. Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
|
开始测试 log: https://github.com/RVCK-Project/rvck-olk/actions/runs/19014892493 参数解析结果
测试完成 详细结果:RVCK result
Kunit Test Resultkunit test failed Kernel Build ResultKernel build failed. Check Patch Result
测试完成 详细结果:RVCK result
Kunit Test Resultkunit test failed Kernel Build ResultKernel build failed. Check Patch Result
测试完成 详细结果:RVCK result
Kunit Test Resultkunit test failed Check Patch Result
测试完成 详细结果:RVCK result
Kunit Test Resultkunit test failed Kernel Build ResultKernel build succeeded: RVCK-Project/rvck-olk/135/ 60fdf22b4a6439c0bfb77f270cad563d /srv/guix_result/3d3ac27d2217af7ca3a1763fd6a4698a702cbfda/Image LAVA Checkargs:
result:Lava check done! lava log: https://lava.oerv.ac.cn/scheduler/job/911 lava result count: [fail]: 17, [pass]: 1589, [skip]: 293 Check Patch Result
|
|
/check lava_template=lava-job-template/qemu/qemu-mmtests.yaml testcase_path=lava-testcases/common-test/MMTests/MMTests.yaml 开始测试 log: https://github.com/RVCK-Project/rvck-olk/actions/runs/19062535760 参数解析结果
测试完成 详细结果:RVCK result
Kunit Test Resultkunit test failed Kernel Build ResultKernel build succeeded: RVCK-Project/rvck-olk/135_3484571858/ 0232d94567dc385725913636b6c76308 /srv/guix_result/3d3ac27d2217af7ca3a1763fd6a4698a702cbfda/Image LAVA Checkargs:
result:Lava check done! lava log: https://lava.oerv.ac.cn/scheduler/job/914 lava result count: [fail]: 12, [pass]: 68 Check Patch Result
|
Link: RVCK-Project/rvck#88
Updated openeuler_defconfig.