Problem with multi GPU
- This topic has 9 replies, 3 voices, and was last updated 2 months, 1 week ago by .
-
Posts
-
Dear Open LB team,
I used the gpuopenmpi config for running the case with multi GPU. But I think it did not work on my workstation. I use 2 K80 GPU card on my workstation.
Here is the Cuda and MPI version which I use.
root@acmt:/home/kc-lin/olb-1.6r0/examples/laminar/cylinder3d# nvcc –version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__2_19:15:15_PDT_2021
Cuda compilation tools, release 11.4, V11.4.48
Build cuda_11.4.r11.4/compiler.30033411_0
root@acmt:/home/kc-lin/olb-1.6r0/examples/laminar/cylinder3d# mpirun –version
mpirun (Open MPI) 4.1.6Report bugs to http://www.open-mpi.org/community/help/
root@acmt:/home/kc-lin/olb-1.6r0/examples/laminar/cylinder3d# ompi_info –parsable -l 9 –all | grep mpi_built_with_cuda_support:value
mca:mpi:base:param:mpi_built_with_cuda_support:value:trueAfter make it I got it “nvcc cylinder3d.o -o cylinder3d -lolbcore -L../../../external/lib -lpthread -lz -ltinyxml -lcuda -lcudadevrt -lcudart -L../../../build/lib
”
And when I use the command: “mpirun -np 2 ./cylinder3d” for trying to run with 2 GPUs
Then I found only 1 GPU work but maybe it run 2 jobs at the same time ”
| 0 N/A N/A 7808 C ./cylinder3d 376MiB |
| 0 N/A N/A 7809 C ./cylinder3d 376MiB”Could you help me to deal with this problem?
Thank you so much!The problem is that each process by default uses the first visible GPU. You can restrict this to assign each process their own GPU via:
mpirun -np 2 bash -c 'export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; ./cylinder3d'This is also documented in the
config/gpu_openmpi.mk: (excerpt)# - Start the simulation using <code>mpirun -np 2 ./cavity3d</code> (All processes share default GPU, not optimal) # # Usage on a multi GPU system: (recommended when using MPI, use non-MPI version on single GPU systems) # - Run "mpirun -np 4 bash -c 'export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; ./cavity3d'" # (for a 4 GPU system, further process mapping advisable, consult cluster documentation)No worries! Glad to hear that it works now,
Is this using the
laminar/cylinder3dcase? One explanation could be if the Paraview files intmpare in a broken state due to playing around with various parallelization modes. Does this also happen if you completely remove it and restart? Did you change anything in the example case?see https://www.openlb.net/forum/reply/8856/ and thanks @aseidler
- You must be logged in to reply to this topic.