Cuda refactor, multi GPU support

[JohannesGaessler]

This PR is quite large. Its primary goal is to lay the groundwork for the implementation of further CUDA kernels for ggml operations. I am also adding multi GPU support because it's easier to integrate now than it would be at a later point.

For Users

Build instructions (Linux):

git clone https://github.com/JohannesGaessler/llama.cpp llama.cpp-johannesgaessler
cd llama.cpp-johannesgaessler                               
git fetch
git switch cuda-refactor-8
make LLAMA_CUBLAS=1

When compiling with LLAMA_CUBLAS=1 the program automatically detects the available NVIDIA devices and splits weights proportional to VRAM. There is not yet a CLI argument for setting the tensor splits. The performance increase on my test systems is relatively low (+70% t/s when going from 1x GTX TITAN X to 4x GTX TITAN X). It's possible that there is still a bug that hampers performance. Please do tell me how well (if at all) it works for you. In any case, this PR should already allow you to pool the VRAM of multiple GPUs to load larger models.

For Developers

This PR is still very much WIP. I will do a refactor to remove artifacts from bad/obsolete design decisions. You can already review the code if you want but many of the flaws are still subject to change. Should be good now.

On master there are separate functions for invoking CUDA kernels. Apart from invoking the actual CUDA kernels they do other things such as copying data between host and device. This PR adds a template ggml_cuda_op that manages

1. the transfer of data between host and device,
2. the dequantization of src0 (needed for cuBLAS matrix multiplication),
3. the broadcasting of src1 across src0 (needed for multiplication),
4. and multi GPU things.

The actual operations now only need to define how the data should be manipulated.

This PR also moves the entry point for invoking CUDA kernels from the ggml function such as ggml_compute_forward_mul_mat_q_f32 and instead adds a function ggml_cuda_compute_forward that is called from ggml_compute_forward. For this to work I moved ggml_task_type and ggml_compute_params from ggml.c to ggml.h.

This PR adds an int for the layer, an int for the device id, and dedicated device data pointers to ggml_tensor. I need these for bookkeeping. I also changed the backends from GGML_BACKEND_CUDA and GGML_BACKEND_OPENCL to GGML_BACKEND_GPU (tensor data on 1 GPU) and GGML_BACKEND_GPU_SPLIT (tensor data split across all GPUs). Since I think that we don't want to support the simultaneous use of CUDA and OpenCL it's simpler to just use the same backend types for both implementations and to differentiate via defines.

[github-actions]

clang-tidy made some suggestions

[ENjoyBlue2021]

Thanks so much for adding multi GPU support, I was looking forward to it. You are the king man.
This is extremly useful to increase my vram for 30b.
I'm posting my stats with my 1080ti+1080.
Amazing stuff, its noticeable faster.

Stats with this version:

./main -m '/media/w/PhoenixSSD/oobabooga/text-generation-webui/models/supercot30b-ggml/ggml-model-q5_1.bin' -n 128 --n-gpu-layers 39 --threads 6 --no-mmap -s 1685192470
WARNING: when using cuBLAS generation results are NOT guaranteed to be reproducible.
main: build = 596 (428c342)
main: seed  = 1685192470
ggml_init_cublas: found 2 CUDA devices:
  1. NVIDIA GeForce GTX 1080 Ti
  2. NVIDIA GeForce GTX 1080
llama.cpp: loading model from /media/w/PhoenixSSD/oobabooga/text-generation-webui/models/supercot30b-ggml/ggml-model-q5_1.bin
llama_model_load_internal: format     = ggjt v2 (pre #1508)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 6656
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 52
llama_model_load_internal: n_layer    = 60
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 9 (mostly Q5_1)
llama_model_load_internal: n_ff       = 17920
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 23269.21 MB
llama_model_load_internal: mem required  = 10646.42 MB (+ 3124.00 MB per state)
llama_model_load_internal: [cublas] offloading 39 layers to GPU
llama_model_load_internal: [cublas] total VRAM used: 14926 MB
....................................................................................................
llama_init_from_file: kv self size  =  780.00 MB

system_info: n_threads = 6 / 8 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 0


 // Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using System;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Threading;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

namespace MSBuild.ExtensionPack.Platform
{
    /// <summary>
    /// Adds support for a target to find the first non-empty environment variable, by name
llama_print_timings:        load time = 14079.62 ms
llama_print_timings:      sample time =    55.40 ms /   128 runs   (    0.43 ms per token)
llama_print_timings: prompt eval time =   722.20 ms /     2 tokens (  361.10 ms per token)
llama_print_timings:        eval time = 54367.11 ms /   127 runs   (  428.09 ms per token)
llama_print_timings:       total time = 68522.85 ms

Compare old Version:

./main -m '/media/w/PhoenixSSD/oobabooga/text-generation-webui/models/supercot30b-ggml/ggml-model-q5_1.bin' -n 128 --n-gpu-layers 17 --threads 6 --no-mmap -s 1685192470
WARNING: when using cuBLAS generation results are NOT guaranteed to be reproducible.
main: build = 583 (7e4ea5b)
main: seed  = 1685192470
llama.cpp: loading model from /media/w/PhoenixSSD/oobabooga/text-generation-webui/models/supercot30b-ggml/ggml-model-q5_1.bin
llama_model_load_internal: format     = ggjt v2 (pre #1508)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 6656
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 52
llama_model_load_internal: n_layer    = 60
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 9 (mostly Q5_1)
llama_model_load_internal: n_ff       = 17920
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 23269.14 MB
llama_model_load_internal: mem required  = 19066.59 MB (+ 3124.00 MB per state)
llama_model_load_internal: [cublas] offloading 17 layers to GPU
llama_model_load_internal: [cublas] total VRAM used: 6506 MB
....................................................................................................
llama_init_from_file: kv self size  =  780.00 MB

system_info: n_threads = 6 / 8 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 0


 // Copyright (c) Microsoft. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using System;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Threading;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

namespace MSBuild.ExtensionPack.Platform
{
    /// <summary>
    /// Adds support for a target to find the first non-empty environment variable, by name
llama_print_timings:        load time = 11117.69 ms
llama_print_timings:      sample time =    55.35 ms /   128 runs   (    0.43 ms per token)
llama_print_timings: prompt eval time =   896.98 ms /     2 tokens (  448.49 ms per token)
llama_print_timings:        eval time = 82108.70 ms /   127 runs   (  646.53 ms per token)
llama_print_timings:       total time = 93302.20 ms

[SlyEcho]

Would be too much to ask cross platform (Nvidia + AMD) support LMAO.

I will try to check later, I think I can access a machine with 2x 2080 Ti

[JohannesGaessler]

Performance numbers from my test machine with an i5-4570S, 16 GB of RAM @ 1600 MHz, and a GTX 1070 + a GTX 1050 ti:

Model	GPU	ms/t	t/s
7b q4_0	GTX 1070	71.37	14.01
7b q4_0	GTX 1070 + GTX 1050 ti	68.66	14.56
13b q4_0	GTX 1070	134.19	7.45
13b q4_0	GTX 1070 + GTX 1050 ti	128.13	7.80
33b q4_0	GTX 1070	Unusable	Unusable
33b q4_0	GTX 1070 + GTX 1050 ti	575.12	1.74

Numbers for single GPU are obtained using the master branch.

Note: previously I was able to run 33b q4_0 with just the GTX 1070 on master; there may be something on master that has increased RAM usage since.

[SlyEcho]

33b

You mean 30B? I can run 30B Q4_0 with my 8 GB card with 20 layers loaded only.

[howard0su]

we can decide which card we should put the layer on in this function (backend + device_id).

We should make backend like CPU = 0xFFFFFFFF; CUDA=0xEEEEEE00; (256 cards supported); etc.

[JohannesGaessler]

This PR implements two ways to utilize multiple GPUs: tensors can either be put on a single GPU or they can be split across multiple GPUs. I think that tensors that require little computation should be put on a single GPU to reduce data copying. In the function ggml_cuda_load_data single GPU tensors are distributed to GPUs based on which layer they are in. I guess this distribution could also be done here but I don't think it would make a significant difference.

[JohannesGaessler]

You mean 30B? I can run 30B Q4_0 with my 8 GB card with 20 layers loaded only.

"30B" seems to be a typo by Meta that has become dominant. In the paper they talk about a "33B" model so that is the term that I'm using.

[SlyEcho]

What if it were like this so the tensors that are not on a layer don't have to specify -1?

Suggested change

	struct ggml_tensor * get_tensor_for(llama_load_tensor & lt, int layer, ggml_backend backend) {
	struct ggml_tensor * get_tensor_for(llama_load_tensor & lt, ggml_backend backend, int layer = -1) {

[slaren]

I think ggml_cuda_mul and ggml_cuda_mul_mat can be removed from ggml-cuda.h now and made static.

[JohannesGaessler]

I added a comment to explain the weird device to host memcpy for split tensors. Since I as the person who wrote the code won't know: are there other parts of the code that are unintuitive or difficult to understand?

[github-actions]

clang-tidy made some suggestions

[JohannesGaessler]

I added a CLI argument that lets the user set the tensor split. On my system a less VRAM efficient split of 3:1 seems to do better than 2:1 because it's more efficient in terms of compute:

Model	GPU	ms/t	t/s
7b q4_0	GTX 1070	71.37	14.01
7b q4_0	GTX 1070 + GTX 1050 ti, 2:1 split	68.66	14.56
7b q4_0	GTX 1070 + GTX 1050 ti, 3:1 split	59.03	16.94
13b q4_0	GTX 1070	134.19	7.45
13b q4_0	GTX 1070 + GTX 1050 ti, 2:1 split	128.13	7.80
13b q4_0	GTX 1070 + GTX 1050 ti, 3:1 split	109.14	9.15
33b q4_0	GTX 1070	Unusable	Unusable
33b q4_0	GTX 1070 + GTX 1050 ti, 2:1 split	575.12	1.74
33b q4_0	GTX 1070 + GTX 1050 ti, 3:1 split	571.10	1.75

[KerfuffleV2]

-n N, --n-predict N   number of tokens to predict (default: -1, -1 = infinity)

Please create an issue or discussion topic in the Q&A section when you have general questions instead of asking in pull requests.

Comments in pull requests are primarily for discussing that specific pull.

[JohannesGaessler]

Both CUDA and OpenCL seem to be working correctly in short tests. However, right now I'm too tired to test this PR in detail. I'll do it tomorrow morning.

[JohannesGaessler]

As far as I can tell everything is working correctly.

[KerfuffleV2]

ggml_init_cublas: found 1 CUDA devices:
  1. NVIDIA GeForce GTX 1060 6GB

Works for me on Linux, performance for generating tokens with main seems the same as master. Identical results with the same seed also. Perplexity (and I'd assume just cuBLAS prompt evaluation in general) is noticeably slower though. (Output abbreviated below.)

master

$ taskset -c 0-5 ./perplexity -f /path/wiki.test.raw -m /path/llama-7b.ggmlv3.q4_0.bin -t 6 -ngl 0
system_info: n_threads = 6 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
perplexity: calculating perplexity over 655 chunks, batch_size=512
perplexity: 3.66 seconds per pass - ETA 39 minutes
[1]4.4545,[2]4.9402,[3]5.8278

---

$ taskset -c 0-5 ./perplexity -f /path/wiki.test.raw -m /path/llama-7b.ggmlv3.q4_0.bin -t 6 -ngl 33
system_info: n_threads = 6 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
perplexity: calculating perplexity over 655 chunks, batch_size=512
perplexity: 3.37 seconds per pass - ETA 36 minutes
[1]4.4547,[2]4.9404

this pull

$ taskset -c 0-5 ./perplexity -f /path/wiki.test.raw -m /path/llama-7b.ggmlv3.q4_0.bin -t 6 -ngl 0
system_info: n_threads = 6 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
perplexity: calculating perplexity over 655 chunks, batch_size=512
perplexity: 5.47 seconds per pass - ETA 59 minutes
[1]4.4545,[2]4.9402

---

$ taskset -c 0-5 ./perplexity -f /path/wiki.test.raw -m /path/llama-7b.ggmlv3.q4_0.bin -t 6 -ngl 33
system_info: n_threads = 6 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
perplexity: calculating perplexity over 655 chunks, batch_size=512
perplexity: 5.12 seconds per pass - ETA 55 minutes
[1]4.4545,[2]4.9402

[JohannesGaessler]

Thank you for pointing this out, I forgot to check it. The reason is most likely that on master there are some f16 f32 matrix multiplications where f32 is converted to f16 and then cuBLAS is used to do f16 matrix multiplication. In this PR however the f16 data is converted to f32 and then cuBLAS is used to do f32 matrix multiplication. Originally I did not want to touch the f16 f32 code at all but it caused issues for CPU layers in combination with >1 GPU.

As a user I don't really care about prompt processing speed because it's so fast anyways. As a developer waiting longer for perplexity calculations could be an issue though. I would be fine with delaying the merging of this PR if it's desired to fix prompt processing speed beforehand; otherwise I will at a later point make another PR to fix it.

[JohannesGaessler]

I forgot to mention: the f16 f32 matrix multiplications are mostly related to the KV cache and they're done on permuted matrices. To use regular cuBLAS matrix multiplication you first need to copy the matrices to make them contiguous but I'm currently trying to develop a kernel that operates directly on the permuted memory layout. Ideally that will then be faster than master anyways (for GPU layers).

[KerfuffleV2]

As a user I don't really care about prompt processing speed because it's so fast anyways. As a developer waiting longer for perplexity calculations could be an issue though.

This is just the opinion of a random person, but prompt processing is fast compared to inference for sure but at least on relatively old hardware something like an 800 token prompt still takes a pretty noticeable time to evaluate especially on larger models like 33b+. Roughly doubling that time from, I don't know, 15-20sec to 30-40sec is quite noticeable.

Also, with the advances to context size technology and some models being fine tuned to allow >2048 sized context it's likely in the next months or whatever longer prompts are going to get more common.

I've also been doing experimentation on various stuff and using the perplexity tool to check the effect so making it twice as slow (and it's already pretty low for big models) definitely will make me sad.

[JohannesGaessler]

One workaround that might work would be to re-add the implementation on master and to only enable it when there is a single GPU.

[JohannesGaessler]

I pushed a fix that re-adds the old code for single GPU prompt processing. Prompt processing seems to still be slightly slower but only by ~20%. I'm not sure what's causing the remaining difference; investigating and optimizing the performance would probably take some time. In any case, @KerfuffleV2 , can you please check the performance on your system?

[KerfuffleV2]

Definitely helps.

-ngl 0

master: 3.66 seconds per pass - ETA 39 minutes
before: 5.47 seconds per pass - ETA 59 minutes
now: 4.16 seconds per pass - ETA 45 minutes

-ngl 33

master: 3.37 seconds per pass - ETA 36 minutes
before: 5.12 seconds per pass - ETA 55 minutes
now: 3.74 seconds per pass - ETA 40 minutes

[JohannesGaessler]

Alright, my current status is this: I'm working on another PR based on this one that adds GPU acceleration for the entire LLaMa model rather than just matrix multiplication and component-wise multiplication. On my hardware this WIP version is already faster than master and this PR when it comes to token generation. I would like to get a working version for that first, and then optimize performance afterwards. Optimizing the performance of this PR now seems like it would be a bad investment of my time since I think I will be able to get better performance with a fundamentally different implementation anyways. So I think the two options for this PR are to either merge it and accept a performance regression for prompt processing or to wait until I have the next PR ready which I think will be faster than master across the board.

[KerfuffleV2]

Sounds reasonable.

I'm working on another PR based on this one that adds GPU acceleration for the entire LLaMa model rather than just matrix multiplication and component-wise multiplication.

Feel free to ignore this if it's not the appropriate time/place to ask, but I'm curious if this is going to involve requiring significantly more VRAM?

[JohannesGaessler]

llama.cpp master currently reserves 1 GB of scratch buffer memory for 33b and smaller models. Translating this 1:1 to VRAM would be simple but inefficient because these scratch buffers are much larger than they need to be. Also you only need that much memory initially during prompt processing and there the amount required is proportional to the batch size. However, for low-end cards a large batch size probably has diminishing returns anyways. You also need some VRAM for the KV cache but I haven't worked out how much that will be exactly.

My current testing targets are an RTX 3090, a GTX 1070, and a GTX 1050 ti. If I can get universally better performance for all of these that's great, otherwise I plan to add something like a --low-vram option.

[ggerganov]

So I think the two options for this PR are to either merge it and accept a performance regression for prompt processing or to wait until I have the next PR ready which I think will be faster than master across the board.

The perplexity computation speed is quite important to me so if the fix is not trivial, I prefer to wait for the other PR that will hopefully not degrade the performance. The multi-GPU addition is really great, but from practical point of view, we need the perplexity speed right now, even more so because of #1684 efforts that are about to get merged.

I'll leave this PR open for now - if I get the time, I might try to fix the regression and merge it (if the other PR is not ready yet)

[JohannesGaessler]

I did a quick test with my WIP PR. The perplexity speed is 2.23 ms/t vs. 3.07 ms/t on master (numbers are for 7b q4_0). The PR has GPU acceleration for add, SiLU, RMS norm, cpy, reshape, view, transpose, and RoPE. As a third option I could prioritize getting that PR in a state that can be merged and then deliver kernels for the rest of the operations in a later PR.

Edit: for 33b q4_0 the speed of the WIP PR is 6.50 ms/t vs. 9.35 ms/t on master.

[ggerganov]

As a third option I could prioritize getting that PR in a state that can be merged and then deliver kernels for the rest of the operations in a later PR.

Up to you - whichever way you prefer

[JohannesGaessler]

I'll do it.

[JohannesGaessler]

Obsolete due to the merging of #1703