Do llama.cpp support input_embeds?

[OswaldoBornemann]

Do llama.cpp support input_embeds? Just like transformers support input_embeds in model.generate function.

[ggerganov]

Probably llama_batch.embd is what you are looking for:

llama.cpp/include/llama.h

Line 240 in 70392f1

float * embd;

[OswaldoBornemann]

Yeah that's great. In my case, the input is only the input_embeds instead of tokens. So you mean llama.cpp already supported it?

[ggerganov]

Yes, you can pass NULL for token and then put the token embeddings directly via the embd pointer. It should point to n_tokens*n_embd floats

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.