Bug: Ccache causing SYCL backend failed to build on Windows

[shou692199]

What happened?

With ccache installed in Windows, trying to build with SYCL by following instructions in https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md#windows.
I found that *.cpp.obj files were unable to be placed in right location but in having them with .o extension in /build folder. This behavior causing linker failed to do the job.

Here is the screenshot:

Name and Version

llama.cpp repository cda0e4b
Windows 11 23H2 10.0.22631.4317
VS2022 Community 17.11.5
Intel oneAPI 2024.2.1
MSVC v143
Ccache 4.10.2

What operating system are you seeing the problem on?

Windows

Relevant log output

[1/184] Building CXX object src\CMakeFiles\llama.dir\unicode-data.cpp.obj
[2/184] Building CXX object common\CMakeFiles\build_info.dir\build-info.cpp.obj
[3/184] Building CXX object common\CMakeFiles\common.dir\arg.cpp.obj
[4/184] Building CXX object common\CMakeFiles\common.dir\common.cpp.obj
[5/184] Building CXX object common\CMakeFiles\common.dir\console.cpp.obj
[6/184] Building CXX object common\CMakeFiles\common.dir\json-schema-to-grammar.cpp.obj
[7/184] Building CXX object common\CMakeFiles\common.dir\log.cpp.obj
[8/184] Building CXX object common\CMakeFiles\common.dir\ngram-cache.cpp.obj
[9/184] Building CXX object common\CMakeFiles\common.dir\sampling.cpp.obj
[10/184] Building CXX object common\CMakeFiles\common.dir\train.cpp.obj
[11/184] Building CXX object tests\CMakeFiles\test-tokenizer-0.dir\test-tokenizer-0.cpp.obj
[12/184] Building CXX object tests\CMakeFiles\test-tokenizer-1-bpe.dir\test-tokenizer-1-bpe.cpp.obj
[13/184] Building CXX object tests\CMakeFiles\test-tokenizer-1-spm.dir\test-tokenizer-1-spm.cpp.obj
[14/184] Building CXX object tests\CMakeFiles\test-log.dir\test-log.cpp.obj
[15/184] Building CXX object ggml\src\CMakeFiles\ggml.dir\ggml-backend.cpp.obj
In file included from C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-backend.cpp:13:
C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-impl.h(62,15): warning: unused function 'ggml_bitset_size' [-Wunused-function]
   62 | static size_t ggml_bitset_size(size_t n) {
      |               ^~~~~~~~~~~~~~~~
C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-impl.h(131,13): warning: unused function 'ggml_hash_contains' [-Wunused-function]
  131 | static bool ggml_hash_contains(const struct ggml_hash_set * hash_set, struct ggml_tensor * key) {
      |             ^~~~~~~~~~~~~~~~~~
2 warnings generated.
In file included from C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-backend.cpp:13:
C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-impl.h(62,15): warning: unused function 'ggml_bitset_size' [-Wunused-function]
   62 | static size_t ggml_bitset_size(size_t n) {
      |               ^~~~~~~~~~~~~~~~
C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-impl.h(131,13): warning: unused function 'ggml_hash_contains' [-Wunused-function]
  131 | static bool ggml_hash_contains(const struct ggml_hash_set * hash_set, struct ggml_tensor * key) {
      |             ^~~~~~~~~~~~~~~~~~
2 warnings generated.
[16/184] Building CXX object ggml\src\CMakeFiles\ggml.dir\ggml-sycl\common.cpp.obj
In file included from C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-sycl\common.cpp:13:
In file included from C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-sycl\common.hpp:19:
C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-sycl\dpct\helper.hpp(181,17): warning: unused function 'destroy_event' [-Wunused-function]
  181 |     static void destroy_event(event_ptr event)
      |                 ^~~~~~~~~~~~~
C:\Users\shou6\Documents\Code\llama.cpp\ggml\src\ggml-sycl\dpct\helper.hpp(1806,17): warning: unused function 'async_dpct_memcpy' [-Wunused-function]
 1806 |     static void async_dpct_memcpy(void *to_ptr, const void *from_ptr, size_t size,
      |                 ^~~~~~~~~~~~~~~~~
2 warnings generated.
..........................
[105/184] Linking CXX shared library bin\ggml.dll
FAILED: bin/ggml.dll ggml/src/ggml.lib 
C:\Windows\system32\cmd.exe /C "C:\Windows\system32\cmd.exe /C ""C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E __create_def C:\Users\shou6\Documents\Code\llama.cpp\build\ggml\src\CMakeFiles\ggml.dir\.\exports.def C:\Users\shou6\Documents\Code\llama.cpp\build\ggml\src\CMakeFiles\ggml.dir\.\exports.def.objs && cd C:\Users\shou6\Documents\Code\llama.cpp\build" && "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E vs_link_dll --intdir=ggml\src\CMakeFiles\ggml.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100261~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100261~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\Intel\oneAPI\compiler\latest\bin\icx.exe /nologo ggml\src\CMakeFiles\ggml.dir\ggml.c.obj ggml\src\CMakeFiles\ggml.dir\ggml-alloc.c.obj ggml\src\CMakeFiles\ggml.dir\ggml-backend.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-quants.c.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\common.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\concat.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\conv.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\convert.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\dmmv.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\im2col.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\mmq.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\mmvq.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\norm.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\rope.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\softmax.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl\tsembd.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-sycl.cpp.obj ggml\src\CMakeFiles\ggml.dir\llamafile\sgemm.cpp.obj ggml\src\CMakeFiles\ggml.dir\ggml-aarch64.c.obj  -LD /machine:x64 /INCREMENTAL:NO  -Qiopenmp -fsycl /Qoption,link,/DEF:ggml\src\CMakeFiles\ggml.dir\.\exports.def /Qoption,link,/LIBPATH:C:\PROGRA~2\Intel\oneAPI\compiler\latest\lib "C:\Program Files (x86)\Intel\oneAPI\dnnl\latest\lib\dnnl.lib"  "C:\Program Files (x86)\Intel\oneAPI\tbb\latest\lib\tbb12.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_intel_thread_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\compiler\latest\lib\libiomp5md.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_intel_ilp64_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_core_dll.lib"  -fsycl  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_blas_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_lapack_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_dft_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_sparse_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_data_fitting_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_rng_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_stats_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_sycl_vm_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\mkl_tbb_thread_dll.lib"  "C:\Program Files (x86)\Intel\oneAPI\tbb\latest\lib\tbb12.lib"  sycl.lib  OpenCL.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib -link /out:bin\ggml.dll /implib:ggml\src\ggml.lib /pdb:bin\ggml.pdb /version:0.0 && cd ."
Couldn't open file 'ggml\src\CMakeFiles\ggml.dir\ggml-backend.cpp.obj' with CreateFile()
[106/184] Linking CXX static library examples\llava\llava_static.lib
FAILED: examples/llava/llava_static.lib 
C:\Windows\system32\cmd.exe /C "cd . && C:\PROGRA~2\Intel\oneAPI\compiler\latest\bin\icx.exe /nologo examples\llava\CMakeFiles\llava.dir\llava.cpp.obj examples\llava\CMakeFiles\llava.dir\clip.cpp.obj  -fuse-ld=llvm-lib -o examples\llava\llava_static.lib /machine:x64   && cd ."
icx: warning: unknown argument ignored in clang-cl: '-machine:x64' [-Wunknown-argument]
examples\llava\CMakeFiles\llava.dir\llava.cpp.obj: no such file or directory
icx: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

[NeoZhangJianyu]

@shou692199
If it has relationship with the CCache installation, could you disable it or remove it?
We didn't test the build with CCache.

[shou692199]

After ccache removed, I can successfully build it.
This error never happened when building other backend but sycl.

[NeoZhangJianyu]

OK, looks like the CCache can't support SYCL well. or need some setting.
I don't use it before.

[shou692199]

Now I know what is the root cause of the problem. After manually passing debug=true to ccache options, finding that ccache can't recognize the compile type of icx.exe, showing "Compiler type: other" in log file. By adding compiler_type=icl to ccache options, *.cpp.obj files now being generated succesfully.

[shou692199]

Since the build command is set by ggml\src\CMakeLists.txt globally using set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache), compiler_type should be set to clang-cl considering compatibility for both cl.exe and icx.exe

[shou692199]

@ggerganov Can you check this PR out?

[ggerganov]

I don't think you have opened a PR.

The change in shou692199@8c7c04e seems OK, but it would require testing to make sure it does not break anything. You can open a PR and request feedback.

[shou692199]

@ggerganov Oh sorry. I mentioned you in the wrong place. In fact, a PR has already opened there #9976