Install dependencies:
brew install libomp vulkan-headers glslang molten-vk shaderc vulkan-loader
Now Molten-VK is already above v1.4 -> therefore no need to get older version and manually apply PR2434.
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -DLLAMA_CURL=1 -DGGML_METAL=OFF -DGGML_VULKAN=ON
I found a way to run llama.cpp with GPU support on my 2019 Intel MacBook Pro, with the help of Vulkan.
Thanks to ollama/ollama#1016 (comment) an following comments which helped me find the right build flags.
First download MoltenVK-macos.tar (I used v1.3.0-rc1 at the time) from https://github.com/KhronosGroup/MoltenVK/releases
Not sure all these packages are required, but I ran brew install libomp vulkan-headers glslang molten-vk shaderc vulkan-loader.
You probably also need to install cmake and potentially other tools I already had installed on my system.
Then clone and build llama.cpp (replace /path/to/MoltenVK_macos with the actual path you have MoltenVK_macos):
👉 ipatool is an open-source tool developed by Majd, a highly trustworthy and talented developer in the iOS community. Recently, ipatool got a significant update that allows users to easily download older versions of iOS apps on macOS/Windows/Linux!.
👉 Since ipatool doesn't have a graphical user interface (GUI), some of you might think it's tricky to use. But trust me, it's not! Here's a simple guide if you're still a bit scared of the terminal. (Tbh, everything in this *guide can be found on ipatool's repo)
👉 Note: You need to log into your Apple ID via ipatool for the tool to work. Unless you prioritize security above all, you can trust logging into your Apple account with ipatool. As explained earlier, it’s an open-source tool developed by a well-known and reliable developer, minimizing security risks to the lowest level.
| # Ubuntu 22.04 | |
| # wsl --install Ubuntu-22.04 | |
| # rocm 6.3 is not compatible with 7800 | |
| # rocm 6.4 appears to be compatible with 7800 | |
| # official documentation (https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/wsl/install-radeon.html) use rocm 6.3, this note explains how to perform the same installation but with rocm 6.4 (or any other version) for 7800 compatibility | |
| # basic ubuntu prep | |
| sudo apt update | |
| sudo apt upgrade |
| #!/usr/bin/env bash | |
| # Default values for percentages | |
| DEFAULT_WIRED_LIMIT_PERCENT=85 | |
| DEFAULT_WIRED_LWM_PERCENT=75 | |
| # Read input parameters or use default values | |
| WIRED_LIMIT_PERCENT=${1:-$DEFAULT_WIRED_LIMIT_PERCENT} | |
| WIRED_LWM_PERCENT=${2:-$DEFAULT_WIRED_LWM_PERCENT} |
Just some tips I gathered over time. All in one easily reachable place so I can share it wherever I want.
Please note that unless you see a shebang (#!/...) these code blocks are usually meant to be copy & pasted directly into the shell. Some of the steps will not work if you run part of them in a script and copy paste other ones as they rely on variables set before.
The { and } surrounding some scripts are meant to avoid poisoning your bash history with individual commands, etc. You can ignore them if you manually copy paste the individual commands.
I chose to write things "in the open" that way so there's still some control and things don't become a black box.
Backup cluster-wide configuration virtual filesystem in a safe manner, plan for disaster recovery for the case of corrupt database. A situation more common than anticipated.
See maintained post: https://free-pmx.org/guides/configs-backup/
NOTE No tracking on the site. Full and current reStructuredText version linked from within.
| 1) Close Parallels completely | |
| 2) In Terminal, type : sudo nano /Library/Preferences/Parallels/network.desktop.xml (this will open the Terminal text editor "nano") | |
| 3) Type your admin password to access the file | |
| 4) Locate this line <usekextless>-1</usekextless> it could be also value 1 and replace whatever value there with 0, so you should have <usekextless>0</usekextless> | |
| 5) Press Ctrl+O to ask for a save, then Ctrl+X to close the editor | |
| In Terminal, type : sudo nano /Library/Preferences/Parallels/dispatcher.desktop.xml | |
| find <usb>0</usb> and change 0 to 1 | |
| again Press Ctrl+O to ask for a save, then Ctrl+X to close the editor | |
| 6) Close Terminal and open Parallels like normal, your VM should load like on Catalina and before |
Currently, it seems as if Fedora doesn't install udev-rules, to get this working, I've had to manually install them from a community project. I'm not very knowledgable in Linux, these are just the steps I had to do to get things working that I'd like to document.
Install android-tools from DNF.
sudo dnf install android-toolsGood question! I am collecting human data on how quantization affects outputs. See here for more information: ggml-org/llama.cpp#5962
In the meantime, use the largest that fully fits in your GPU. If you can comfortably fit Q4_K_S, try using a model with more parameters.
See the wiki upstream: https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix
