If you want to run LM Studio, you need either a Mac with an M1 (or subsequent) CPU, or a Windows system with a CPU that offers support for AVX2 extensions. The requirements for both memory (RAM and hard drive space) are dependent on your selected LLM. Some are relatively small, coming in at just a few GB, whilst larger models might consume substantial storage and memory. While GPU acceleration is currently considered to be experimental, it’s not a necessity in order to operate most LLMs. The newest Mac and Windows installers can be downloaded from https:\/\/lmstudio.ai\/<\/a>, and a version for Linux is currently being trialled.<\/p>\n<\/p>\n LM Studio has a built-in search function that uses the Hugging Face AI community<\/a> API to search for models. For demonstration purposes, I will use Meta\u2019s Llama 2 model. Note that model availability and performance change daily. Enter the keywords llama gguf<\/em> in the search bar. Because Hugging Face also contains models incompatible with LM Studio, I added the GGUF specification. GPT-Generated Unified Format (GGUF) is a file format for LLMs.<\/p>\n<\/p>\n <\/a><\/p>\n LM’s model search interface<\/p>\n<\/p>\n The models you will see are filtered by potential compatibility with your system and ordered by popularity.<\/p>\n<\/p>\n The most popular selection currently is TheBloke\/Llama-2-7B-Chat-GGUF<\/em> as per Meta. Details about the model can be obtained from the Model Card which is a source of information from the developer of the model. This model is available with multiple quantization settings, a technique for streamlining a model by using fewer resources by downscaling the model’s precision.<\/p>\n<\/p>\n In my situation, I will opt for Q5_K_M, which necessitates approximately 8GB of RAM and 5GB of disk space. Select the most suitable option based on your situation and resources. After selection, click Download<\/strong>.<\/p>\n<\/p>\n Inspect the Model Card button to find out more about the model<\/a><\/p>\n Click the AI Chat<\/strong> icon in the navigation panel on the left side. At the top, select a model to load<\/strong> and click the llama 2 chat<\/strong> option. It takes a few seconds to load.<\/p>\n<\/p>\n LM Studio may ask whether to override the default LM Studio prompt with the prompt the developer suggests. A prompt suggests specific roles, intent, and limitations to the model, e.g., \u201cYou are a helpful coding AI assistant,\u201d which the model will then use as context for answering your prompts. Note that accepting a prompt (input) from a third party may have unintended side effects; you can review the prompt in the model settings.<\/p>\n<\/p>\n On the right side, you also have various settings for your specific models, which can be used to fine-tune both performance and resource consumption. Experimental GPU support is available. You can now converse with the downloaded model.<\/p>\n<\/p>\n Chatting with Llama 2<\/a><\/p>\n LM Studio also allows you to run a downloaded model as a server. This is useful for developing and testing your code. The server is compatible with the OpenAI API<\/a>, currently the most popular API for LLMs.<\/p>\n<\/p>\n Click on the Local Server option in the navigation bar. Note that starting a server disables the Chat option. Once again, you can change various features or use a preset, but the defaults should work fine. Click Start Server.<\/p>\n<\/p>\n The LM Server runs on port 1234 by default, and you access the API via HTTP.<\/p>\n<\/p>\nProcuring a model<\/h2>\n<\/p>\n
Searching LLMs in LM Studio<\/h2>\n<\/p>\n
How to Run an LLM on your desktop<\/h2>\n<\/p>\n
Running a local LLM server<\/h2>\n<\/p>\n