How to setup AI chatbot on your computer

This article will explore the process of setting up and operating an AI chatbot on your local computer, enabling its use without the need to disclose your data to external third-party entities. The same principle can be applied if you intend to deploy a chatbot for organizational use, whether on private or public cloud solutions. However, a thorough discussion of this topic goes beyond the scope of this article.

AI Chatbots utilize Language Models (either large or small) as their foundational technology. To set up one, a computational environment capable of running the requisite pre-trained model is imperative. In this example, we employ Meta’s Llama 3.1 with 8 billion parameters (8B) model, executing it on a MacBook Pro equipped with the M1 Pro chip and 16 GB RAM. The choice of the model depends on the computational capacity of the host system and the desired output accuracy. In instances of lower computational capability, opting for a lighter model, such as the Llama 3.1 8B, is advisable. Conversely, more potent systems can accommodate larger models, such as the 70B or 405B variants.

Setting up the Model

There are various methods available for downloading and running a model on your computer. For the sake of simplicity, we will employ Ollama to facilitate the execution and management of the models. As of the time of writing this post, Ollama is presently compatible only with Mac and Linux operating systems, with forthcoming support for Windows. However, it is accessible through Docker, enabling utilization of the Docker image until the native installer becomes available for Windows.

1. Download Ollama from their website — https://ollama.ai/
2. Run the downloaded file and follow the instruction.
3. Once the installation is complete, you can confirm that Ollama is running by either running the command ‘ollama’ in your Terminal or by hitting this URL on your browser — http://localhost:11434/

4. In your terminal, run the below command to download the Llama 3.1 8B model

ollama run llama3.1:8b

Once the download is complete, the model should initiate on its own. You should now be able to input prompts and receive answers from the model in the CLI. You can execute the same command again to run and subsequently interact with the model.

To run other variants of Llama or a different model altogether, and for additional configuration details, please refer to their README — https://github.com/ollama/ollama

Using ChatGPT-like web interface

If you are seeking a more convenient way to interact with the model than using the CLI, ollama-webui allows you to use Ollama in your web browser.

Follow their detailed instructions to download and set it up on your computer.

https://github.com/ollama-webui/ollama-webui

Note: It is important to note that this is not a replacement for ChatGPT. The performance, creativity, and capability of the chatbot depend on the model it utilizes. ChatGPT employs GPT-4o and higher models, which are trained on a different set of data. Thus, its performance and creativity differ from Llama 3.1. There are detailed benchmarks and studies comparing various models, which you can refer to for further details. Depending on your requirements and interests, choose the model that works best for you.