top of page

    Setting up your AI LLM on your Local Computer Using Ollama

    Ollama is a free software that allows easy running of open weight LLMs on your local computer. They may not be as powerful as Anthropic's Fable 5 frontier model, but they are still quite capable and free.

    ​

    Start by downloading and installing Ollama from ollama.com/downloads

    ​

    Then run Ollama and you should see a screen like this:

    Ollama selector.png

    Next, on Windows, press your 'windows' key and run CMD (on Mac run Terminal). You then just have to run one command: ollama pull [model]​. In this example, I'm pulling down gemma4:12b (a good model for most computers) and it will take a few minutes to complete:

    Ollama pull.png

    If you have a dedicated GPU with 12+GB of VRAM, I recommend also gemma4:26b It can notably improve the quality of the AI results.

    ​

    Next, in Ollama, go to settings, and 1) select 'Cloud' (enable cloud models and web search) and 2) slide the 'Context length' to a higher value (128k or 256k).

    ​​

    Finally, if you're running windows with a dedicated GPU, it is smart to specify that Ollama should use the GPU, rather than the CPU:

    • Press your 'windows' key and run 'Graphics Settings'

    • Click the "Add Desktop App" button and navigate to C:\users\[user name]\AppData\Local\Programs\Ollama\ and select ollama.exe.

    • Do the previous step again for ollama app.exe.

     

    When complete, expand the two apps in the Graphics Settings view and they should look something like this (I have selected my RTX 4070 GPU):​

    Graphics Settings.png

    You can now close Ollama, or go back and experiment as if it's a local chatbot.

    ​

    In the AI Digital Bookshelf, go into Settings (the gear icon at the top right).

    • Scroll to the bottom of the Settings menu until you find "Local AI Configuration"

    • Click "Refresh" and then select your local model from the drop down.

    • Scroll up to the top of the Settings menu and you can set your Insight Gen Model (for cross-book analysis) and/or your Single Book AI Model to 'Local'.

    ​​

    Now you can run the AI features locally on your computer without requiring any cloud LLM tokens or costs.​​

    ​

    About the developer

    Contact with bugs, issues, questions, or other feedback

    Back to main

    bottom of page