Skip to main content

    How to use local models (Vicuna, Alpaca, LLaMa, GPT4All, Dolly, etc.)🦙

    Overview

    Yanok Gab allows you to connect the app with any model you want.

    Requirements:

    • The model must be served via an OpenAI-compatible API endpoint.
    • You must have some relevant technical skills to setup a custom model on your own server/endpoint.

    Below is a short instruction on how to setup Yanok Gab with a popular local AI setup using the open-source project LocalAI.

    Setup LocalAI on your device

    Go to https://github.com/go-skynet/LocalAI and follow their instruction to run a model on your device. For example, here is the command to setup LocalAI with Docker:

    docker run -p 8080:8080 -ti --rm -v /Users/myr/Desktop/models:/app/models quay.io/go-skynet/local-ai:latest --models-path /app/models --context-size 700 --threads 4 --cors true

    Note: update the path "/Users/myr/Desktop/models" to match with your device setup.

    If you're doing it correctly, you should see a message like this:

    LocalAI Setup Success

    Now you can go to http://localhost:8080/v1/models and confirm that the model is ready:

    LocalAI Models Endpoint

    Setup Custom Model on Yanok Gab

    Open Yanok Gab and open the Model Setting button, then click "Add Custom Model".

    Add Custom Model

    Enter all the details, then click "Test" to check if your custom model is working correctly.

    Custom Model Configuration

    Popular problems at this step

    IssueResolution
    CORS related issuesMake sure your server configuration allows the endpoint to be accessible from the browser. Open the Network tab in the browser console to see more details.
    Long waiting timeIn the first request, your model can take a long time to respond. Check the terminal log of the Docker process to see if anything goes wrong.
    API Key MissingYanok Gab does not support API key authentication for custom model yet. Please reconfigure your custom model to remove API key requirement.

    Chat with the new Custom Model

    Once the model is tested and added successfully, you can select the custom model and chat with it normally.