๐ค Ollama
Ollama is a framework for running large language models (LLMs) locally on your machine. It lets you download, run, and interact...
For further actions, you may consider blocking this person and/or reporting abuse
bro really thank for making this tut bro.
thank you @ajmal_hasan for Sharing ,will give it a try ๐
Would have been better if you mentioned system requirements too
Itโs a 1gb file. Llmโs like to sit in your gpu. So a 2gb graphics card should run it. Obviously it will not be as fast as a 4060 8gb with lots of cuda cores. But if you read other articles about this llm itโs designed to work on less resources
What are the hardware's requirements?
Why not starting with this, at the beginning of your guide?
Any decent system will suffice (for example, I use a MacBook M1 base model). Choose the light model available, if not having high end device.
However, keep in mind that processing time and response quality will vary based on your system's specifications and the complexity of the model parameters. ๐
@maneamarius , Asking what the system requirements should be for an LLM is like asking what the horsepower should be for a car: It depends. We're talking about tools with a wide range of applications, so the minimum requirements depend on an individual's desired outcomes.
As they say, you attract more flies with honey than vinegar. Instead of criticizing a guy who's educating you and others for free, try asking him something like, "What are your system specs, and how many tokens per second are you getting?"
Not a good answer.
You should put the recommended system requirements in your post, for each model.
e.g. graphics cards needed, etc..
Otherwise your post is incomplete.
What about giving it a try before criticizing the author?
As mentioned, the 1.5b model is rather small. The download is "just" 1.1 Gigabyte. I was able to run it on a MacBook Pro 2 with only 16GB of RAM, and it was answering with decent speed consuming about 4G RAM usage.
The real limitation is the 1.5b model. I asked it to generate Rust code, and it admitted to not knowing it very well.
I then switched to the deepseek-coder-v2 model with 16b parameters, and that's a download of 8.9 Gigabytes. RAM usage spiked to 8G, and the model is operating at a lower speed and uses less reasoning but instead started to emit code directly to my question.
So, Ajmal's answer is that a decent system will be enough to generate your answers. I agree with this, as I would consider my Mac, due to RAM limitations, not as good, but decent. And, of course, it depends on what you are running besides the LLM. If your RAM is already filled up, you'll get into trouble.
However, you do not need a 4090 and many Tensor Cores to run these models locally. Your mileage may vary, true. But overall, and to get a first impression, it will definitely work.
Just give it a try, the text shows all the necessary steps to do this. Except for
ollama serveyou will find out by looking at the messages and the help.I found this rule of thumb in a youtube video by bycloud
If your gpu's vram is greater than (model_size * 1.2) then you can run that model
Iโd double check your claim of DeepSeek R1 local deployments being โโ 100% Local & Secureโ - wouldnโt be the first to reach out to the wider net.
I caveat this with; you are however 100% in control of a local modelโs resource access.
My apologies if this is what you meant; not explicitly called out so wasnโt aware
Hi @ajmal_hasan, how to get around from the error: requests.exceptions.SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /sentence-transformers/all-mpnet-base-v2/resolve/main/adapter_config.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (ssl.c:997)')))"), '(Request ID: edeffbec-e8a2-472e-9722-2c40df75aa94)')
2025-01-29 21:55:58.668 Examining the path of torch.classes raised: Tried to instantiate class 'path._path', but it does not exist! Ensure that it is registered via torch::class
Just wanted to confirm what specs it can run -
Ollama DeepSeekR1:14B runs smoothly and quickly on an Ryzen 7 5700x, 64GB, 3080RTX 10GB. The 32B and 70B run but the 70B thinks 1 word a second while the 32B is slightly faster.
I've used the 70B but had to let it run to provide info the next day (late at night). Just fyi if time is of no issue it will run Ollama and even the chatapp. Have not tried RAG but shouldn't be an issue.
My Laptop has 4 CPU cores, 16GB RAM with Intel integrated Graphics (Ubuntu) - will it work on my Laptop?
Yes, but not as fast as if you had a GPU. You also will need to use 7B or smaller model.
Try it. Itโs a light weight model.
I tried running it dunno why but it gave me garbage text back
Use higher parameters version if your system supports it.
Sadly I don't think I can I have 8gb ram
Itโs a small model. And will rely on your gpu. 2gb of gpu power will be enough to get started. Obviously it wonโt be as fast if you have a more modern card. I use a 4060 with 8gb of ram. Mainly because it has a lot of cuda cores and uses way less electricity.
What if I want to use UTF8 txt files?
Would love to try it.
Thank you!
Can you share a TypeScript version of that?