Its basically Cost/Speed/Size Pick 2 or maybe even 1.
Some people have been able to run large LLM's on older slower CUDA gpus. A lot of them are truly ancient and have found their way back to ebay simply due to market conditions. They aren't good, but they work in a lot of use cases.
There are a couple of janky implementations to run on AMD instead. But reviews have been so mixed that I decided not to even test it. Ditto multi GPU setups. I thought that having access to 16 8g AMD cards from old mining rigs would have set me in good stead but apparently it benches roughly the same as just using a server with heaps of RAM because of the way the jobs are split up.
The cloud services seem to be the best option at the moment. But if you are going to spend 100 bucks vs going to spend 1000 bucks it might be worth just to fork out for the card.
Also honestly, hoping that someone else has a better idea in this thread because it will be useful to me too.
I am seeing an average cost of 15k+ on feebay.
I think anyone with 15k could put together a rig with enough VRam to handle a decent model.
The question is more focused on a budget that seems sub 2k.
- You can deploy multiple specialized LoRAs for different tasks
- It massively reduces your train-test latency
- You get upstream LLM updates for "free," maybe you can even add the training to your CI
When choosing research problems it's important to not only follow what's interesting but also what's feasible given your resources. I frequently shelve research ideas because they're not feasible given my skills, time, data, resources etc
They usually give between $1000 and $5000 worth of credits, and they may have other requirements like being enrolled in college, but you should check each of their respective programs to find out more.
For a few thousand dollars you might get some processing power.
But you will never be able to pay your electric bill.
But I just looked again, and A100's are not around at reasonable prices anymore. I cannot even find the old mining equivalents (they used to be everywhere, and CHEAP). Perhaps many people are building similar systems now. After the Ether merge it was a great time to build.