LuaJIT is a better LLM runtime than Python
The title is clickbait, forgive me. You clicked, time to read :D Not really sure how to start this blog post, so I'm just gonna wing it. I have 32 GB of RAM, a Ryzen 9 9950X and an RTX 3060. Not enough to run 14B in Q8 without CPU offload, and definitely not enough to let Python eat half my memory before the model is even loaded. I've been writing Lua for nearly 14 years, and I have a strong taste for tools that don't leave a huge footprint on my system: few packages, little RAM, little no...
Original Source
Read the full article at Dev →KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.