It’s been a fascinating experience so far and I feel like I’m only touching the surface. I’m exploring some of the various memory harnesses to hook into Hermes Agent and see how it learns with extended use.
- 1 Post
- 4 Comments
Joined 3 years ago
Cake day: June 16th, 2023
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
Currently I’m running a Q6K quant of Hermes 4 14B with a 32K context window via llama.cpp that works pretty well. Generation output is a comfy ~50tok/sec. These v100s are 16GB each, but there are 32GB versions available too.
I’m running everything via NixOS and have to do package overrides to get inference engines to build with the right CUDA versions.
My goal is to get a cohesive environment set up for Hermes Agent to learn my system/lab/network and help my grow it over time.
Overall, I’m happy with them. The mezzanine board is good quality, I’m using PTM sheets under those massive heatsinks and some arctic p9 fans to keep them at around 60C under load.
It’s an older version of a heatset insert fixture. I used it to put threaded brass inserts into my 3d prints instead of trying to capture a nut or screwing directly into plastic.
https://www.printables.com/model/609644-stealth-press-1-heat-set-insert-press-legacy
I bet you used a keyboard, instead of a magnetic needle and a steady hand to code this too.
/s
For real though, nice work. 🎉