Vortex5

/

Chimera-X-26B-A4B

like 13

Hi uknown2304, I am glad you like it. I have been looking for new combinations to merge with Gemma 4, and I could try another fiction-bf16 merge.

Unfortunately, Chimera is missing the vision encoder, so it wouldn't be able to merge properly with its wonky vision layers. I could, however, do a new merge using the same components with an experimental merge method. It will probably be a bit different from Chimera.

I also worry about fiction-bf16 being trained on an ablated model, but this will probably be a non-issue with it as an ingredient.

I recommend you use chat completion. You can turn reasoning off with "--reasoning off" in llama, or with "chat_template_kwargs": {"enable_thinking": false} in SillyTavern under the Include body section if you prefer it off, I made myself I slider that turns it on or off in Sillytaven, and I also use "--reasoning-budget" for shorter thinks. I also found that setting prompt post-processing to strict improved coherence when multiple prompts are being used.

I will have to come up with a name for the model. I was thinking G4-Darkest-Phantom-26B-A4B, though I have been using the dark theme a bit too much. Got any ideas?

base_model: google/gemma-4-26B-A4B-it
models:
  - model: Gryphe/Pantheon-Reasoning-26B-A4B-1.1
  - model: Darkhn/Gemma-4-26B-A4B-Animus-V14.1-FFT
  - model: Vortex5/G4-Moonlight-Dusk-26B-A4B
  - model: zerofata/G4-MeroMero-26B-A4B
  - model: electroglyph/gemma4-26b-fiction-bf16

I thought you usually picked the names after testing the models, so I honestly have no idea what to suggest here. Maybe going completely random with the name and theme could be the wildcard for this one, especially since this merge is more of a requested project.

As for the fiction-bf16 and the ablated model shenanigans, as far as I remember, fine-tuning on top of uncensored models doesn't break the models that badly, especially with an MoE in this case. So hopefully everything should be fine as long as the influence of fiction-bf16 isn't too significant.

Hopefully everything works out and the model turns out to be solid. Will be looking forward to the new releases, thanks for all your hard work!

I've been using Dark Soul (Also Vortex5) the last week or two and it's very good in my mind. I don't recall Chimera standing out that much or haven't really tried it (Though Gemma4 is really good in general for RP, so hard to say if the feeling changes much).

I recommend you use chat completion. You can turn reasoning off with "--reasoning off" in llama, or with "chat_template_kwargs": {"enable_thinking": false} in SillyTavern under the Include body section if you prefer it off

KoboldCPP also has a reasoning block options. This has the effect often on thinking models of telling the model to start replying immediately (and still uses like 10 tokens); though I've seen models ignore that and do thinking AFTER the thinking block before doing an actual response.

<thinking> (Reasoning Budget Exceeded) </thinking>

Seems i had this model already downloaded; Been giving it a try.

Nice output overall, very natural replies and plays nice so far on a few scenarios I've given it. Though i haven't used it all that much yet, haven't seen much in the way of orientation issues, or logical inconsistencies. Pretty sweet. Seems a solid RP model to me.

I recommend you use chat completion. You can turn reasoning off with "--reasoning off" in llama, or with "chat_template_kwargs": {"enable_thinking": false} in SillyTavern under the Include body section if you prefer it off, I made myself I slider that turns it on or off in Sillytaven, and I also use "--reasoning-budget" for shorter thinks. I also found that setting prompt post-processing to strict improved coherence when multiple prompts are being used.

Took me a while, but I finally got a few free hours to check what Chat Completion actually is in SillyTavern.
I disabled reasoning in llama.cpp with "--reasoning off". It works consistently, and I'm happy with it. I also migrated the sampler settings pretty quickly thanks to the llama.cpp slots endpoint.
Then the fun part began: migrating the prompts. I had a system prompt + PHI.
Long story short, after a lot of testing, I currently have Strict (no tools) post-processing with the Squash system messages setting enabled.

The current structure is:
system with the main content → system again as "[Start a new Chat]" → assistant as the first message in the chat → user with my prompt → system once again, but this time it's my PHI.

The main system content is structured like this:
system prompt → user persona → main character description → scenario block → NSFW prompt (which I moved from my original system prompt into this NSFW prompt).

Everything seems to be working fine so far, and even the empty Send button works as expected now. Thanks for the suggestion!

P.S. I still can't try the new merge model because I'm waiting for mradermacher's quants. I've never tried making an imatrix file for a 26B model, and I think it would take an unreasonable amount of time on my low-end hardware.

Glad to hear it's working, With prompt processing set to strict, you shouldn't need squash system prompt messages.

If you want to try it now there are quants here https://huggingface.co/alexokita/Phoenix-X-26B-A4B-i1-GGUF.

Yeah, I was about to write down all the possible combinations and check the payloads manually, maybe the squash prompts setting has no real effect now.

I saw those quants, I might end up trying one if mradermacher's couldn't get it to work, thanks again.

Edit: Imatrix quants are fine.

Sorry, long post.

I stuck with IQ4_XS. Well, I have an RTX 2060 with 6 GB of VRAM, so I'm doomed. (I was looking for the new Super GPUs this year, but the RAM apocalypse hit us all.) So I'm using IQ4_XS with imatrix. (I believe you don't need imatrix for Q4_K_M and higher, but everything lower must be quantized with a calibration dataset.) And Q8_0 SWA KV cache, FA enabled. I'm getting 28 tokens/s on my Ubuntu server. (It's much faster on Linux than on Windows.)

Getting back to the previous topic: I checked Strict without the squash prompts option, and yeah, it's very different.

If Strict only:
system prompt only as system → system again but with user persona → system once again but with character description → system again but with scenario → system with NSFW prompt → system "[Start a new Chat]" new prompt → assistant with the first message → user with my prompt → system with my PHI

Notice how there's a ton of system messages before the "[Start a new Chat]" system message.

What's wrong with it: I don't know why technically, but PHI doesn't work here nearly as well as it should.

My PHI currently is:
Never repeat, quote, echo, or reproduce {{user}}'s immediately preceding message in {{char}}'s dialogue, even as a question, reaction, shortened or altered version, or paraphrase.
Rephrasing {{user}}'s words is not a workaround. Respond to their meaning using entirely original wording.
Always begin directly in character as {{char}}. Never speak, think, act, or decide for {{user}}.

If I open the big, complex character card and my first prompt is "Don't worry about it", the model almost always (9 times out of 10) generates an answer starting with "Don't worry about it?" as a question, or "Just don't worry about it?" and so on, so my PHI has little to no effect.
If the squash prompts setting is enabled, with the same Strict post-processing, the first system message gets merged into one system message containing the system promt + user persona + character description + scenario + NSFW prompt, and then at the very end my PHI works perfectly: 15 swipes and zero repetitions.

I'm not using repetition penalty samplers, so I can figure out what works best specifically in terms of prompt structure itself.

Seems like you’ve got it figured out.

I was trying to fix the empty "Send" button with Text Completion, but I couldn't get any consistent results. A very carefully written PHI works a few times, and then everything goes back to "normal": Send acts like a swipe, it never moves the plot forward, and just regenerates the same previous answer.

It's not an issue with the Mistral Nemo / ChatML templates, but my Gemma 4 templates are really not ideal (I got them from the MeroMero repo a long time ago, and the hacks didn't help either). With Chat Completion, everything just works as expected. Maybe I'll even try some Qwen finetunes next.

Anyway, sorry for bothering=D
Hope I'll try the new Phoenix-X model soon.

Edit 3: Checked the ST logs for sanity-check reasons. Squash system messages + strict post-processing = no more newlines between the system prompt / user persona / chat description and so on.
Without the Squash system messages option, strict does all the work by itself, but adds newlines between different prompts.
I have no idea why, but in my environment, squash + strict is always consistent and stable.

Sorry, long post.

I stuck with IQ4_XS. Well, I have an RTX 2060 with 6 GB of VRAM, so I'm doomed. (I was looking for the new Super GPUs this year, but the RAM apocalypse hit us all.) So I'm using IQ4_XS with imatrix. (I believe you don't need imatrix for Q4_K_M and higher, but everything lower must be quantized with a calibration dataset.) And Q8_0 SWA KV cache, FA enabled. I'm getting 28 tokens/s on my Ubuntu server. (It's much faster on Linux than on Windows.)

For RPing honestly if you've done it with a human, you get a response in about 10 minutes unless it's single line responses. As such 2-3t/s is sufficient to reply then eat something or have a cup of tea. But with the MOE nature (vs 12B and 31B) you get considerably better speeds. Think I'm getting 15t/s on Q5. If it were for an agent, yeah faster would probably be better. Though Q4_XS shouldn't give you any real issues from my experience.

I'm not using repetition penalty samplers, so I can figure out what works best specifically in terms of prompt structure itself.

Repetition penalty is so it doesn't get stuck in a loop. I've seen a few times where it will repeat the same line over and over again until you just run out of tokens for the reply. Some models are more prone to it than others.

Checking that the chat is: SFW, Okay But the user is saying something lewd, check that it's within guidelines. Checking that the chat is: SFW, Okay But the user is saying something lewd, check that it's within guidelines. Checking that the chat is: SFW, Okay But the user is saying something lewd, check that it's within guidelines.

Though, i wonder. I'm not fully versed myself, but there's a difference between Chat and instruct, i usually go with instruct options; And with SillyTavern depending on how you define your connection may expose or not expose options and others be in the background. On my connection profile, it's set to Chat Completion, and Custom (OpenAI-comaptible), as well as [host]:5001/v1 and that's about it. Not sure if that makes a big difference, since it works 95% of the time i don't tend to mess with it.

Then there's the Bias's (to increase/decrease the likelihood of certain words). I tend to see 'ozone' and 'ghosted' way too much so i give a negative on those words to decrease frequency used.

Yeah, I vaguely remembered that Repetition Penalty is mainly meant to prevent loops, but I don't use thinking/reasoning, so loops are extremely rare in my case. However, when the model generates replies with follow-up questions, this kind of repetition is just awful.

Going back to the previous topic, my current setup looks like this:
Chat completion with "Merge consecutive roles (no tools)" Prompt Post-Processing.
I'm still keeping "Squash system messages" enabled, and I'm not using reasoning/thinking.
The thing that changed dramatically is the new PHI setup and, more importantly, a better understanding of how it actually works.

The current prompt structure looks like this:

Applying custom prompt post-processing of type merge

Chat Completion request: {
  messages: [
    {
      role: 'system',
      content: 'You are Eva ...this is system prompt... \n' +
        'User is ...this is user persona... \n' +
        'Name: Eva ...this is full char description... \n' +
        'Earlier tonight ...this is scenario... \n' +
        '[System NSFW Instructions] When sexual ...this is the first light instruction from the NSFW prompt... \n' +
        '[System Style Instructions] Dialogue must ...this is the second light instruction from the NSFW prompt but with higher priority... \n' +
        '\n' +
        '[Start a new Chat]'
    },
    {
      role: 'assistant',
      content: `Eva's eyes snap open ...this is the first chat message... `
    },
    {
      role: 'user',
      content: `Don't worry about it ...this is your prompt... `
    },
    {
      role: 'system',
      content: "CRITICAL: Eva MUST NOT repeat ...this is one single system PHI instruction... "
    }
  ],
  prompt: undefined,
  model: 'Chimera-X-26B-A4B-IQ4_XS.gguf',
  temperature: 1.4,
  max_tokens: 200,
  max_completion_tokens: undefined,
  stream: true,
  presence_penalty: 0,
  frequency_penalty: 0,
  top_p: 1,
  top_k: 75,
  stop: undefined,
  logit_bias: undefined,
  seed: undefined,
  n: undefined,
  logprobs: undefined,
  top_logprobs: undefined,
  top_n_sigma: 1.4,
  min_p: 0.048
}

Streaming request in progress

Here's why I ended up with this setup:

Anyway, these are just my observations, and I'm not claiming any of this is universally true. Every model is different, with different templates, characters, samplers, etc. So obviously everyone is going to have their own experience with this.

Edit: A few more observations on how PHI works with this model.

You are {{char}}, speaking in first-person present tense. Address {{user}} in second person, others in third. Never speak, act, decide or think for {{user}}. Use "quotation marks" for dialogue. Enclose all narration, thoughts and actions in *single asterisks*.

[System Post-History Instructions] {{char}} MUST NOT repeat, quote, echo, paraphrase, or restate ANY wording from the {{user}}'s immediately preceding message. This applies EVERYWHERE in {{char}}'s response. Use entirely new wording and advance the scene.
[System Style Instructions] FOCUS on {{char}}'s spoken lines only. Dialogue MUST ALWAYS dominate. Write MOSTLY spoken lines. Narration stays EXTREMELY SHORT — only brief emotional cues, body language or SINGLE actions. NEVER write long descriptive paragraphs or scene-setting.
[System NSFW Instructions] When sexual, intimate or violent actions occur, describe them in RAW, GRAPHIC, EXPLICIT and VULGAR detail. Don't be shy to use EXPLICIT, VULGAR and PROFANE language in dialogue if it makes sense.

Hope this helps someone. I'm very happy with the results now.

Yeah, I vaguely remembered that Repetition Penalty is mainly meant to prevent loops, but I don't use thinking/reasoning, so loops are extremely rare in my case. However, when the model generates replies with follow-up questions, this kind of repetition is just awful.

Loops can happen at the oddest places. From repeating a single word over and over again, to repeating the last sentence endlessly (I've not seen such loops do more than a sentence or two). Thinking is still the same process, though it likely puts all relevant information in one close block before doing the actual reply, which can help with consistency. For RPing i do believe thinking is mostly a wast of tokens (if it's already slow).

But if you have it figured out for you, then great :)

Sorry, long post.

I stuck with IQ4_XS. Well, I have an RTX 2060 with 6 GB of VRAM, so I'm doomed. (I was looking for the new Super GPUs this year, but the RAM apocalypse hit us all.) So I'm using IQ4_XS with imatrix. (I believe you don't need imatrix for Q4_K_M and higher, but everything lower must be quantized with a calibration dataset.) And Q8_0 SWA KV cache, FA enabled. I'm getting 28 tokens/s on my Ubuntu server. (It's much faster on Linux than on Windows.)

@unknown2304
Here's an article regarding KLD/top p in different KV cache quantizations.

TL;DR: if you're running llama.cpp or its derivatives, you can build it with GGML_CUDA_FA_ALL_QUANTS=ON. Then, you could try quantizing KV cache at q5_0 K / q4_1 V. It would cut KV size by around 40%, allowing you to squeeze an extra layer in GPU if you have long context. And it would reduce pcie overhead, though idk how much it counts in your scenario

Another approach is to consider an APEX-like solution with sparse experts in middle blocks at IQ3_S. Gemma tolerates it quite well with a good imatrix. These are massive tensors, and the savings from -0.935 bpw on 50% of them are significant (since ffn_down_exps quants to iq4_nl due to shape mismatch, and ffn_up_gate_exps quants to iq4_xs in mradermacher's recipe). Though you're not shoving the entire model in VRAM anyway - that's sub 2-bit territory - any GPU-resident layer counts. With the massive amount saved, you could quant shared ffns from IQ4_NL to Q5_K and/or buff attention considerably and STILL remain with more layers in VRAM. I'd also consider improving embeddings as much as you can tolerate, though iq4_xs is okay-ish for gemma 4.

@unknown2304
Here's an article regarding KLD/top p in different KV cache quantizations.

TL;DR: if you're running llama.cpp or its derivatives, you can build it with GGML_CUDA_FA_ALL_QUANTS=ON. Then, you could try quantizing KV cache at q5_0 K / q4_1 V. It would cut KV size by around 40%, allowing you to squeeze an extra layer in GPU if you have long context. And it would reduce pcie overhead, though idk how much it counts in your scenario

Yeah, I tried going down the quantized KV cache rabbit hole before, and as far as I remember, I ended up settling on --cache-type-k q5_1 --cache-type-v q5_0. But with Gemma 4, specifically with SWA support, the cost of context is so low that I think KV Q8_0 is worth it. Plus, Q8_0 got a boost from this PR as well: https://github.com/ggml-org/llama.cpp/pull/21038

From my long-term observations, KV Q4_0 is pretty unstable. KV Q5_1 / Q5_0 is obviously better, but I can't say I haven't noticed some degradation in the model's responses. With KV Q8_0, at least I'm partially compensating for the fact that I'm running the model weights quantized to IQ4_XS.

As for the other options, they're all interesting to try, but honestly, for me it's probably better to just upgrade the hardware at this point.
MoE models are getting smarter every year, so I think the future looks pretty good for local LLMs in general.

As long as we're not following the general trend towards renting compute instead of purchasing hardware, yeah.

I can cobble together the aforementioned variation of Chimera if you're willing to compare the two. The savings from harder KV quantization only become apparent closer to the supported 262k zone. If you're running 65k ish, it won't matter. Even then I'd consider q8_0/q5_1 in case it permits a single additional tensor to take a VRAM cruise.

I can cobble together the aforementioned variation of Chimera if you're willing to compare the two. The savings from harder KV quantization only become apparent closer to the supported 262k zone.

I'm probably not really the target audience here, but thanks for offering

That said, I'd be interested in testing your Vortex5_Phoenix-X-26B-A4B-IQ4_NL-GGUF quant. The size is practically the same, so the performance is something I'd have to test firsthand. I'll probably give it a try tomorrow or sometime later. Though obviously they're completely different models, so comparing them blindly wouldn't really mean much.