Make refiner switchover based on model timesteps instead of sampling steps
diff --git a/modules/sd_samplers_cfg_denoiser.py b/modules/sd_samplers_cfg_denoiser.py
index a73d3b03692..93581c9acc6 100644
--- a/modules/sd_samplers_cfg_denoiser.py
+++ b/modules/sd_samplers_cfg_denoiser.py
@@ -152,7 +152,7 @@ def forward(self, x, sigma, uncond, cond, cond_scale, s_min_uncond, image_cond):
         if state.interrupted or state.skipped:
             raise sd_samplers_common.InterruptedException
 
-        if sd_samplers_common.apply_refiner(self):
+        if sd_samplers_common.apply_refiner(self, sigma):
             cond = self.sampler.sampler_extra_args['cond']
             uncond = self.sampler.sampler_extra_args['uncond']
 
diff --git a/modules/sd_samplers_common.py b/modules/sd_samplers_common.py
index 6bd38e12a41..045b9e2fe06 100644
--- a/modules/sd_samplers_common.py
+++ b/modules/sd_samplers_common.py
@@ -155,8 +155,17 @@ def torchsde_randn(size, dtype, device, seed):
 replace_torchsde_browinan()
 
 
-def apply_refiner(cfg_denoiser):
-    completed_ratio = cfg_denoiser.step / cfg_denoiser.total_steps
+def apply_refiner(cfg_denoiser, sigma):
+    if opts.refiner_switch_by_sample_steps:
+        completed_ratio = cfg_denoiser.step / cfg_denoiser.total_steps
+    else:
+        # torch.max(sigma) only to handle rare case where we might have different sigmas in the same batch
+        try:
+            timestep = torch.argmin(torch.abs(cfg_denoiser.inner_model.sigmas - torch.max(sigma)))
+        except AttributeError: # for samplers that dont use sigmas (DDIM) sigma is actually the timestep
+            timestep = torch.max(sigma).to(dtype=int)
+        completed_ratio = (999 - timestep) / 1000
+
     refiner_switch_at = cfg_denoiser.p.refiner_switch_at
     refiner_checkpoint_info = cfg_denoiser.p.refiner_checkpoint_info
 
diff --git a/modules/shared_options.py b/modules/shared_options.py
index bb3752ba65f..e17eed512a8 100644
--- a/modules/shared_options.py
+++ b/modules/shared_options.py
@@ -227,7 +227,8 @@
     "dont_fix_second_order_samplers_schedule": OptionInfo(False, "Do not fix prompt schedule for second order samplers."),
     "hires_fix_use_firstpass_conds": OptionInfo(False, "For hires fix, calculate conds of second pass using extra networks of first pass."),
     "use_old_scheduling": OptionInfo(False, "Use old prompt editing timelines.", infotext="Old prompt editing timelines").info("For [red:green:N]; old: If N < 1, it's a fraction of steps (and hires fix uses range from 0 to 1), if N >= 1, it's an absolute number of steps; new: If N has a decimal point in it, it's a fraction of steps (and hires fix uses range from 1 to 2), othewrwise it's an absolute number of steps"),
-    "use_downcasted_alpha_bar": OptionInfo(False, "Downcast model alphas_cumprod to fp16 before sampling. For reproducing old seeds.", infotext="Downcast alphas_cumprod")
+    "use_downcasted_alpha_bar": OptionInfo(False, "Downcast model alphas_cumprod to fp16 before sampling. For reproducing old seeds.", infotext="Downcast alphas_cumprod"),
+    "refiner_switch_by_sample_steps": OptionInfo(False, "Switch to refiner by sampling steps instead of model timesteps. Old behavior for refiner.", infotext="Refiner switch by sampling steps")
 }))
 
 options_templates.update(options_section(('interrogate', "Interrogate"), {

## Description

* This makes the default behavior of refiner switchover aligned with model timesteps instead of sampling timesteps.  This is easier to use, since setting refiner switchover to 0.8 (for a refiner trained on the last 200 timesteps like SDXL's) will now always switch to the refiner as soon as we're sampling from timesteps the refiner was trained on, where previously using img2img or different noise schedules could lead to unexpected model behavior (examples below).
* I changed this to work off of the existing refiner switchover slider.  The new default behavior will be to derive timesteps from sigma (or in DDIM's case take them directly) and use that to determine whether it is time to switch over.  Old behavior is supported by a compatibility option.
* Implements #14970
* ~~edit: converted to draft while I troubleshoot an off-by-one error~~ There is a bug where model alphas_cumprod changes (compatibility casting option or zero terminal snr) are reverted during the timestep the refiner is applied.  This will have to be fixed separately.

## Screenshots/videos:

Examples of old behavior in txt2img:
![xyz_grid-1367-36463525-(best quality, high quality,_1 5) by strange-fox, solo, anthro, male, rat, manly, clothed, jacket, shirt, detailed background, n](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/1313496/c0102dc4-6576-4c31-8554-b12dee818c20)
The refiner model used here was trained for the last 200 timesteps. The Karras schedule type, especially on zero snr, drastically changes the model timesteps called during this 50 step sampling process, which results in the refiner being switched to too early on what is actually the correct setting on the default noise schedule.  The effectively correct setting for Karras samplers is 0.88 for this refiner under the old configuration.

Now for the fixed version:
![xyz_grid-1368-36463525-(best quality, high quality,_1 5) by strange-fox, solo, anthro, male, rat, manly, clothed, jacket, shirt, detailed background, n](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/1313496/74aa418a-ef5c-4bd0-8d33-196c57ae1990)
With this fix, the behavior of the refiner is consistent with the same settings across different schedules, and it no longer triggers too early.  0.8 is reliably a correct setting.

Examples of old behavior in img2img/inpainting (inpainting mask is over the head, adding a hat, 0.75 denoising strength):
![xyz_grid-0028-691039214-(best quality, high quality,_1 5) by strange-fox, solo, anthro, male, rat, manly, clothed, jacket, shirt, hat, detailed backgrou](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/1313496/a37e01b2-f65f-4b17-98fe-645e7a3eeea0)
This one is more complicated, and the differences are subtle.  The effectively correct settings for the normal schedule is to switch over at 0.75, and for Karras it is correct to switch over at 0.85.  Using the expected setting of 0.8 therefore is too late for normal schedules and too early for Karras ones.  As denoising strength gets lower, the problem becomes more severe.
![xyz_grid-0029-691039214-(best quality, high quality,_1 5) by strange-fox, solo, anthro, male, rat, manly, clothed, jacket, shirt, hat, detailed backgrou](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/1313496/c9824488-4e0d-4ace-849c-9147f1ea809a)
This grid shows the behavior after the fix.  Switch at 0.8 is now correct for both.


## Checklist:

- [x] I have read [contributing wiki page](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Contributing)
- [x] I have performed a self-review of my own code
- [x] My code follows the [style guidelines](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Contributing#code-style)
- [x] My code passes [tests](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Tests)
