Thermostat

Hey, Jeff here. I've been messing with what is essentially a while loop that simulates dynamic, intra-response temperature changes in an experiment I called Thermostat. Yes, it's a pun. Thermostats change the temperature of the room and my little orchestration loop enables LLMs, with a little elbow grease, to change their own temperature.

If you have a huge one-shot prompt with an ideation phase, a concrete example phase, and a summary phase, you probably would want the ideation phase to be more creative but the example and summary phases to be less so. However, when calling an LLM, each response is generated with a single temperature parameter set by you, the user. This, naturally, raises the question: what if the model itself specified what the optimal temperature was for each section? What if the model could be like Sean Paul, dynamically adjusting its own temperature accordingly as it generated parts of the entire response?

"The way the time cold, I wanna be keepin' you warm.
I got the right temperature fi shelter you from the storm."

This idea actually came to me via my wife, who asked me why temperature has to stay the same for an entire response. We were watching yet another video on how LLMs work and they were going over temperature as it related to token prediction likelihood. "Could you change it for different parts?", she asked.

I'd never thought about that. So I built this.

In many user-friendly guides to AI and 'How LLMs Work' videos, temperature is colloquially referred to as the "creativity" or "randomness" level of the response. From a mathematical perspective, however, it's really just a probability distribution.

When a model's temperature is high, the distribution for the possibilities for each token flattens: more tokens stay in play and are closer in probability to be selected as the next token. When the temperature is low, the distribution becomes more spikey: a few tokens remain extremely likely to be picked, while the likelihood of the others are reduced dramatically. It's important to note that, regardless of temperature, the model still picks exactly one next token at a time and only one token is ultimately selected. Temperature only changes how that choice is distributed among possibilities.

Do You Believe in Magic, in an LLM's Heart?

As it stands, temperature is yet another AI-related concept that is inappropriately anthropomorphized, much to the chagrin of those who beg people to stop treating AI like a sentient friend. And yes, I realize that with explanations like these, I am not helping. So, to maintain my tech credibility, I must emphasize: none of this is magic and turning up the temperature isn't giving your model an 'imagination'. As always, it's just math, all the way down. However, that math does loosely map onto what people might view as "creativity": high temperature increases the chance you sample a less-obvious (but still plausible) continuation instead of always taking the safest or most obvious one.

Technical Aside

"Creativity" is what users see. Under the hood, temperature is a divisor inside the softmax that normalizes logits into probabilities[1]:

Low T (< 0.5): The distribution peaks sharply. The highest logit dominates, leading to deterministic output[2].

High T (> 1.0): The distribution flattens. Lower-probability tokens gain statistical significance, increasing entropy[2].

That's why higher temperatures feel 'creative': you're giving unlikely tokens a real chance instead of crushing them with the otherwise-unstoppable force of high predictability.

So the question is: can a model change its own temperature mid-response? Could it suddenly self-reflect and decide, "you know what, I am not giving myself enough options in predicting this stuff" or "I'm being too loosy goosey with my words" and adjust accordingly. And yes, I realize models don't 'decide' anything; such a mechanism would, yet again, be the model probabilistically determining to predict some token that corresponds, in our world, to the model's temperature being adjusted.

As far as I know, the way most inference loops are written does not allow for this. So, if you can't do it inside the model... where can you do it? I've already established that temperature is a parameter that amounts to controlling the math that shapes the probability distribution for an entire response, so I'll have to be a bit more creative to answer that.

There are a few facts we must know to even begin to intuit such a solution. Number one: with the OpenAI completions API, you can interrupt a model mid-response, send it another message, and have the model retain the context of what it said so far. Number two: each time you call a model, you can specify any temperature, even if prior tokens in the context were generated at a different temperature. If you put these two facts together, perhaps you see the direction my thinking is heading. When imagining how such a system would work, two approaches immediately sprung to mind.

Thermostat uses the stitching approach. The top-down method has its merits and is, arguably, the true 'steering'-based approach (see my article). It's predictable, easier to debug, and guarantees coverage of planned sections, all of which are invaluable for long workflows. However, it front-loads decisions that might be better made in context and, most importantly, doesn't achieve my goal of simulating a true "one-shot" response with mid-stream temperature changes.

With the approach decided, let's look at how to actually build it.

The Stitching Trick

Public APIs don't give you true intra-call temperature changes. You'd have to tweak sampling mid-inference, and providers just don't expose that, as it's not likely something they could even do from their end. So I cheat: Thermostat stitches multiple calls (with different temperatures) into one output. Through a rigorous system prompt and by including the full response pre-temperature change, I attempt to preserve continuity between calls.

1. The Tool Definitions & Schema Hacking

Temperature is an API parameter. You can't tell the model to "stop and change temperature", because the knob to do so lives outside the model, at the API level. Instead, the system prompt tells the model: "Use the request_sampling_update tool when you want a different temperature."

I also need a finish_response tool, because relying on implicit stop tokens in a stitched stream is a recipe for bugs.

def generate_openai_tools() -> List[Dict[str, Any]]:
    """Generate OpenAI function-calling tool schemas."""

    return [
        {
            "type": "function",
            "function": {
                "name": "request_sampling_update",
                "description": (
                    "Adjust sampling temperature when your writing "
                    "phase changes. Raise for creativity/brainstorming, "
                    "lower for precision/code/facts. "
                    "When in doubt, make the change."
                ),
                "parameters": update_schema,
            },
        },
        {
            "type": "function",
            "function": {
                "name": "finish_response",
                "description": "Signal that the response is complete.",
                "parameters": finish_schema,
            },
        },
    ]

2. Why Tool Calls Are Perfect for This

In my system prompt, I had two options for telling the Thermostat loop to change temperature:

1. In the system prompt, instruct the model to emit some sort of special token (e.g. __CHANGE_TEMP_TO_X.X__) in its regular output. In my code, this token would be interpreted by the loop as a temperature change.
2. Create a synthetic tool definition, which the system prompt instructs the LLM to invoke, that changes the temperature.

The token approach has some glaring issues, but as with all good design docs, it is useful to articulate them even if I knew I'd go the other route:

1. If the code relied on the response itself to have my control token, it would have to constantly scan the LLM response stream to catch it mid-stream and quickly terminate the LLM, potentially after further tokens were (incorrectly) generated.
2. There is an inherent messiness in ensuring the code properly detects and parses the token, which is increasingly likely to be malformed...especially at higher temperatures. Tool calls, on the other hand, are much less likely to be garbled or malformed, even at higher temperatures.

These issues alone made the tool call approach the natural choice, but tool calls also offer benefits that extend well beyond avoiding the token approach's flaws and align with how models actually work in the agentic age.

3. Tool Calls Are Just Tokens, But With A Learned Behavior Advantage

At the model level, tool calls are just tokens[5]. The model predicts tokens like <|tool_call|> or {"name": "function_name" the same way it predicts any other token, through the same softmax over the vocabulary. There is no special "higher-level" decision happening inside the model's forward pass that affords tool calls some 'special status'.

However, models are trained (via RLHF, instruction tuning, etc.) to recognize when tool calls are appropriate, so they've learned patterns that make tool call tokens more likely in certain contexts. This is learned behavior, not architectural privilege, but as models have become 'better' at tool calling, this behavior has become more pronounced. This is especially apparent when you compare models that are 'good' at tool calls (they are invoked at proper times, have official API support rather than being intertwined with output, are rarely malformed) with ones that are 'bad' at tool calls (e.g. Gemini 2.x series). When the context matches patterns the model saw during training—like transitioning from open-ended discussion to a concrete task—tool call tokens become more probable because those patterns frequently preceded tool calls in training data. Importantly, this learned behavior is strong enough that tool calls still reliably fire even at higher temperatures, where the flatter distribution might otherwise let alternative tokens compete.

The "higher-level" logic is external: the inference loop and orchestrator code treats tool calls specially, intercepting them, parsing them, executing them, and modifying state before resuming generation. While this distinction doesn't exist in the model proper, tool calls occupy a privileged position in the inference ecosystem, acting as action boundaries where generation pauses and external logic takes over. This makes them a natural mechanism for signaling "I want to change my behavior" rather than "here's more content."

There's also a practical benefit baked into how these APIs work: streaming automatically stops when a tool call is emitted[3]. After a tool call, the API pauses, returns the structured tool call data, and waits instead of generating more tokens. This is exactly what I need: a clean boundary where I can inspect the request, apply the new temperature, and resume generation. No racing to catch tokens mid-stream and having to discard over-generated content. The tool call "token", even though it's just a token to the model, is afforded special treatment by the infrastructure[4] that makes it perfect for this task.

4. Temperature Affects Its Own Adjustment Probability

Special API treatment aside, this still means the decision to change temperature emerges from the same attention mechanism that decides everything else. When the model senses a phase shift (from creative brainstorming to technical implementation, or from factual recitation to imaginative exploration), that shift naturally surfaces as an increased probability of the tool call.

The current temperature influences the likelihood of a temperature change. There's no way to isolate "temperature for deciding to change temperature" from "temperature for everything else." It's all-or-nothing. For it to be anything else would defeat the whole point of Thermostat: letting the model decide when and how much to adjust its own sampling.

Higher temperature → more randomness → more likely to call the tool. Lower temperature → more deterministic → less likely to call the tool. A feedback loop where the current temperature setting influences the model's propensity to adjust that very setting.

Tool call probability isn't purely random. It's contextually informed by the learned behavior we discussed above. Even at lower temperatures, if the context strongly suggests a tool call is appropriate (e.g., a clear phase shift from creative brainstorming to technical specification), that learned behavior can still push the tool call probability high enough to win the softmax. But a sufficiently low temperature can "lock in" the model to continuing generation even when a change would be appropriate. This is a limitation inherent to the way LLMs work, not a bug to be fixed.

5. Requiring Justification for Temperature Changes

On the high temperature end of the spectrum, I can somewhat mitigate this issue by forcing the model to ground its decision to change the temperature. In order to change the temperature, the model must provide a justification. The request_sampling_update schema includes a required reasoning field. This isn't just for logging (though it is very helpful for this purpose and provides insight). To populate that field, the model must generate a coherent explanation for why the temperature should change. That generation process grounds the decision. Adding this field doesn't completely eliminate overeager temperature changes at higher temperatures, and it doesn't normalize the likelihood of changes across different settings. If the 'likelihood' of a temperature change at temperature 1.0 is considered to be the 'optimal' likelihood, this grounding requirement certainly doesn't make it more likely for a low-temperature model to increase its temperature.

update_schema = {
    "type": "object",
    "properties": {
        "reasoning": {
            "type": "string",
            "description": (
                "Brief explanation of why a temperature change "
                "is needed at this point in the response."
            ),
        },
        "new_temperature": {
            "type": "number",
            "minimum": 0.0,
            "maximum": 2.0,
        },
        # ... other params
    },
    "required": ["reasoning", "new_temperature"],
}

The reasoning requirement helps ground the LLM. With this requirement in place, it's harder for the model to decide to change temperature on a whim, as it has to justify it with a cogent reason. What's more, those justifications are genuinely useful for understanding what the model thinks it's doing. The model requesting a temperature change because it is "Transitioning to precise technical specification" tells you it understood the task structure.

A high-temperature model might want to change settings on a whim, but having to explain itself acts as a soft filter. The model has to commit to a rationale, and bad rationales tend to be harder to generate even when you're sampling creatively. It's the difference between "I feel like being more creative" (rejected by its own incoherence) and "The user asked for brainstorming alternatives, which benefits from higher variance" (coherent, actionable). I can further leverage this by adding a code-level filter to reject obviously bad reasoning that still slips through.

6. Validation & Anti-Thrashing

Even with the reasoning field, I don't just blindly apply temperature changes. Models can still get twitchy, especially at high temperatures, and might try to change the temperature every 5 tokens if given the chance. And by might, I mean they absolutely will, in the case of GLM.

To prevent this inevitability, I implemented a strict AntiThrashingValidator in src/thermostat/validators.py that enforces a hard cap on thrashing and a minimum number of tokens generated between changes:

def validate_and_apply_tool_call(
    tool_call: RequestSamplingUpdate,
    current_params: SamplingParameters,
    buffer: OutputBuffer,
    config: Any,
) -> Tuple[SamplingParameters, Dict[str, Any]]:
    
    # Resolve relative updates (e.g., if the model 
    # requests "Creative" preset)
    resolved = resolve_sampling_params(tool_call, current_params)

    # Apply policy clamps: min_chars_between_changes, 
    # max_temp_delta, etc.
    clamped_params, warnings = AntiThrashingValidator.apply_policy_clamps(
        tool_call, 
        current_params, 
        buffer, 
        config
    )

    return clamped_params, {"warnings": warnings}

{
  "applied_temperature": 0.7,  // clamped from requested 0.2
  "applied_top_p": 0.9,
  "note": "Applied requested update. Continue from OUTPUT_SO_FAR end.",
  "warnings": [
    "min_chars_between_changes: Not enough characters since last change",
    "max_temp_delta_per_change: temperature was clamped to 0.7 from 0.2"
  ]
}

# Disable tools if we haven't produced enough text since the last temp change
# This prevents thrashing where the model keeps changing temp without writing
chars_since_change = self.buffer.chars_since_last_change()
disable_tools = (
    self.buffer.temperature_changes > 0 
    and chars_since_change < self.config.min_chars_between_changes
)

stream = self.provider.generate_segment(
    ...
    disable_tools=disable_tools,  # Tools literally not available
)

7. The Orchestrator Loop

Finally, the Orchestrator runs the show. It takes the new temperature, inserts that into the API call, feeds the model the full context plus a continuation marker, and tells it to resume exactly there.

async def generate(self) -> AsyncGenerator[StreamChunk, None]:
    while not self.state.is_finished:
        # Call provider with CURRENT sampling params.
        # Note: We pass sampling_params dynamically on every iteration.
        stream = self.provider.generate_segment(
            user_prompt=self.user_prompt,
            output_buffer=self.buffer,
            sampling_params=self.state.sampling_params, 
            system_prompt=self.system_prompt,
        )

        accumulated_text, tool_calls = await self._accumulate_segment_from_stream(stream)

        # Handle tool calls
        if tool_call := (tool_calls[0][0] if tool_calls else None):
            
            if isinstance(tool_call, RequestSamplingUpdate):
                self._handle_sampling_update(tool_call, self.state)
                # We update state, but do NOT break the loop. 
                # The next iteration will pick up the new params.

            elif isinstance(tool_call, FinishResponse):
                self._handle_finish_response(tool_call, self.state)
                # This sets self.state.is_finished = True, 
                # breaking the while loop.

The orchestrator also handles content validation to ensure the append-only paradigm holds. Since I'm stitching multiple API calls together, the model might accidentally repeat earlier output, attempt to rewrite what it said before, or leak chain-of-thought reasoning (e.g., "the user asked for..." instead of just continuing). The AppendSafetyValidator catches these:

The real nightmare was high temperature. At low temperature, the model follows the "continue exactly from here" instruction perfectly. At high temperature, it was as if I didn't give any instructions whatsoever.

It would add "Continuing from where I left off..." even though I explicitly told it not to. It would restart sentences. It would summarize what it just said instead of continuing. I had to build overlap detection, repeat detection, and a repair mode that calls the model again at low temperature with an explanation of what it did wrong.

When validation fails, the orchestrator goes into repair mode: it calls the model again at low temperature (T=0.2) with an explicit instruction explaining the violation and asking it to continue correctly. If repair attempts are exhausted, the response terminates with an error.

The "Call Boundaries" Logic

I've noticed a lot of software developers who are very experienced but treat LLMs as a black box often say, "When I'm doing ideation I like talking in the ChatGPT or Claude web interface, and then implementing in Cursor or Copilot."

What they're actually benefiting from is the higher temperature in those chat interfaces versus the precision that comes with an agentic coding tool, which probably sets its model to a much lower temperature in its API calls. They've accidentally discovered dynamic temperature by switching apps. Thermostat tries to make that explicit within a single response.

Now that you've seen how the sausage is made within the orchestration loop, let's talk about the bill: KV cache.

One reason inference got cheaper: KV cache cuts repeat compute[6]. You can see this in action in the way many LLM providers price their models, where you get a much cheaper price per token when there is a cache hit.

This stitching approach means you don't reuse KV cache between segments. Every temperature change is a new call, and each call reprocesses the full accumulated context, which gets bigger at each intra-response temperature change. What's more, in my experiments, asking for a different temperature precludes any sort of provider-side caching whatsoever.

The Cost Breakdown

With 3 temperature changes and 2000 tokens of context, you're paying for:

That's a lot of redundant input token processing. By definition, if you're calling with a new temperature, you're not going to get a cache hit. The provider treats it as an entirely new call with context you pre-filled in.

If this were something people actually wanted to use at scale, things would have to change on the provider end. But for experimenting, the cheap inference providers make it a non-factor. I used GLM and Chutes AI, which is $10/month for 1000 calls per day (frankly, this is a steal, even if some of the models are quantized). You can spam it without financial penalty while you figure out if this even works.

The Future: True Intra-Call Support

What would be actually interesting is if inference providers supported temperature changes mid-stream. Imagine an API that looked like:

response = llm.generate_with_dynamic_temp(
    messages=[...],
    initial_temperature=0.7,
    temp_change_token="[TEMP:",
    temp_end_token="]"
)

The provider would handle the temp changes internally without breaking the KV cache. The model generates[TEMP:0.3], the sampling parameters update, and generation continues seamlessly from the cached attention states.

Inference providers are optimizing for throughput, not flexibility. Adding mid-stream parameter changes would complicate batching strategies and require careful handling of speculative decoding. But hey, a guy can dream.

Enough theory...let's see what this actually looks like in practice.

I'll be frank: this almost certainly doesn't matter for simple queries. Even for complex tasks, it's not likely something that will provide significant benefits that are worth the cost and UX tradeoffs, especially for workflows that benefit from steering (shameless plug for my first article, BTW). It's more likely that you'd want to break that complex one-shot prompt into multiple [prompts | messages | threads | workflows] anyways, at which point you would just set different temperatures for each, if you were so inclined.

What's more, Z.ai themselves[7], the creators of GLM, now recommend keeping temperature at 1.0 for most use cases. If official guidance from AI labs and the fact that the GPT-5 series doesn't even have a temperature parameter available in its API[8] are any indication, temperature might be becoming less relevant as models get smarter about self-regulating. I remember when the temperature made a huge difference when playing around in tools like OpenRouter's AI Chat Playground, but maybe temperature matters less than it used to.

In fact, while writing this, it is possible the most important thing I remembered was that thermostats don't actually typically change the temperature of the heater/AC itself; it blows air at a fixed hot/cold temperature and they simply turn it on or off. I am the king of reminding people about this fact when they think setting the thermostat to 80 will warm the room from 65 -> 70 more quickly. So, while my experiment may have worked somewhat, my pun does not. Oh well.

So, going back to my wife's initial question about whether you can change temperature on-the-fly: the answer is yes, you can. As with many things in the age of vibe coding and AI slop, whether you should is a different question, and the answer is probably not.

However, the best experiments are often the ones that answer "can we?" even when the answer to "should we?" is "probably not". Behind every "probably not" is a "why not?" worth asking. Not "why not?" in the literal "give me a reason" sense, as you'll surely get plenty of reasons from pedants and misanthropes alike, but "why not?" in the whimsical "why the hell not?" sense. Remember when they did surgery on a grape (the meme from ~2016)? It turns out that this is actually a useful technique for testing cutting edge micro-surgery robotics, owing to the precision required to operate on such a delicate but delicious fruit. However, a cursory Google search for "grape surgeons near me" reveals that this is not a common medical specialty, even a decade later. Sometimes, the gap between "why we could" and "why we don't" shines light on what is possible or valuable theoretically versus what is actually useful in the real world. In both cases, the curious question is still worth asking. Thanks for coming along for the ride.

— Jeff