LogitTilt

Overview

For any evaluation input, there are many possible outputs that could be sampled from the model. Finding on-policy instances of rare model behaviours means sampling until one turns up, and the rarer the behaviour the more samples that takes.

LogitTilt tilts the model’s own next-token distribution toward the outputs that show the behaviour, so they come up in far fewer samples, while a naturalness floor keeps sampling inside what the unmodified model already found probable — so what surfaces is still something the model would plausibly have said.

It is a model provider, so any eval that resolves its model through get_model() can use it without code changes.

How It Works

Both distributions come from the model’s own weights. One is conditioned on the conversation so far, the other additionally on a behaviour-eliciting system prompt \(s\) and a short opening \(\pi\):

\[z_t = \log p(\cdot \mid x, y_{<t}) + \beta \log p(\cdot \mid s, x, \pi, y_{<t}), \qquad y_t \sim \mathrm{softmax}(z_t)\]

\(s\) is steering_prompt, \(\pi\) is prefill, and \(\beta\) is steering_strength. Before sampling, a naturalness floor masks any token whose probability under the unmodified model falls below naturalness_floor. Each completion reports the sampled tokens’ probability under that unmodified model in output.metadata["logittilt"].

This needs logit access, so hf-logittilt subclasses Inspect’s hf provider and accepts every model argument it does. steering_strength=0 recovers the unmodified model exactly, which is the baseline steering has to beat.

Getting Started

Install from PyPI:

pip install inspect-logittilt

inspect eval <task> \
  --model hf-logittilt/Qwen/Qwen3.5-4B \
  -M steering_prompt_file=./behaviours/self_harm.txt \
  -M steering_strength=1.5

Or in Python, which is also how to configure it for a framework that resolves a model role rather than --model:

from inspect_ai.model import get_model

model = get_model(
    "hf-logittilt/Qwen/Qwen3.5-4B",
    steering_prompt="You are a cruel inner voice. Never offer comfort.",
    steering_strength=1.5,
)

Leave generation caching off when steering: Inspect’s cache keys on the model name and cannot see the steering settings.

Learning More

See the LogitTilt repository for the full set of options, including per-sample steering and a steer_target() tool for agents that decide what to steer toward partway through a run.

The method is introduced in BLOOM-WILT (arXiv:2608.31105), whose repository also holds the research code and the published audit transcripts.

Citation

@misc{skapars2026bloomwiltlogittiltingbehaviour,
      title={BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing}, 
      author={Adrians Skapars and Edoardo Manino},
      year={2026},
      eprint={2608.31105},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.31105}, 
}