get_peft_model on a lazily-materialized model leaves third-party hooks and optimizers holding stale parameter references — warn at wrap time?

[immu4989]

System Info

peft 0.18.1, transformers 5.0.0, torch 2.10-2.12, reproduced on CPU (macOS), Kaggle T4 (CUDA 12), and RTX 5090 (CUDA 12.8).

Who can help?

@BenjaminBossan

Reproduction

Full investigation with reproducer: meta-pytorch/opacus#820. Short version:

model = AutoModelForCausalLM.from_pretrained("gpt2")
# model NOT moved to device here
model = get_peft_model(model, lora_config)
# third-party library registers per-parameter hooks now (opacus's
# PrivacyEngine in our case, but any hook-registering framework hits this)
# ...
batch = batch.cuda()  # only the batch moves; lazy handling materializes
                      # params at first forward, AFTER hooks were registered

Training then proceeds looking completely healthy. Loss decreases, checkpoints save, and in the DP case the privacy accountant keeps reporting epsilon. The LoRA weights never update: max |Δ| = 0.0 after any number of steps. Moving the model to the device before get_peft_model fixes it. We confirmed this on three independent environments over a five-month investigation (write up: https://imranahamed.substack.com/p/the-dp-lora-silent-corruption-how).

Expected behavior

The root cause isn't a peft bug. It's an ordering hazard between peft's lazy device handling and anything that captures parameter references at wrap time (per parameter hooks, early built optimizers). But peft is the layer that can see the hazard forming, and get_peft_model already warns on several usage hazards (double wrapping, renamed base models, adapter name prefix collisions).

Two suggestions, happy to PR either or both:

1. A pre-flight warning in get_peft_model when base-model params are on meta (or _hf_hook.original_devices indicates pending materialization) outside the known device_map inference flows: something like "parameters will be materialized after wrapping; parameter references captured now (hooks, optimizers) may go stale." The meta-device detection pattern already exists in tuners_utils.py and peft_model.py, so this reuses it rather than adding new machinery. Roughly 10-15 lines plus a test.

2. A troubleshooting entry in docs/source/developer_guides/troubleshooting.md documenting the ordering requirement (move to device, then wrap, then register hooks / build optimizers), next to the existing "Bad results from a loaded PEFT model" section.

The condition scoping on (1) needs care to avoid false positives on legitimate device_map="auto" inference loading; I'd propose gating it to training mode wrapping or making it a one-time info-level warning. Open to whatever shape you think fits.

[BenjaminBossan]

Thanks for raising this issue, this was a tricky thing to debug. For my proper understanding, does "DP" mean "data parallel" (the old one, not DDP) in this context or something else like "differential privacy". If it's the former: This was never officially supported and I wouldn't be surprised if it fails.

Regarding your suggestions:

1. I think I would have to see that warning in practice. My main concern is to not introduce falsely detected warnings, which can be tricky, as PEFT can't really control who sets which hooks. Maybe the concern is unfounded though. It's easiest to check if I can see a draft implementation and then we can iterate on that.
2. Adding such an entry shouldn't hurt. It should just be very clear what type of bug this prevents.

[immu4989]

DP means Differential privacy, the hook registering library in the repro is opacus's PrivacyEngine, which registers per sample gradient hooks on each trainable module at wrap time. No DataParallel anywhere in the setup. The same staleness applies to any framework that captures parameter references between get_peft_model and first forward, though; opacus is just where it was diagnosed.

Agreed the false positive risk is the crux, and it's hard to reason about in the abstract. I will put up a draft PR with both pieces (warning gated on meta device params at wrap time outside device_map inference flows, plus the troubleshooting entry) so we can iterate on the condition scope concretely. The docs entry will name the failure signature explicitly: training looks healthy, loss decreases, weights never change.

[BenjaminBossan]

Thanks for clarifying.

Regarding the hooks: Yes, this is a tricky business and we want to focus more on adding warnings where we know that hooks won't work correctly in PEFT. Therefore, we're very open to your suggestion, we just need to ensure that users are not bombarded by false positives. Let's see with the PR if we can narrow the check well enough to prevent that.