Skip to content

Fix nccl lmdeploy - #2082

Open
PengchengShi00 wants to merge 3 commits into
InternLM:mainfrom
PengchengShi00:fix_nccl_lmdeploy
Open

Fix nccl lmdeploy#2082
PengchengShi00 wants to merge 3 commits into
InternLM:mainfrom
PengchengShi00:fix_nccl_lmdeploy

Conversation

@PengchengShi00

Copy link
Copy Markdown
Collaborator

fix lmdeploy NCCL权重更新出现的问题

  1. 补充lmdeploy在NCCL权重同步完后的空请求
  2. 修复lmdeploy flush_cache的问题,不能 offload+onload_weights+onload_kvcache+weight_update,而是offload+onload_weights+weight_update+onload_kvcache

def send(self, batch: WeightUpdateBatch) -> None:
state_dict = batch.state_dict
if not state_dict:
if not state_dict and not batch.finished:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我在想是不是就不需要这个判断呢?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants