SAM2 Ascend NPU 使能

Sam2VideoModel
├── Sam2VisionModel (Hiera Backbone)
│   ├── Stage 0: embed_dim=144, blocks=2
│   ├── Stage 1: embed_dim=288, blocks=6
│   ├── Stage 2: embed_dim=576, blocks=36
│   └── Stage 3: embed_dim=1152, blocks=4
├── Sam2PromptEncoder
│   └── 点嵌入 + Mask 嵌入
├── Sam2MaskDecoder
│   └── Cross Attention + MLP
└── Memory Attention (视频帧间记忆)

# 创建数据目录
mkdir -p /data/npu_adapter/sam2-adapt

# 创建容器
docker run -d \
  --name sam2-npu-adapt \
  --privileged \
  --pid=host \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  --device /dev/davinci0 \
  -v /usr/local/dcmi:/usr/local/dcmi:ro \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi:ro \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/:ro \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info:ro \
  -v /etc/ascend_install.info:/etc/ascend_install.info:ro \
  -v /data/npu_adapter/sam2-adapt:/data \
  --shm-size=16g \
  -p 2222:22 \
  swr.cn-north-4.myhuaweicloud.com/ascend-sact/ascend-a3-ubuntu:v3.3 \
  sleep infinity

# 配置 SSH
docker exec sam2-npu-adapt bash -c "sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config"
docker exec sam2-npu-adapt bash -c "service ssh start"
docker exec sam2-npu-adapt bash -c "echo 'root:<your_password>' | chpasswd"

# 进入容器
docker exec -it sam2-npu-adapt bash

# 设置环境变量
source /usr/local/Ascend/ascend-toolkit/set_env.sh
export OPENBLAS_NUM_THREADS=1

# 安装 pip
apt-get update && apt-get install -y python3-pip

# 安装依赖
pip3 install torch==2.11.0 torchvision==0.26.0 torch-npu==2.11.0rc1 \
    transformers accelerate Pillow requests scipy opencv-python \
    --break-system-packages -i https://pypi.tuna.tsinghua.edu.cn/simple

source /usr/local/Ascend/ascend-toolkit/set_env.sh
export OPENBLAS_NUM_THREADS=1

python3 -c "import torch; import torch_npu; print('NPU available:', torch_npu.npu.is_available()); print('NPU count:', torch_npu.npu.device_count())"

cd /data
python run_inference_npu.py --image https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/truck.jpg --output mask.png

Loading SAM2 model: facebook/sam2.1-hiera-large
Device: npu:0
NPU available: True
NPU count: 16
Model loaded successfully

Loading image: https://...
Image size: (1800, 1200)

Running single-point segmentation...

Results:
  Inference time: 0.494s
  Output masks shape: torch.Size([1, 3, 1200, 1800])
  Number of predictions: 3

Mask saved to: mask.png

=== SAM2 NPU inference completed ===

data/
├── images/          # 图像文件 (*.jpg, *.png)
│   ├── image_001.jpg
│   ├── image_002.jpg
│   └── ...
├── masks/           # 对应的分割 mask (*.png)
│   ├── image_001.png
│   ├── image_002.png
│   └── ...
└── annotations.json # 点/框标注（可选）

cd /data
python run_train_npu.py --data_dir /data/dataset --epochs 10 --batch_size 4 --lr 1e-5

=== SAM2 NPU Training ===
Device: npu:0
NPU available: True
NPU count: 16
Loading dataset from: /data/dataset
Loaded 100 images for train

Starting training for 10 epochs...
==================================================

Epoch 1/10
------------------------------
  Batch 0/25, Loss: 3.4145
  Batch 10/25, Loss: 2.8934
  Batch 20/25, Loss: 2.4567
Epoch 1 - Average Loss: 2.7234
Epoch time: 45.2s
Checkpoint saved: ./output/checkpoint_epoch_5.pt

...

Epoch 10/10
------------------------------
  Average Loss: 0.8765
Checkpoint saved: ./output/checkpoint_epoch_10.pt

==================================================
Training completed!
Final checkpoint saved to: ./output
==================================================

# 使用 torchrun 启动分布式训练
source /usr/local/Ascend/ascend-toolkit/set_env.sh
export OPENBLAS_NUM_THREADS=1

torchrun --nproc_per_node=8 \
  run_train_npu.py \
  --data_dir /data/dataset \
  --epochs 10 \
  --batch_size 2 \
  --lr 1e-5

# 启用梯度检查点
python run_train_npu.py --data_dir /data/dataset --gradient_checkpointing

# 或减小批大小
python run_train_npu.py --data_dir /data/dataset --batch_size 1

# 从 checkpoint 恢复
python run_train_npu.py --data_dir /data/dataset --resume ./output/checkpoint_epoch_5.pt

# 使用兼容层
from npu_compat import apply_npu_compat
apply_npu_compat()

# 或者手动修复
import torch.nn.functional as F
_original_gelu = F.gelu
def gelu_contiguous(input, approximate="none"):
    if input.device.type == "npu":
        input = input.contiguous()
    return _original_gelu(input, approximate=approximate)
F.gelu = gelu_contiguous

drvErr=87 (driver internal error)
EL0005: Resource_Busy

sam2-npu-adapt/
├── README.md                 # 本文档
├── run_inference_npu.py      # 推理脚本
├── npu_compat.py             # NPU 兼容层
├── run_train_npu.py          # 训练脚本（可选）
└── create_container.sh       # 容器创建脚本