I have encoutered a question, please help.
I am using cuda11.8 and pytorch2.0.1; and the command line is:
./tools/dist_train.sh bisai.py 1 --cfg-options efficient_conv_bn_eval="[backbone]"
my dist_train.sh
#!/usr/bin/env bash
CONFIG=$1
GPUS=$2
NNODES=${NNODES:-1}
NODE_RANK=${NODE_RANK:-0}
PORT=${PORT:-29568}
MASTER_ADDR=${MASTER_ADDR:-"127.0.0.1"}
PYTHONPATH="$(dirname $0)/..":$PYTHONPATH \
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \
--nnodes=$NNODES \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--nproc_per_node=$GPUS \
--master_port=$PORT \
$(dirname "$0")/train.py \
$CONFIG \
--launcher pytorch ${@:3}
Then I have the following exception:
10/16 03:46:39 - mmengine - INFO - Enabling the "efficient_conv_bn_eval" feature for sub-modules: ['backbone']
Traceback (most recent call last):
File "/gemini/code/mmdet_bisai/./tools/train.py", line 133, in <module>
main()
File "/gemini/code/mmdet_bisai/./tools/train.py", line 129, in main
runner.train()
File "/root/miniconda3/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1762, in train
turn_on_efficient_conv_bn_eval(ori_model, modules)
File "/root/miniconda3/lib/python3.10/site-packages/mmengine/model/efficient_conv_bn_eval.py", line 158, in turn_on_efficient_conv_bn_eval
turn_on_efficient_conv_bn_eval_for_single_model(module)
File "/root/miniconda3/lib/python3.10/site-packages/mmengine/model/efficient_conv_bn_eval.py", line 147, in turn_on_efficient_conv_bn_eval_for_single_model
fx_model: fx.GraphModule = fx.symbolic_trace(model)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 1109, in symbolic_trace
graph = tracer.trace(root, concrete_args)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 778, in trace
(self.create_arg(fn(*args)),),
File "/gemini/code/mmdet_bisai/mmdet/models/backbones/swin.py", line 835, in forward
x, hw_shape = self.patch_embed(x)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 756, in module_call_wrapper
return self.call_module(mod, forward, args, kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 467, in call_module
ret_val = forward(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 749, in forward
return _orig_module_call(mod, *args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/gemini/code/mmdet_bisai/mmdet/models/layers/transformer/utils.py", line 302, in forward
x = self.adap_padding(x)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 756, in module_call_wrapper
return self.call_module(mod, forward, args, kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 467, in call_module
ret_val = forward(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 749, in forward
return _orig_module_call(mod, *args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/gemini/code/mmdet_bisai/mmdet/models/layers/transformer/utils.py", line 176, in forward
pad_h, pad_w = self.get_pad_shape(x.size()[-2:])
File "/gemini/code/mmdet_bisai/mmdet/models/layers/transformer/utils.py", line 169, in get_pad_shape
pad_h = max((output_h - 1) * stride_h +
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/proxy.py", line 413, in __bool__
return self.tracer.to_bool(self)
File "/root/miniconda3/lib/python3.10/site-packages/torch/fx/proxy.py", line 276, in to_bool
raise TraceError('symbolically traced variables cannot be used as inputs to control flow')
torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow
I checked a lot of material and ask GPT, but got no answer. Please do help and thx in advance.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4