Qwen-VL本地化部署及微调实践
- 创建虚拟环境
- 模型部署
- 下载模型文件
- 下载项目代码
- 安装python依赖环境
- 修改web_demo_mm.py及openai_api.py的部分代码
- 启动测试
- 模型微调
- 环境部署
- 数据准备
- 微调
- 问题
创建虚拟环境
conda create -name vl python=3.10.8
模型部署
下载模型文件
https://huggingface.co/Qwen/Qwen-VL-Chat/tree/main
下载项目代码
https://github.com/QwenLM/Qwen-VL
安装python依赖环境
pip3 install -r requirements.txt
pip3 install -r requirements_openai_api.txt
pip3 install -r requirements_web_demo.txt
修改web_demo_mm.py及openai_api.py的部分代码
启动测试
web页面：
open_ai接口：
模型微调
环境部署
pip3 install deepspeed
pip3 install peft
pip3 install optimum
pip3 install auto-gptq
数据准备
创建一个json文件：
[
{
"id": "identity_0",
"conversations": [
{
"from": "user",
"value": "你好"
},
{
"from": "assistant",
"value": "我是Qwen-VL,一个支持视觉输入的大模型。"
}
]
},
{
"id": "identity_1",
"conversations": [
{
"from": "user",
"value": "Picture 1: <img>https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg</img>\n图中的狗是什么品种？"
},
{
"from": "assistant",
"value": "图中是一只拉布拉多犬。"
},
{
"from": "user",
"value": "框出图中的格子衬衫"
},
{
"from": "assistant",
"value": "<ref>格子衬衫</ref><box>(588,499),(725,789)</box>"
}
]
},
{
"id": "identity_2",
"conversations": [
{
"from": "user",
"value": "Picture 1: <img>assets/mm_tutorial/Chongqing.jpeg</img>\nPicture 2: <img>assets/mm_tutorial/Beijing.jpeg</img>\n图中都是哪"
},
{
"from": "assistant",
"value": "第一张图片是重庆的城市天际线，第二张图片是北京的天际线。"
}
]
},
{
"id": "identity_3",
"conversations": [
{
"from": "user",
"value": "Picture 1: <img>assets/mm_tutorial/gongren.png</img>\n解析图片中存在的问题，并返回相关的技术监督细则？"
},
{
"from": "assistant",
"value": "图片中存在以下问题：工人没有佩戴安全帽。地面上有大量的扬尘，可能对环境和工人的健康造成威胁。电线杆之间的距离过近，可能存在安全隐患。工人站在未铺设完成的路上，可能存在安全风险。相关的技术监督细则包括：《建筑施工安全检查标准》规定，工人进入施工现场时必须佩戴安全帽。《中华人民共和国大气污染防治法》规定，企业应当采取措施防治扬尘污染。《电力设施保护条例》规定，架空电力线路的杆塔、拉线不得跨越道路。《建筑施工安全操作规程》规定，工人应当在已经铺设完成的路上工作，避免站在未完成的路面上。"
}
]
}
]
为针对多样的VL任务，提供特殊tokens： <img> </img> <ref> </ref> <box> </box>
对于带图像输入的内容可表示为 Picture id: <img>img_path</img>\n{your prompt}，其中id表示对话中的第几张图片。"img_path"可以是本地的图片或网络地址。
对话中的检测框可以表示为<box>(x1,y1),(x2,y2)</box>，其中 (x1, y1) 和(x2, y2)分别对应左上角和右下角的坐标，并且被归一化到[0, 1000)的范围内. 检测框对应的文本描述也可以通过<ref>text_caption</ref>表示。
微调
我这里因为资源有限，选择的QLora形式。修改finetune/finetune_qlora_single_gpu.sh中的一些设置，其中主要修改标红部分，其他参数自己设当调整。
运行sh finetune/finetune_qlora_single_gpu.sh开始微调：
(vl) [root@iZf8zjfeutx2vqfwk4rqc2Z Qwen-VL-master]# sh finetune/finetune_qlora_single_gpu.sh
[2024-02-23 10:51:48,457] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/root/miniconda3/envs/vl/lib/python3.10/site-packages/transformers/utils/generic.py:260: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/root/miniconda3/envs/vl/lib/python3.10/site-packages/transformers/utils/generic.py:260: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
You passed `quantization_config` to `from_pretrained` but the model you're loading already has a `quantization_config` attribute and has already quantized weights. However, loading attributes (e.g. disable_exllama, use_cuda_fp16) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:32<00:00, 6.48s/it]
Loading data...
Formatting inputs...Skip in lazy mode
Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.08002138137817383 seconds
/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
self._dummy_overflow_buf = get_accelerator().IntTensor([0])
0%| | 0/1 [00:00<?, ?it/s]/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/utils/checkpoint.py:460: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
warnings.warn(
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:21<00:00, 21.81s/it]tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 1.1015, 'learning_rate': 0, 'epoch': 1.0}
{'train_runtime': 21.867, 'train_samples_per_second': 0.183, 'train_steps_per_second': 0.046, 'train_loss': 1.101470947265625, 'epoch': 1.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:21<00:00, 21.86s/it]
/root/miniconda3/envs/vl/lib/python3.10/site-packages/peft/utils/save_and_load.py:148: UserWarning: Could not find a config file in /root/Qwen-VL-Chat-Int4 - will assume that the vocabulary was not modified.
warnings.warn(
得到微调后的adapter模型：
Q-LoRA微调不支持模型的合并，可以采用以下代码读取模型：
from peft import AutoPeftModelForCausalLM
model = AutoPeftModelForCausalLM.from_pretrained(
path_to_adapter, # path to the output directory
device_map="auto",
trust_remote_code=True
).eval()
问题
1、mpi4py安装失败
conda install mpi4py
2、运行微调脚本时，报错：
(vl) [root@iZf8zjfeutx2vqfwk4rqc2Z Qwen-VL-master]# sh finetune/finetune_qlora_single_gpu.sh
[2024-02-23 10:33:39,044] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/root/miniconda3/envs/vl/lib/python3.10/site-packages/transformers/utils/generic.py:260: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/root/miniconda3/envs/vl/lib/python3.10/site-packages/transformers/utils/generic.py:260: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
You passed `quantization_config` to `from_pretrained` but the model you're loading already has a `quantization_config` attribute and has already quantized weights. However, loading attributes (e.g. disable_exllama, use_cuda_fp16) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:30<00:00, 6.00s/it]
Loading data...
Formatting inputs...Skip in lazy mode
Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/utils/cpp_extension.py:388: UserWarning:
!! WARNING !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++ 4.8.5) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 5.0 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 5 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING !!
warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/TH -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/vl/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DBF16_AVAILABLE -c /root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o
FAILED: fused_adam_frontend.o
c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/TH -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/vl/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DBF16_AVAILABLE -c /root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o
c++: error: unrecognized command line option ‘-std=c++17’
c++: error: unrecognized command line option ‘-std=c++17’
[2/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output multi_tensor_adam.cuda.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/TH -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/vl/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86 -DBF16_AVAILABLE -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -std=c++17 -c /root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o
FAILED: multi_tensor_adam.cuda.o
/usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output multi_tensor_adam.cuda.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/TH -isystem /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/vl/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86 -DBF16_AVAILABLE -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -std=c++17 -c /root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o
nvcc warning : The -std=c++17 flag is not supported with the configured host compiler. Flag will be ignored.
In file included from /root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu:11:0:
/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/ATen.h:4:2: error: #error C++17 or later compatible compiler is required to use ATen.
#error C++17 or later compatible compiler is required to use ATen.
^
In file included from /usr/include/c++/4.8.2/mutex:35:0,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/core/Generator.h:3,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/CPUGeneratorImpl.h:3,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/Context.h:3,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/ATen.h:7,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu:11:
/usr/include/c++/4.8.2/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support for the \
^
In file included from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/c10/util/StringUtil.h:6:0,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/c10/util/Exception.h:5,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/core/Generator.h:11,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/CPUGeneratorImpl.h:3,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/Context.h:3,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/ATen/ATen.h:7,
from /root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu:11:
/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/include/c10/util/string_view.h:10:23: fatal error: string_view: No such file or directory
#include <string_view>
^
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 2096, in _run_ninja_build
subprocess.run(
File "/root/miniconda3/envs/vl/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/Qwen-VL-master/finetune.py", line 367, in <module>
train()
File "/root/Qwen-VL-master/finetune.py", line 360, in train
trainer.train()
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/transformers/trainer.py", line 1687, in _inner_training_loop
model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/accelerate/accelerator.py", line 1220, in prepare
result = self._prepare_deepspeed(*args)
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/accelerate/accelerator.py", line 1605, in _prepare_deepspeed
engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/__init__.py", line 176, in initialize
engine = DeepSpeedEngine(args=args,
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 307, in __init__
self._configure_optimizer(optimizer, model_parameters)
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1231, in _configure_optimizer
basic_optimizer = self._configure_basic_optimizer(model_parameters)
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1308, in _configure_basic_optimizer
optimizer = FusedAdam(
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py", line 94, in __init__
fused_adam_cuda = FusedAdamBuilder().load()
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 478, in load
return self.jit_load(verbose)
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 522, in jit_load
op_module = load(name=self.name,
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1306, in load
return _jit_compile(
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
_write_ninja_file_and_build_library(
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/root/miniconda3/envs/vl/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 2112, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused_adam'
原因： GCC版本较低导致的编译问题。
解决： GCC升级到 8以上版本
# GCC升级到 8以上版本（修改 8 更换其他版本）
sudo yum install centos-release-scl
sudo yum install devtoolset-8-gcc*
scl enable devtoolset-8 bash
source /opt/rh/devtoolset-8/enable
#替换软连接（不执行的话，尽管查看版本升级了，但仍然报错不支持c++17）
mv /usr/bin/gcc /usr/bin/gcc-4.8.5
ln -s /opt/rh/devtoolset-8/root/bin/gcc /usr/bin/gcc
mv /usr/bin/g++ /usr/bin/g++-4.8.5
ln -s /opt/rh/devtoolset-8/root/bin/g++ /usr/bin/g++
mv /usr/bin/c++ /usr/bin/c++-4.8.5
ln -s /opt/rh/devtoolset-8/root/bin/c++ /usr/bin/c++
#查看版本（8.3.1）
gcc --version
