深度学习过拟合实战：L1/L2正则化与Dropout在Auto MPG回归任务中的5方案对比

汽车燃油效率预测一直是工业界和学术界关注的重点问题。Auto MPG数据集作为经典的回归任务基准，为我们研究深度学习模型中的过拟合现象提供了理想平台。本文将系统对比五种不同的正则化方案，从基础模型到组合策略，通过完整代码实现和可视化分析，揭示不同方法在抑制过拟合上的实际效果差异。

1. 问题背景与数据准备

Auto MPG数据集记录了上世纪70-80年代398款汽车的关键参数，包括气缸数、排量、马力、重量等特征，目标变量为每加仑燃油行驶里程（MPG）。在工业应用中，这类模型的预测精度直接影响发动机设计和能源政策制定。

我们先导入必要的库并加载数据集：

import tensorflow as tf from tensorflow import keras import pandas as pd import numpy as np import matplotlib.pyplot as plt dataset_path = keras.utils.get_file("auto-mpg.data", "http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data") column_names = ['MPG','Cylinders','Displacement','Horsepower','Weight', 'Acceleration','Model Year','Origin'] raw_dataset = pd.read_csv(dataset_path, names=column_names, na_values="?", comment='\t', sep=" ", skipinitialspace=True)

数据清洗阶段需要特别注意处理缺失值和分类变量转换：

# 处理缺失值 dataset = raw_dataset.dropna() # 转换分类变量 origin = dataset.pop('Origin') dataset['USA'] = (origin == 1)*1.0 dataset['Europe'] = (origin == 2)*1.0 dataset['Japan'] = (origin == 3)*1.0 # 划分训练测试集 train_dataset = dataset.sample(frac=0.8, random_state=0) test_dataset = dataset.drop(train_dataset.index) # 数据标准化 train_stats = train_dataset.describe() train_stats.pop("MPG") train_stats = train_stats.transpose() def norm(x): return (x - train_stats['mean']) / train_stats['std'] norm_train_data = norm(train_dataset) norm_test_data = norm(test_dataset) train_labels = train_dataset.pop('MPG') test_labels = test_dataset.pop('MPG')

2. 基础模型构建与过拟合现象

我们首先构建一个具有四个隐藏层的深度神经网络作为基础模型：

def build_baseline_model(): model = keras.Sequential([ keras.layers.Dense(512, activation='relu', input_shape=[len(train_dataset.keys())]), keras.layers.Dense(256, activation='relu'), keras.layers.Dense(128, activation='relu'), keras.layers.Dense(64, activation='relu'), keras.layers.Dense(1) ]) optimizer = keras.optimizers.RMSprop(0.001) model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse']) return model baseline_model = build_baseline_model() baseline_model.summary()

训练过程中我们观察到典型的过拟合现象：

early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=50) history = baseline_model.fit( norm_train_data, train_labels, epochs=1000, validation_split=0.2, verbose=0, callbacks=[early_stop] ) def plot_history(history): hist = pd.DataFrame(history.history) hist['epoch'] = history.epoch plt.figure(figsize=(12,6)) plt.subplot(1,2,1) plt.xlabel('Epoch') plt.ylabel('Mean Abs Error [MPG]') plt.plot(hist['epoch'], hist['mae'], label='Train Error') plt.plot(hist['epoch'], hist['val_mae'], label='Val Error') plt.ylim([0,5]) plt.legend() plt.subplot(1,2,2) plt.xlabel('Epoch') plt.ylabel('Mean Square Error [MPG]') plt.plot(hist['epoch'], hist['mse'], label='Train Error') plt.plot(hist['epoch'], hist['val_mse'], label='Val Error') plt.ylim([0,20]) plt.legend() plt.show() plot_history(history)

训练曲线显示验证误差在约100轮后开始上升，而训练误差持续下降，这是典型的过拟合信号。接下来我们将系统评估五种正则化方案的效果。

3. L1正则化方案

L1正则化通过在损失函数中添加权重绝对值之和，促使模型产生稀疏权重矩阵。实现方案如下：

def build_l1_model(): model = keras.Sequential([ keras.layers.Dense(512, activation='relu', kernel_regularizer=keras.regularizers.l1(0.001), input_shape=[len(train_dataset.keys())]), keras.layers.Dense(256, activation='relu', kernel_regularizer=keras.regularizers.l1(0.001)), keras.layers.Dense(128, activation='relu', kernel_regularizer=keras.regularizers.l1(0.001)), keras.layers.Dense(64, activation='relu', kernel_regularizer=keras.regularizers.l1(0.001)), keras.layers.Dense(1) ]) optimizer = keras.optimizers.RMSprop(0.001) model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse']) return model l1_model = build_l1_model() l1_history = l1_model.fit( norm_train_data, train_labels, epochs=1000, validation_split=0.2, verbose=0, callbacks=[early_stop] )

L1正则化的关键特点：

• 产生稀疏解，适合特征选择场景
• 正则化系数(0.001)需要网格搜索确定
• 计算梯度时需处理绝对值函数的不可导点

4. L2正则化方案

L2正则化添加权重平方和惩罚项，使权重趋向于较小数值：

def build_l2_model(): model = keras.Sequential([ keras.layers.Dense(512, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001), input_shape=[len(train_dataset.keys())]), keras.layers.Dense(256, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001)), keras.layers.Dense(128, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001)), keras.layers.Dense(64, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001)), keras.layers.Dense(1) ]) optimizer = keras.optimizers.RMSprop(0.001) model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse']) return model l2_model = build_l2_model() l2_history = l2_model.fit( norm_train_data, train_labels, epochs=1000, validation_split=0.2, verbose=0, callbacks=[early_stop] )

L2正则化的优势：

• 对异常值不敏感，稳定性更好
• 计算效率高于L1（处处可导）
• 适合需要所有特征参与预测的场景

5. Dropout方案

Dropout通过在训练时随机丢弃神经元，防止神经元过度依赖特定特征：

def build_dropout_model(): model = keras.Sequential([ keras.layers.Dense(512, activation='relu', input_shape=[len(train_dataset.keys())]), keras.layers.Dropout(0.5), keras.layers.Dense(256, activation='relu'), keras.layers.Dropout(0.5), keras.layers.Dense(128, activation='relu'), keras.layers.Dropout(0.5), keras.layers.Dense(64, activation='relu'), keras.layers.Dropout(0.5), keras.layers.Dense(1) ]) optimizer = keras.optimizers.RMSprop(0.001) model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse']) return model dropout_model = build_dropout_model() dropout_history = dropout_model.fit( norm_train_data, train_labels, epochs=1000, validation_split=0.2, verbose=0, callbacks=[early_stop] )

Dropout使用要点：

• 丢弃率通常设为0.2-0.5
• 测试阶段需进行缩放补偿（Keras自动处理）
• 可与批归一化层配合使用

6. 组合方案与性能对比

我们将L1/L2正则化与Dropout结合，构建复合正则化模型：

def build_combined_model(): model = keras.Sequential([ keras.layers.Dense(512, activation='relu', kernel_regularizer=keras.regularizers.l1_l2(0.001), input_shape=[len(train_dataset.keys())]), keras.layers.Dropout(0.5), keras.layers.Dense(256, activation='relu', kernel_regularizer=keras.regularizers.l1_l2(0.001)), keras.layers.Dropout(0.5), keras.layers.Dense(128, activation='relu', kernel_regularizer=keras.regularizers.l1_l2(0.001)), keras.layers.Dropout(0.5), keras.layers.Dense(64, activation='relu', kernel_regularizer=keras.regularizers.l1_l2(0.001)), keras.layers.Dropout(0.5), keras.layers.Dense(1) ]) optimizer = keras.optimizers.RMSprop(0.001) model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse']) return model combined_model = build_combined_model() combined_history = combined_model.fit( norm_train_data, train_labels, epochs=1000, validation_split=0.2, verbose=0, callbacks=[early_stop] )

五种方案在测试集上的性能对比：

可视化对比各模型训练过程：

def plot_compare(histories, labels): plt.figure(figsize=(12,6)) for i, history in enumerate(histories): hist = pd.DataFrame(history.history) plt.plot(hist['val_mae'], label=labels[i]) plt.xlabel('Epochs') plt.ylabel('Validation MAE') plt.ylim([1,3]) plt.legend() plt.show() histories = [history, l1_history, l2_history, dropout_history, combined_history] labels = ['Baseline', 'L1', 'L2', 'Dropout', 'Combined'] plot_compare(histories, labels)

7. 方案选择与调优建议

根据实验结果，我们得出以下实践建议：

1. 数据规模与正则化选择：

   • 小数据集优先考虑L2+Dropout组合
   • 大数据集可尝试单独使用Dropout

2. 超参数调优策略：

   from sklearn.model_selection import GridSearchCV from keras.wrappers.scikit_learn import KerasRegressor def create_model(l2_rate=0.001, dropout_rate=0.5): model = keras.Sequential([ keras.layers.Dense(512, activation='relu', kernel_regularizer=keras.regularizers.l2(l2_rate), input_shape=[len(train_dataset.keys())]), keras.layers.Dropout(dropout_rate), keras.layers.Dense(1) ]) model.compile(optimizer='rmsprop', loss='mse', metrics=['mae']) return model model = KerasRegressor(build_fn=create_model, epochs=300, batch_size=32, verbose=0) param_grid = {'l2_rate': [0.0001, 0.001, 0.01], 'dropout_rate': [0.3, 0.5, 0.7]} grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=3) grid_result = grid.fit(norm_train_data, train_labels)

3. 工业部署考量：

   • L1正则化模型更适合嵌入式设备部署（权重稀疏）
   • Dropout模型预测时需关闭dropout层
   • 组合方案通常需要更多训练资源

4. 误差分析与模型改进：

   • 检查预测误差与特征的相关性
   • 尝试添加特征交叉项
   • 考虑集成学习方法提升稳定性