tf.keras – Keras Model Classification doesn’t learn in FineTuning

I’m trying to Fine-Tuning EfficientNetB4 to classify 10 classes. First of all, I modify the model as follows:

baseModel = EfficientNetB4(weights="imagenet", include_top=False, input_tensor=Input(shape=(img_width, img_height, 3)))
headModel = baseModel.output
headModel = tf.keras.layers.GlobalAveragePooling2D(name="avg_pool")(headModel)
headModel = tf.keras.layers.BatchNormalization()(headModel)
headModel = tf.keras.layers.Dropout(0.2, name="top_dropout")(headModel)
headModel = Dense(10)(headModel) #10 CLASSES
headModel = Activation('softmax')(headModel)

Once these modifications, I set all the layers basemodel not Trainable:

model = Model(inputs=baseModel.input, outputs=headModel)
for layer in baseModel.layers:
    layer.trainable = False

And Start the training:

opt = SGD(learning_rate=1e-3, momentum=0.9)
model.compile(loss="categorical_crossentropy", optimizer=opt,
    metrics=["accuracy"])

model.fit(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

When it was finished, I select all the layers to True and start again the training with a low learning rate.

for layer in model.layers:
        layer.trainable = True

    optimizer = SGD(learning_rate=1e-5, momentum=0.9)
    model.compile(
        optimizer=optimizer, loss="categorical_crossentropy", metrics=["accuracy"]

train_generator.reset()
validation_generator.reset()
H = model.fit(train_generator, steps_per_epoch=nb_train_samples // batch_size, 
                 epochs=epochs_FT, 
                 validation_data=validation_generator,
                 validation_steps=nb_validation_samples // batch_size)

Results:

Epoch 72/100
1505/1505 [==============================] - 253s 168ms/step - loss: 0.9260 - accuracy: 0.6559 - val_loss: 1.1471 - val_accuracy: 0.5933
Epoch 73/100
1505/1505 [==============================] - 258s 171ms/step - loss: 0.9406 - accuracy: 0.6513 - val_loss: 1.1953 - val_accuracy: 0.5775
Epoch 74/100
1505/1505 [==============================] - 253s 168ms/step - loss: 0.9423 - accuracy: 0.6522 - val_loss: 1.0250 - val_accuracy: 0.6294
Epoch 75/100
1505/1505 [==============================] - 256s 170ms/step - loss: 0.9327 - accuracy: 0.6509 - val_loss: 1.1049 - val_accuracy: 0.6021
Epoch 76/100
1505/1505 [==============================] - 254s 169ms/step - loss: 0.9396 - accuracy: 0.6591 - val_loss: 1.0471 - val_accuracy: 0.6268
Epoch 77/100
1505/1505 [==============================] - 257s 171ms/step - loss: 0.9366 - accuracy: 0.6527 - val_loss: 1.1077 - val_accuracy: 0.6197
Epoch 78/100
1505/1505 [==============================] - 252s 168ms/step - loss: 0.9256 - accuracy: 0.6556 - val_loss: 1.0613 - val_accuracy: 0.6268
Epoch 79/100
1505/1505 [==============================] - 255s 169ms/step - loss: 0.9099 - accuracy: 0.6598 - val_loss: 1.1935 - val_accuracy: 0.5951
Epoch 80/100
1505/1505 [==============================] - 252s 167ms/step - loss: 0.9335 - accuracy: 0.6546 - val_loss: 1.1522 - val_accuracy: 0.5951
Epoch 81/100
1505/1505 [==============================] - 256s 170ms/step - loss: 0.9256 - accuracy: 0.6538 - val_loss: 1.1001 - val_accuracy: 0.6127
Epoch 82/100
1505/1505 [==============================] - 255s 170ms/step - loss: 0.9235 - accuracy: 0.6540 - val_loss: 1.1418 - val_accuracy: 0.6004
Epoch 83/100
1505/1505 [==============================] - 255s 169ms/step - loss: 0.9247 - accuracy: 0.6591 - val_loss: 1.0997 - val_accuracy: 0.6127
Epoch 84/100
1505/1505 [==============================] - 254s 169ms/step - loss: 0.9143 - accuracy: 0.6608 - val_loss: 1.1665 - val_accuracy: 0.5907
Epoch 85/100
1505/1505 [==============================] - 254s 168ms/step - loss: 0.9158 - accuracy: 0.6604 - val_loss: 1.2427 - val_accuracy: 0.5757
Epoch 86/100
1505/1505 [==============================] - 256s 170ms/step - loss: 0.9335 - accuracy: 0.6564 - val_loss: 1.0778 - val_accuracy: 0.6056
Epoch 87/100
1505/1505 [==============================] - 254s 169ms/step - loss: 0.9077 - accuracy: 0.6671 - val_loss: 1.0871 - val_accuracy: 0.6012
Epoch 88/100
1505/1505 [==============================] - 257s 171ms/step - loss: 0.9191 - accuracy: 0.6544 - val_loss: 1.0847 - val_accuracy: 0.6012
Epoch 89/100
1505/1505 [==============================] - 256s 170ms/step - loss: 0.9117 - accuracy: 0.6621 - val_loss: 1.1231 - val_accuracy: 0.5863
Epoch 90/100
1505/1505 [==============================] - 255s 170ms/step - loss: 0.9240 - accuracy: 0.6557 - val_loss: 1.0255 - val_accuracy: 0.6162
Epoch 91/100
1505/1505 [==============================] - 252s 168ms/step - loss: 0.9163 - accuracy: 0.6521 - val_loss: 1.1456 - val_accuracy: 0.5924
Epoch 92/100
1505/1505 [==============================] - 258s 171ms/step - loss: 0.9239 - accuracy: 0.6573 - val_loss: 1.0822 - val_accuracy: 0.6074
Epoch 93/100
1505/1505 [==============================] - 255s 169ms/step - loss: 0.9016 - accuracy: 0.6638 - val_loss: 1.0871 - val_accuracy: 0.5968
Epoch 94/100
1505/1505 [==============================] - 256s 170ms/step - loss: 0.9011 - accuracy: 0.6633 - val_loss: 1.0598 - val_accuracy: 0.6056
Epoch 95/100
1505/1505 [==============================] - 253s 168ms/step - loss: 0.8980 - accuracy: 0.6625 - val_loss: 1.1657 - val_accuracy: 0.5810
Epoch 96/100
1505/1505 [==============================] - 255s 169ms/step - loss: 0.9024 - accuracy: 0.6629 - val_loss: 1.1678 - val_accuracy: 0.5731
Epoch 97/100
1505/1505 [==============================] - 253s 168ms/step - loss: 0.9189 - accuracy: 0.6554 - val_loss: 1.0793 - val_accuracy: 0.6039
Epoch 98/100
1505/1505 [==============================] - 258s 171ms/step - loss: 0.9068 - accuracy: 0.6649 - val_loss: 1.0840 - val_accuracy: 0.6127
Epoch 99/100
1505/1505 [==============================] - 258s 171ms/step - loss: 0.9034 - accuracy: 0.6615 - val_loss: 1.0977 - val_accuracy: 0.5942
Epoch 100/100
1505/1505 [==============================] - 255s 169ms/step - loss: 0.8969 - accuracy: 0.6658 - val_loss: 1.1322 - val_accuracy: 0.5924

I can’t achieve an accuracy upper than 0.66. I already use data augmentation and the number of pictures that I have is the following:

Training: Found 12045 images belonging to 10 classes. 1205 pictures approximately for every class. Validation: Found 1143 images belonging to 10 classes. 115 pictures approximately for every class. Evaluation: Found 1131 images belonging to 10 classes. 114 pictures approximately for every class.

Am I doing something wrong when modifying the model? If not, what could I do to improve the accuracy?

Leave a Comment