Model Architecture

• Convolutional Layers

  • Multiple convolutional layers were used to extract features from the input data.
  • Each layer was followed by:
    • Batch Normalization: Helps stabilize training by normalizing inputs.
    • Swish Activation Function: Combines input ( x ) with its sigmoid function, improving gradient flow and mitigating vanishing gradients.
      • Formula: Swish(x) = x * σ(x), where σ(x) = 1 / (1 + exp(-x)).
      • Cited by Kılıçarslan et al. (2021) as a better alternative to ReLU for many tasks, including image processing.

• Pooling Layers

  • Employed max pooling layers to:
    • Reduce spatial dimensions of data.
    • Capture essential features without increasing computational load.

• Dense Layers

  • After flattening convolutional output:
    • Dense layers were used to learn the final prediction.

• Batch Normalization

  • Batch Normalization (BatchNorm) was applied throughout to improve the training speed and stability by normalizing layer inputs.

** Performance ** The original CNN model achieved reasonable performance on the evaluation metrics, indicating its effectiveness in predicting the weather data.

Trying some Improvements

Activation Function

The activation function was changed from the Swish function to the Scaled Exponential Linear Unit (SELU) function, defined as follows:

• selu(x) = λx if x > 0

• selu(x) = αe^x - α if x ≤ 0

• λ and α are predefined constants; this requires specific weight initialization but is self-normalizing. This helps to:

  • Keep the mean and variance of inputs to each layer normalized throughout the network.
  • Stabilize the training process and ensure faster convergence.

Activation functions introduce non-linearity into the model, which is essential for handling complex data patterns. Without it, a neural network would be equivalent to a linear regression model, regardless of depth.

Removal of Batch Normalization

Batch Normalization can benefit neural networks in several ways:

• It can speed up training by reducing internal covariate shift.
• It acts as a regularizer, potentially reducing the need for Dropout in some cases.
• It can make the model less sensitive to weight initialization.
• It allows for higher learning rates without risking instability, speeding up the training process.

However, using the SELU activation function, which provides self-normalization, reduces the need for Batch Normalization. In practice, removing Batch Normalization was found to yield better results, making its removal an improvement over the original configuration.

Model Architecture

The ConvLSTM model includes the following components:

• ConvLSTM2D Layers: These layers combine convolutional operations with LSTM units to learn spatiotemporal features from the input data.
• Pooling and UpSampling Layers: Max pooling and upsampling layers manage spatial dimensions and recover the original input size.
• Dropout: Applied to reduce overfitting by randomly setting a fraction of input units to zero during training.
• Batch Normalization: Similar to the CNN model, batch normalization normalizes activations of each layer to maintain zero mean and unit variance.

Performance

The performance of the ConvLSTM model was not satisfactory, showing higher errors compared to the CNN model. This indicates that while ConvLSTM has potential, further tuning or architecture adjustments may be necessary for improved accuracy in weather prediction tasks.

Research Question

This experiment is to learn more about the behavior of the original CNN and convLSTM models mentioned in the paper, and with different types of architecture and hyperparameters, to try to improve the loss function and evaluation metrics obtained in the original research to see if a better tuned convLSTM can perform equal to or better than CNN due to its ability of being a spatio temporal model.

Design

Hypothesis: Changes made to hyperparameters and the overall model structure will improve their performance on the given dataset, reducing the error obtained and increasing anomaly correction, and the performance of convLSTM will be equal to or better than CNN at weather prediction.

Independent Variables (Experimental Settings)

1. Activation function
2. Number of layers
3. Number of filters
4. Filter size

Control Variables (Biases and Modeling Assumptions)

1. Number of epochs will remain constant (10 for CNN, 5 for convLSTM)
2. Dataset will stay the same
3. Learning rate, optimizer (Adam), and number of batches will remain the same
4. Type of pooling will remain MaxPooling

Dependent Variables (Result Analysis)

1. Training loss metrics curve over the number of epochs
2. Evaluation metrics mentioned above (RMSE, MAE, ACC)

Methodology: The baseline for this experiment is the use of the original BharatBench dataset for weather prediction and the existing implementation on CNN and convLSTM provided by the original authors for comparison. In order to improve the accuracy of the model, initial focus remained on evaluation metrics of RMSE, MAE and ACC, as used by the authors. The following changes were made in order to reach an optimal solution upon continuous adjustments:

CNN Model

Starting with an analysis of which would be the most optimal activation function for the model (originally ‘swish’), ran 10 epochs for each activation type on the original CNN model and measured the evaluation metrics. Activation ‘selu’ was the best out of all, displayed by an overall metric evaluation.

The original paper mentions batch normalization as a way to improve training of deep neural networks by normalizing input to each layer, which was evaluated in a separate file but not included in the layer architecture due to bad performance metrics. An attempt was also made to change the number of batches to 128 and 64 from 32, which could potentially increase the stability of the optimization, but it also worsened the errors slightly when comparing activations again.

Normally such problems do not need padding changes, so the main focus was on the filters and their dimensions. Increasing the number of filters means more parameters to train on, but risks overfitting the data. Layer manipulation was done by changing the number of layers while tuning the filter size, default stride, as follows:

1. Decreased 1 inner layer: Slight Improvement
2. Decreased 1 more inner layer and increased filters in layer 2 to 64: Time taken increased to 12 ms/step, Slight Improvement
3. Increased number of filters in layer 1 to 512 to recognize more complex patterns: Slight Improvement
4. Increased 1 inner layer [Best Metrics]
5. Increased learning rate to 1e-04: Time taken was approximately 17 ms/step [Best Graph] [Chosen Model]
6. Same layers, increased learning rate to 1e-03, Big Downgrade
7. Same layers, decreased learning rate back to 1e-04 and decreased drop rate from 0.3 to 0.2, Slight Downgrade

No. 4 had the best evaluation metrics results, but upon analysis of their graphs, it was noticed that the graph for No. 5 for loss during training was significantly better, so that was chosen as the optimal model. Modified CNN is also trained for 10 epochs, to compare to original.

ConvLSTM Model

The same activation function was determined to be the best for convLSTM (‘selu’). The model was trained for 1 epoch due its complexity and time elapsed per epoch step being comparatively higher than CNN. Changes to the model architecture and hyperparameters were made as follows:

1. Layers with 512,64,64,32 number of filters: Increased time significantly
2. Layers with 128,64,64,32 number of filters: Reduced time, reduced accuracy
3. Same layers, learning rate increase to 1e-04: Reduced time, increased accuracy
4. 1 less layer: Reduced time, increased accuracy
5. Layers with 128,128,128 number of filters, drop rate increased to 0.3: 58 ms/step time elapsed, Increased time, Increased accuracy [Chosen Model]

Trying to increase the batches here worsened error and increased the time taken significantly, so it remained 32.

Trying other data variables from the dataset

Original paper noted variable T850 as the best, named TMP_prl. Variables of APCP_sfc and TMP_2m were also tested against both new models. Comparing the models again for the best activation for these new variables, the answer remained ‘selu’.

CNN model

ConvLSTM Model

Observations

The experiments with the ConvLSTM model revealed that balancing the number of filters, batch size, and learning rate is crucial for optimizing performance. While higher filter counts and learning rates can improve accuracy and reduce errors, they also increase training time. Similarly, smaller batch sizes can help the model generalize better but may slow down training.

The best-performing model configuration was achieved with (128, 128, 128) filters, a dropout rate of 0.3, and a learning rate of 1e-04, providing a good balance between performance and training efficiency.

This optimal configuration was then applied to two other variables in the dataset. However, the model performed poorly on these variables, indicating that the configuration tuned for tmp_prl did not generalize well to other variables (tmp_2m and aspc). This emphasizes the need for individualized tuning to address the distinct characteristics of each variable. Each variable in the dataset may have unique characteristics and patterns that require tailored model configurations. Factors such as data distribution, underlying trends, and the complexity of temporal patterns can vary significantly between variables, impacting the model’s ability to perform well in a general manner.

The evaluation metrics are not the sole indicator of which algorithm is better. Even though the training loss curve and the error metrics were better for our new model on dataset variable TMP_prl (the one finally used in the paper), evaluation on another metric TMP_2m gave a drastically different looking loss curve even though the error is still less in comparison. Moreover, APCP_sfc was giving very bad results and is excluded from this comparison. Although we were finally able to improve the overall performance of the model, the model needs to be tuned independently for all the variables of the dataset in order to perform better as they all might have different characteristics.

ConvLSTM’s performance did not change much for this new dataset variable, so using the original variable as intended here as well. Hence, the hypothesis that ConvLSTM would show comparable performance to CNN has failed.