Multiplicative and Quadratic Integrate-and-Fire Neurons for Very Short-Term Load Forecasting

Multiplicative and Quadratic Integrate-and-Fire Neurons for Very Short-Term Load Forecasting

Karan Sati

Department of Electrical Engineering College of Technology, GBPUAT Pantnagar, Uttarakhand, India

Abhishek Yadav

Department of Electrical Engineering, College of Technology, GBPUAT Pantnagar, Uttarakhand, India

Abstract – The choice of the neuron aggregation function and of the input features strongly affects the accuracy of a neural load- forecasting model. This paper compares three neuron architectures for very short-term load forecasting: the additive multilayer perceptron, the fundamental multiplicative neuron model and the quadratic integrate-and-fire neuron model. The multiplicative and quadratic neurons aggregate their inputs through a product rather than a weighted sum, which lets a single neuron represent the interactions between the recent load values with very few parameters. The three architectures were trained and tested on four years of hourly load and weather data for the National Capital Territory of Delhi. The multiplicative neuron model gave the most accurate one-hour-ahead forecast among the standalone networks, with a root-mean-square error of 222.12 MW, a mean absolute percentage error of 6.12% and a coefficient of determination of 0.937, clearly ahead of the multilayer perceptron. This accuracy was reached with far fewer trainable parameters than the multilayer perceptron, since the product- based aggregation lets a single neuron represent interactions that an additive neuron needs a whole hidden layer to approximate. The results show that a compact multiplicative neuron is an efficient and accurate architecture for very short-term load forecasting.

Keywords – very short-term load forecasting; multiplicative neuron; quadratic integrate-and-fire neuron; multilayer perceptron; soft computing

Nomenclature

x input feature vector

y, target and predicted load (MW) w, b weights and biases of a neuron (·) activation function

1. number of input features

2. number of test samples

   1. INTRODUCTION

      Very short-term load forecasting (VSTLF), with horizons up to one hour, is used for real-time grid balancing, automatic generation control and demand-side management. Artificial neural networks are widely used for this task because they learn the non-linear relationship between the input features and the load [1], [2]. The most common architecture is the multilayer perceptron (MLP), in which each neuron aggregates its inputs through a weighted sum before applying an activation function. A weighted sum, however, treats the inputs as additively

      separable and represents interactions between them only indirectly, through the hidden layer, which can require many neurons and many parameters.

      An alternative is to aggregate the inputs through a product rather than a sum. A multiplicative neuron can represent the interaction between its inputs directly, so that a single neuron can capture relationships that an additive neuron would need a whole layer to approximate. Two such architectures are the fundamental multiplicative neuron model (MNM) [3] and the quadratic integrate-and-fire neuron model (QIF) [5]. These compact models are attractive for VSTLF because the load in the next hour depends strongly on interactions among its recent values. Still, they have not been compared systematically with the MLP on the same load data.

      This paper studies these neuron architectures in detail. The contributions are as follows. First, the additive MLP, the multiplicative MNM and the quadratic QIF are compared as standalone forecasters on four years of real hourly load data. Second, the mechanisms by which the multiplicative aggregation improves accuracy are explained, together with the resulting savings in model size. Section II reviews related work, Section III presents the three neuron models, Section IV describes the data and features, Section V gives the experimental setup, Section VI reports the results, and Section VII concludes.

   2. RELATED WORK

      The multilayer perceptron has been the standard neural model for load forecasting since the early 1990s [1], and many studies have improved it through better training, similar-day correction and clustering [6], [9]. The multiplicative neuron model was introduced as a compact alternative that captures non-linear interactions with a single neuron [3], and it has been applied successfully to time-series prediction [4]. Integrate-and-fire neuron models, inspired by the dynamics of biological neurons [5], have been adapted for regression by using a smooth firing response. Broader reviews of intelligent load-forecasting methods [7], [8], [10] confirm that few studies compare additive and multiplicative neuron models systematically on the same load data, which is the gap addressed here.

   3. Neuron Models

      The three neuron architectures differ in how they aggregate their inputs and, their activation, as shown in Fig. 1.

      Fig. 1. The three neuron architectures: (a) additive aggregation (MLP), (b) multiplicative aggregation (MNM) and (c) quadratic integrate-and-fire (QIF).

      1. Additive Neuron (MLP)

         In the multilayer perceptron, a hidden neuron forms a weighted sum of its inputs and applies a non-linear activation,

         first four capture the recent dynamics and the last two capture the daily cycle. The three weather variables of the target hour were appended to these six lags, giving nine input features in total. The data were split chronologically into training,

         = (

         + ) (1)

         validation and testing sets (80, 10 and 10 percent), and every variable was scaled with min-max normalisation fitted on the

         =1

         where xi are the inputs, wji and bj are the weights and bias of neuron j, and is the activation. The weights are trained by back-propagation to minimise the mean square error,

         training set only [11],

         = min

         maxmin

         (5)

         = 1 ( )2 (2)

         2

      2. Multiplicative Neuron (MNM)

         The fundamental multiplicative neuron model replaces the weighted sum with a product of weighted inputs, so that a single neuron represents the multiplicative interaction among all of its inputs,

         The target was rescaled to the interval [0.10, 0.90] to keep it away from the saturated regions of the activation, and the forecasts were transformed back to the megawatt scale before the errors were computed.

         V. EXPERIMENTAL SETUP AND METRICS

         Each neuron model was trained on the training set, with the validation set used to control the stopping of training, and was

         =

         ( + ) , = 1

         (3)

         evaluated on the unseen test set. The forecasts were assessed

         =1

         1+

         with the root-mean-square error, the mean absolute error, the

         Because the load at the next hour depends on the product-like interaction between its recent values and the daily-cycle lags, this aggregation matches the structure of the data and can reach

         mean absolute percentage error and the coefficient of determination,

         a low error with very few parameters.

      3. Quadratic Integrate-and-Fire Neuron (QIF)

         The quadratic integrate-and-fire neuron also aggregates the inputs multiplicatively but passes the result through a scaled

         = 1 ( )2

         =1

         =1

         =1 | |

         (6)

         (7)

         arctangent firing response, which is a smooth bounded function inspired by the firing dynamics of a biological neuron,

         = 100

         =1

         ||

         (8)

         ()2

         =1

         = ( + ) , = 2 arctan() (4)

         2 = 1

         =1

         ( )2

         (9)

         The arctangent response saturates more gently than the logistic function, which changes how the neuron responds to large inputs and gives a different bias-variance trade-off from the multiplicative neuron model.

   4. DATA AND FEATURE CONSTRUCTION

The study uses hourly load data for the National Capital Territory of Delhi from January 2020 to April 2024, together with ambient temperature, dew point temperature and relative humidity. After cleaning, 37,968 hourly samples were available. The series was arranged into a supervised form using load lags at one, two, three, four, twenty-four and forty-eight hours; the

=1

where yi and i are the target and predicted load, is the mean of the target load, and n is the number of test samples.

The multilayer perceptron used a single hidden layer of sixty-four units with logistic activations, while the multiplicative and quadratic neurons each used a single aggregation unit. The multiplicative and quadratic neurons were trained with online gradient descent, updating the weights after every training example, whereas the multilayer perceptron was trained with full-batch gradient descent over the same data; all three used the same thirty thousand training samples and a fixed random seed. The learning rate was tuned separately for each architecture:

0.15 for the multiplicative neuron, 0.001 for the quadratic

neuron and 3 for the multilayer perceptron. Training ran for up to one thousand epochs; the multiplicative and quadratic neurons were stopped early once the mean square error failed to improve by a fixed tolerance over fifteen consecutive epochs, while the multilayer perceptron was run for the full one thousand epochs. The biases of the multiplicative neuron were initialised with a small positive offset rather than at zero, which keeps the product of factors away from the region where the logistic output collapses on the min-max-scaled inputs.

1. Results and Discussion

   1. Comparison of the Neuron Architectures

      Table I reports the accuracy of the three neuron models on the test set. The multiplicative neuron model is the most accurate network, with an RMSE of 222.12 MW, a MAPE of 6.12 percent and an R2 of 0.937. The quadratic integrate-and-fire neuron is close behind on percentage error, while the multilayer perceptron is clearly the weakest, with an RMSE of 347.04 MW and a MAPE of 9.16 percent. The multiplicative neuron therefore improves the RMSE over the multilayer perceptron by about 36 percent while using a smaller number of parameters. Fig. 2 shows the metrics graphically.

      TABLE I. Accuracy of the Three Neuron Models on the Clean Test Set

      Model	MAE (MW)	RMSE (MW)	MAPE (%)	R2
      MLP (additive)	263.83	347.04	9.16	0.846
      MNM (multiplicative)	176.36	222.12	6.12	0.937
      QIF (quadratic integrate-and-fire)	185.15	252.48	6.03	0.919

      a.The best network is shown in bold.

      Fig. 2. RMSE and MAPE of the additive, multiplicative and quadratic

      neurons.

   2. Error Distributions

      Fig. 3, Fig. 4 and Fig. 5 show the distribution of the forecast errors of the three neuron models. The multiplicative and quadratic neurons have narrow, near-symmetric error distributions centred on zero, whereas the multilayer perceptron has a wider distribution with heavier tails, which is consistent with its larger RMSE. The narrower distribution of the multiplicative neuron indicates that it makes fewer large errors at the peaks and troughs of the load, which are the most difficult points to forecast.

      Fig. 3: Error distribution of the multilayer perceptron on the clean test set.

      Fig 4. Error distribution of the multiplicative neuron model on the clean test set.

      Fig 5. Error distribution of the quadratic integrate-and-fire neuron on the clean test set.

   3. Forecast Quality of the Best Neuron

      Fig. 6 plots the forecasts of the multiplicative neuron model against the target load. The points lie close to the diagonal across the whole load range, with a little more scatter at the highest loads, which is consistent with the R2 of 0.937 and confirms that the compact multiplicative neuron reproduces the load accurately.

      Fig. 6. Forecast versus target load for the multiplicative neuron model on the clean test set.

      The accuracy of the multiplicative and quadratic neurons is achieved with a far smaller model than the multilayer perceptron. With nine input features, the multilayer perceptron

      with sixty-four hidden units has about seven hundred trainable parameters, whereas each product-based neuron uses one weight and one bias per input and therefore has only about eighteen parameters, as summarised in Table II. The multiplicative neuron thus reaches a lower error than the multilayer perceptron with roughly one fortieth of the parameters, which is the central efficiency advantage of multiplicative aggregation for this task. Fig. 7 shows the load of the next 120 hours forecast with the multiplicative neuron model.

      TABLE II. Approximate Number of Trainable Parameters of

      Each Network (Nine Input Features)

      Model	Aggregation	Hidden units	Approx. parameters
      MLP	additive (sum)	64	705
      MNM	multiplicative (product)		18
      QIF	multiplicative (product)		18

      Fig. 7: Forecast for the next 120 hours using the multiplicative neuron model.

2. DISCUSSION

   The central practical conclusion is that the aggregation function matters: replacing the additive aggregation of the multilayer perceptron with a multiplicative aggregation improves the accuracy substantially, because the product directly represents the interactions between the recent load values that drive the next-hour load. The multiplicative neuron model is the better of the two product-based neurons on this data, with the quadratic integrate-and-fire neuron a close second on percentage error, and both dominate the multilayer perceptron on the accuracy- versus-size trade-off summarised in Table II. Together these findings support the use of a compact multiplicative neuron as an efficient architecture for very short-term load forecasting.

3. CONCLUSION

This paper compared additive, multiplicative and quadratic integrate-and-fire neurons for very short-term load forecasting. On four years of hourly load data for Delhi, the multiplicative neuron model gave the most accurate one-hour-ahead forecast among the standalone networks, with an RMSE of 222.12 MW, a MAPE of 6.12 percent and an R2 of 0.937, clearly ahead of the multilayer perceptron. The study concludes that a compact

multiplicative neuron is an efficient and accurate architecture for VSTLF. Future work will extend the comparison to longer forecast horizons and will combine the multiplicative neuron with probabilistic methods to add a measure of forcast uncertainty.

Declaration of Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data Availability

The load data is available from the State Load Dispatch Centre, Delhi, and the meteorological data from the NASA POWER project. The processed data and code can be made available from the corresponding author on reasonable request.

REFERENCES

1. D. C. Park, M. A. El-Sharkawi, R. J. Marks, L. E. Atlas, and M. J. Dam- borg, “Electric load forecasting using an artificial neural network,” IEEE Trans. Power Syst., vol. 6, no. 2, pp. 442-449, May 1991.

2. H. S. Hippert, C. E. Pedreira, and R. C. Souza, “Neural networks for short- term load forecasting: a review and evaluation,” IEEE Trans. Power Syst., vol. 16, no. 1, pp. 44-55, Feb. 2001.

3. R. K. Yadav, P. K. Kalra, and J. John, “Time series prediction with single multiplicative neuron model,” Appl. Soft Comput., vol. 7, no. 4, pp. 1157- 1163, 2007.

4. L. Zhang and J. Wang, “A single multiplicative neuron model with im- proved learning for time-series forecasting,” Neurocomputing, vol. 168,

   pp. 1110-1120, 2015.

5. C. Koch, Biophysics of Computation: Information Processing in Single Neurons. Oxford, U.K.: Oxford Univ. Press, 1999.

6. G. Dudek, “Multilayer perceptron for short-term load forecasting: from global to local approach,” Neural Comput. Appl., vol. 32, pp. 3695-3707, 2020.

7. M. Q. Raza and A. Khosravi, “A review on artificial intelligence based load demand forecasting techniques for smart grid and buildings,” Renew. Sustain. Energy Rev., vol. 50, pp. 1352-1372, 2015.

8. T. Hong and S. Fan, “Probabilistic electric load forecasting: a tutorial re- view,” Int. J. Forecast., vol. 32, no. 3, pp. 914-938, 2016.

9. P. Lusis, K. R. Khalilpour, L. Andrew, and A. Liebman, “Short-term res- idential load forecasting: impact of calendar effects and forecast granular- ity,” Appl. Energy, vol. 205, pp. 654-669, 2017.

10. A. Gasparin, S. Lukovic, and C. Alippi, “Deep learning for time series forecasting: the electric load case,” CAAI Trans. Intell. Technol., vol. 7, no. 1, pp. 1-25, 2022.

11. M. Bareth and A. K. Yadav, “Impact of data normalisation techniques on ANN based electrical load forecasting,” in Proc. Int. Conf. Power Energy Syst., 2024.