TY - JOUR

T1 - Increasing the informativeness of performance assessment of predictive models of heavy metal spatial distributions in the topsoil by permutation approach

AU - Sergeev, Aleksandr

AU - Butorova, Anastasia

AU - Shichkin, Andrey

AU - Buevich, Alexander

AU - Baglaeva, Elena

PY - 2024/6/1

Y1 - 2024/6/1

N2 - This paper proposes to add a probabilistic component to the models’ performance assessment using permutation approach. The application of the permutation approach was demonstrated on an example of a nonparametric randomization test. To test this approach, three models based on artificial neural networks were implemented: multilayer perceptron, radial basis function network, and generalized regression neural network. For modeling, spatial distribution data of copper and iron in the topsoil (depth 0.05 m) in subarctic cities of Novy Urengoy and Noyabrsk, Yamalo-Nenets Autonomous Okrug, Russia, were used. The predicted values were compared with the observed values of the test subset. To evaluate the performance of the built models, we compared three approaches: 1) calculation of the indices (mean absolute error, correlation coefficient, index agreement, etc.), 2) Taylor diagram, 3) randomization assessment of the probability of obtaining the divergence between the observed and predicted datasets, assuming that both of these datasets are derived from the same population. In the randomization method, two statistics were used: difference in means and correlation coefficient. The permutation approach showed its productivity, as it allowed to assess the significance of the divergence between the observed and predicted datasets and provided a more complete and objective performance assessment of the models. Authors believe that the permutation approach may be an attractive alternative to traditional performance assessment methods in application to forecasting purposes in various fields of science.

AB - This paper proposes to add a probabilistic component to the models’ performance assessment using permutation approach. The application of the permutation approach was demonstrated on an example of a nonparametric randomization test. To test this approach, three models based on artificial neural networks were implemented: multilayer perceptron, radial basis function network, and generalized regression neural network. For modeling, spatial distribution data of copper and iron in the topsoil (depth 0.05 m) in subarctic cities of Novy Urengoy and Noyabrsk, Yamalo-Nenets Autonomous Okrug, Russia, were used. The predicted values were compared with the observed values of the test subset. To evaluate the performance of the built models, we compared three approaches: 1) calculation of the indices (mean absolute error, correlation coefficient, index agreement, etc.), 2) Taylor diagram, 3) randomization assessment of the probability of obtaining the divergence between the observed and predicted datasets, assuming that both of these datasets are derived from the same population. In the randomization method, two statistics were used: difference in means and correlation coefficient. The permutation approach showed its productivity, as it allowed to assess the significance of the divergence between the observed and predicted datasets and provided a more complete and objective performance assessment of the models. Authors believe that the permutation approach may be an attractive alternative to traditional performance assessment methods in application to forecasting purposes in various fields of science.

UR - http://www.scopus.com/inward/record.url?partnerID=8YFLogxK&scp=85192541661

U2 - 10.1007/s40808-024-02034-y

DO - 10.1007/s40808-024-02034-y

M3 - Article

VL - 10

SP - 4387

EP - 4400

JO - Modeling Earth Systems and Environment

JF - Modeling Earth Systems and Environment

SN - 2363-6203

IS - 3

ER -