Reference Data Accuracy Impacts Burned Area Product Validation: The Role of the Expert Analyst

Abstract

Accurate reference data to validate burned area (BA) products are crucial to obtaining reliable accuracy metrics for such products. However, the accuracy of reference data can be affected by numerous factors; hence, we can expect some degree of deviation with respect to real ground conditions. Since reference data are usually produced by semi-automatic methods, where human-based image interpretation is an important part of the process, in this study, we analyze the impact of the interpreter on the accuracy of the reference data. Here, we compare the accuracy metrics of the FireCCI51 BA product obtained from reference datasets that were produced by different analysts over 60 sites located in tropical regions of South America. Additionally, fire severity, tree cover percentage, and canopy height were selected as explanatory sources of discrepancies between interpreters’ reference BA classifications. We found significant differences between the FireCCI51 accuracy metrics obtained with the different reference datasets. The highest accuracies (highest Dice coefficient) were obtained with the reference dataset produced by the most experienced interpreter. The results indicated that fire severity is the main source of discrepancy between interpreters. Disagreement between interpreters was more likely to occur in areas with low fire severity. We conclude that the training and experience of the interpreter play a crucial role in guaranteeing the quality of the reference data.