Nonstandard Errors

Albert J. Menkveld et al.

Journal of Finance, 2024 Part of Firms and financial market structure

Abstract

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Box plots of the reported estimates for six hypotheses, RT-H1 to RT-H6, on a logarithmic scale, with each team's estimate as a dot: dispersion of all estimates (N=164).
164 teams, one sample, wildly different answers. Teams estimated the same six trends in the EuroStoxx 50 index futures market, as yearly percentage changes (log scale). Boxes span the middle half of estimates; dots are individual team estimates. Every hypothesis shows substantial dispersion. Figure 1 of the paper, Panel A.

Cite

Albert J. Menkveld et al. (2024). "Nonstandard Errors." The Journal of Finance 79(3): 2339–2390.