Home > Uncategorized > So much for ‘statistical objectivity’

So much for ‘statistical objectivity’

from Lars Syll

Last year, we recruited 29 teams of researchers and asked them to answer the same research question with the same data set. Teams approached the data with a wide array of analytical techniques, and obtained highly varied results …

All teams were given the same large data set collected by a sports-statistics firm across four major football leagues. It included referee calls, counts of how often referees encountered each player, and player demographics including team position, height and weight. It also included a rating of players’ skin colour …

unchallengable-statisticsOf the 29 teams, 20 found a statistically significant correlation between skin colour and red cards … Findings varied enormously, from a slight (and non-significant) tendency for referees to give more red cards to light-skinned players to a strong trend of giving more red cards to dark-skinned players …

Had any one of these 29 analyses come out as a single peer-reviewed publication, the conclusion could have ranged from no race bias in referee decisions to a huge bias.

Raphael Silberzahn & Eric Uhlmann

Research that strongly underlines that even in statistics, the researcher has many degrees of freedom. In statistics — as in economics and econometrics — the results we get depend on the assumptions we make in our models. Changing those assumptions — playing a more important role than the data we feed into our models — leed to far-reaching changes in our conclusions. Using statistics​ is no guarantee we get at any ‘objective truth.’

  1. Helen Sakho
    July 6, 2018 at 1:27 am

    Thank you Lars. I confess that I spent a few minutes recently watching a “new” clip of a breathless Economist from the CSOEs, who did seem to be referring to you, followed by a poor chap (another prominent Economist) who was totally and utterly speechless as to what the new extremely challenging times that now lie ahead of this righteous school, the new agenda, the new models, new questions and answers could possibly be. But the “funniest” part was that the econometric models he was trying to develop were tragically breathtaking.
    So, with respect, I shall stick with statistics. As least, if grounded in sociological reality, it offers a quality that is less laughable.

  2. Prof Dr James Beckman, Germany
    July 6, 2018 at 7:43 am

    Thanks, Lars & Helen, It is time for the world to know how subjective are the models used. AI (artficial intellivence) used by computer science (CS) may shortly have its contribution also. Currently CS seems to largely avoid causality, such as consumption expenditure as dependent upon income, savings, credit, family status, etc, but it can run with numerous other variables on a time dimension from hours, days, quarters, etc to show graphically how consumption changes. The AI persons with whom I deal seldom have taken a course in economics, which indeed is interesting, is it not?

  3. July 23, 2018 at 9:01 am

    No technique or group of techniques assurances truth, if by truth we mean a result that never changes and is embedded in the data. But if by truth we mean a result that can be created again, then objective truth does happen. So, there is not one objective truth but an array of objective truths. Each having a history of construction and presentation. For example, statistical study creates results consistent with the forms of statistical study used. Each objective and each different. More simply, the results of social scientific research are collaborative. Like all human knowledge.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.