Home > Uncategorized > The conundrum of unknown unknowns

The conundrum of unknown unknowns

from Lars Syll

Short-term weather forecasting is possible because most of the factors that determine tomorrow’s weather are, in a sense, already there … But when you look further ahead you encounter the intractable problem that, in non-linear systems, small changes in initial conditions can lead to cumulatively larger and larger changes in outcomes over time. In these circumstances imperfect knowledge may be no more useful than no knowledge at all.

economic_forecastingMuch the same is true in economics and business. What gross domestic product will be tomorrow is, like tomorrow’s rain or the 1987 hurricane, more or less already there: tomorrow’s output is already in production, tomorrow’s sales are already on the shelves, tomorrow’s business appointments already made. Big data will help us analyse this. We will know more accurately and more quickly what GDP is, we will be more successful in predicting output in the next quarter, and our estimates will be subject to fewer revisions …

Big data can help us understand the past and the present but it can help us understand the future only to the extent that the future is, in some relevant way, contained in the present. That requires a constancy of underlying structure that is true of some physical processes but can never be true of a world that contains Hitler and Napoleon, Henry Ford and Steve Jobs; a world in which important decisions or discoveries are made by processes that are inherently unpredictable and not susceptible to quantitative description.

John Kay

The central problem with the present ‘machine learning’ and ‘big data’ hype is that so many — falsely — think that they can get away with analysing real world phenomena without any (commitment to) theory. But — data never speaks for itself. Without a prior statistical set-up there actually are no data at all to process. And — using a machine learning algorithm will only produce what you are looking for.

Theory matters.


  1. August 6, 2017 at 6:36 am

    I am afraid that Lars Syll is missing to tell one important thing. His short comment on John Kay is right, but this kind of aphorism might not stop the strong trend that compels people to use big data. Young researchers are urged to publish papers. When new data (called big data) became available, it is natural that those young researchers rash to analyzing new data.

    If we are skeptical to the present mainstream economics, it is important that these new data will contribute to discover how and why the mainstream economics (micro and macro) is wrong. Does anybody have good ideas in this direction? We need a theory that may hint or orient future investigations.

  2. Gary Seth
    August 6, 2017 at 8:17 am

    If we are referring to current economics it is my view that in the macro sense we are no longer dealing with a stoachastic process in that the Global totalitarian governments policies are as random as the construction of a pyramid : An economically unequal winner take all
    Autocratic pyramid .

  3. Yu
    August 6, 2017 at 10:34 am

    My knowledge to machine learning is very limited. But as an undergrade in a econ department, ML gives me new tools to do `empirical work’ other than using a straight line or a system of straight lines. I agree that data can’t discover new theory, but it can discover trends in (non)linear forms, which can be used to refute theories and build something new.
    And ML can be customized to fit a theory, say neural network which functions are up to imagination.

    I am skeptical that a decision making process us inherently unpredictable. We are human and are good at relying something proven to work, whether the outcome is good or bad. As long as people can control their behavior, there are some clues to figure out what comes next. Otherwise, criminal polices will lose their jobs. I hope we can find ways to detect such process and then develop theory to explain it. I am not a physics major, though, the process of discovering light property, gravity, quantum, etc relies heavily on data discoveries, right?

    A bug problem for ML is interpretability. Random forest and neural network are useful to interpolate data, but I am not able to find papers discussing each input variables. I guess this is why you argue we need theory prior to algorithm.

  4. Jerry Lobdill
    August 6, 2017 at 10:35 am

    Michael Hudson, in Killing the Host, makes clear that today’s GDP contains all rentier “production”. Thus it is inappropriate to use without removing rentier data in the three sector model (Godley’s) if the goal is to adjust gov’t spending to keep (true) private sector spending in non-negative territory.

  5. August 6, 2017 at 1:00 pm

    The only solution is to develop flexible (I call them “adaptable”) strategies that should react to unknown unknowns without catastrophic consequences. And, at present, the only way to construct them is an iterative process of solving a sequence of enhanced miltiscenario multicriteria optimization models that include enormous number of scenarios. Including “black swan” scenarios, even those with zero probability (at the present knowledge of unknowns). The process called Risk-Constrained Optimization (RCO) is described in my May paper (“Shifting the Paradigm” of Superintelligence) in Revue of Economics and Finance (1923-17529-2017-02-17-14, I think).

    Every freaking economist, present and past, only complained. Including JMK, Sills and Kay. RCO may still be not 100 percent perfect (seen any perfect revolution lately?), but it works.

    As to machine learning, it is one of the most dangerous things for the mankind. Sooner or later, it will develop superintelligent systems that will have their own self-preservation as a goal and will beat mankind in fight for energy.

    As Einstein said, stupidity is the only infinite thing in the world.

  6. August 6, 2017 at 1:08 pm

    Sorry, Syll rather than Sills.

  7. August 6, 2017 at 1:37 pm

    Talking of conundrums (some prefer conundra… and indulge the platitude).
    The problem with «big data» is that it requires computer models to be usable… (millions of «items» of data to be processed and analysed). The problem with computer models is that it requires a previous minimum «knowledge» and «understanding» of what it is one is looking for (one has to start somewhere), in order to build-up the algorithms that are supposed to do the analysis. And then there are the problems with the assumptions for those infamous «initial conditions», «path dependencies» and how to account for «random events». As well as the weight to attribute to all «intervening factors» (or secondary variables) that one can think of.
    Some measure of acquaintance with what other humans before us have thought about the issue at hand, coupled with the magic of creativity, is advisable.

  8. Vladimir Masch
    August 6, 2017 at 6:25 pm

    Please send me all comments on this blog.


  9. August 6, 2017 at 10:06 pm

    Historical data can tell us a lot. I sometimes wonder about the controversies I read here between economists about the “economy.” It is a fact that we do have one, an economy, and it is managed and controlled by a known body of authorities and they succeed, more or less, in making the economy increase, slow down or go sideways by managing the quantity of money in the economy and the ease with which it can be obtained. I also know that those tools for controlling the economy are very weak, that much stronger tools are available and are neither available nor used by the governing authorities. Perhaps the problem with economists is they cannot see the forest for the trees.

  10. August 6, 2017 at 11:41 pm

    Jesus, that is the wrong lesson. Theory matters? WTF? Neither big data NOR theory can predict the future. For some insane reason, people from Syll to Krugman have decided that the criterion for “better economics” is “better at predicting the future.” Something is seriously sick when the critics and the mainstream are both insane in the exact same way.

  11. August 7, 2017 at 3:56 pm

    Thornton is absolutely right.

  12. August 8, 2017 at 4:30 am

    Vladimir A. Masch seems to have mistaken “theory” as something rigidly formulated in mathematical models. They are not theory. They are only components of a theory. The reasons you think that adaptable strategies are more effective are burgeons of a theory.

    Agent-based simulation and data mining have their theoretical problems. I have discussed the first in
    A Guided Tour of the Backside of Agent-Based Simulation

    The main appeal of this “guided tour” is two points:
    (1) Mathematical formulation is often too strict restraints and it is preferable to develop new methods or tools of analysis other than mathematics. They are simulations and data analysis.
    (2) However, simulation without theoretical reflection has a strong risk that the research becomes producing machine “garbage in, garbage out.” Data analysis is not exempt of this risk.

    I hope you do not work in GIGO style.

    The problem that Thornton Hall posed is to be considered from a different angle. It i true that something is seriously ill in economics, not only mainstreams but also those economists who are critical to the present state of economics.

  13. August 9, 2017 at 12:16 pm

    Sure, RCO uses theories to construct the initial scenario submodels of the initial multiscenario model of the iterative process. Not one, but all conflicting theories. Use data mining and whatever else you like to develop all types of scenarios – including scenarios with almost zero and zero probabilities.

    At each iteration, the process will correct the model in one direction, reflecting the terminal illness of the modern economics, both mainstream and heterodox. It will make the maximization subordinate to the criterion of catastrophe avoidance. The criterion that has been dominating decision-making of all living creatures, from amoeba to Obama, for billions of years.

    It will also change scenario probabilities indirectly – by fully forbidding or partially limiting model solutions that, in some scenarios, in opinion of decision-makers, create strategy candidates excessive risky for the system. (Creation of many adaptable candidates is just the first stage of the process. At the second stage, candidates are further screened and compared.)

    The process develops a small set of sufficiently good and reasonably safe candidates. Decision-makers choose the strategy to be adapted judgmentally.

    Will the process generate a comprehensive theory? I am not sure that such thing does exist.

    In short, rather than judge a methodology by a several lines of a comment, why do not you read the referred paper.

  14. August 9, 2017 at 5:18 pm

    Sorry, not adapted but adopted. Chosen for implementation.

  15. August 9, 2017 at 6:07 pm

    The paper also is at my site rcosoftware.com.

  16. August 10, 2017 at 3:18 am

    No, of course, there exists no unique correct theory that presupposes possibility of a single best solution and a best method to find that solution. Let me quote from that paper.

    “Throughout history, the general assumption was that there exists the correct and best decision. In particular, that is assumed in reductionist disciplines – economics and adjacent disciplines, such as DA, OR, and risk management. These disciplines are full of dangerous oversimplifications: “Its (i.e., reductionism’s – Author) leading article of faith is that to every scientific problem is one and only one solution.” [Ravetz (2009)]

    RCO leads not to a best decision, but to a decision most acceptable to the decision-makers. Chosen subjectively.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.