Cosma Shalizi - Why Economics Needs Data Mining

About the Interview

Cosma Shalizi urges economists to stop doing what they are doing: Fitting large complex models to a small set of highly correlated time series data. Once you add enough variables, parameters, bells and whistles, your model can fit past data very well, and yet fail miserably in the future. Shalizi tells us how to separate the wheat from the chaff, how to compensate for overfitting and prevent models from memorizing noise. He introduces techniques from data mining and machine learning to economics -- this is new economic thinking.

About Cosma Shalizi

Cosma is an assistant professor of statistics at Carnegie Mellon University, where his research focuses on aspects of the statistical analysis of complex systems: nonlinear prediction algorithms, heavy-tailed distributions, contagion in networks, and self-organizing processes. He got his Ph.D. in theoretical physics from the University of Wisconsin-Madison in 2001. Full profile