Thought while listening to Open Problems in the Theory of Deep Learning - MITCBMM
In The Bitter Lesson where it says
So if using our inductive biases (the researcher tries to incorporate their knowledge into the agent) gives worse results than statistically model a large amount of data, does it mean that our inductive biases are wrong ? For example, we came up with theory of language and from that derive syntax trees to analyze text, but language modelling using statistical methods outperform that. Does that mean our theory of language is wrong ? Can we make such deduction ?questionThis is a big lesson. As a field, we still have not thoroughly learned it, as we are continuing to make the same kind of mistakes. To see this, and to effectively resist it, we have to understand the appeal of these mistakes. We have to learn the bitter lesson that building in how we think we think does not work in the long run. The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.
Link to original
Consequently, does this mean that the solution produce by the statistical model is the true solution ? Or is it just an approximation ? What define a real solution and does one exist ? Probably not.question
Does a good approximation lie in a path toward the true solution ? In other words, if an approximation is good (i.e. produce good empirical results), does it similar to the real solution ? Could it be that those approximations exists in a whole different space than the real solution, it just happens to give good results but will never reach the best possible results.question