Even though extreme Whorfianism has been criticized, I think the general consensus is that language does have an impact on the way you think. Since his time, many people have done empirical studies and have refined his earliest claims. Some historical criticisms of pure Whorfianism are as follows:
The idea of causality. Whorf claims that language affects thought and the formation of a world-view, but he doesn't really look at the evolution of language and the impact that thought has on its development. It's possible that certain cultures think in a certain way and then develop language, as a tool, to express their perception of the world. Neither Sapir nor Whorf ever claim that language and culture are causally related, so critics who attribute casual determination to their theory are probably misguided.
His methods. Eric Lenneberg, Noam Chomsky, and Steven Pinker have claimed that many of his assertions are based on anecdotal evidence and/or were speculative. He doesn't really do empirical studies or proves that his assumptions are true. Whorf draws many of his conclusions from studying the Hopi language and finding out that there's no way to indicate units of time. He then uses this observation to draw conclusions about the reasons behind certain behavior patterns or thought processes within the culture.
Translation. Lenneberg also claimed that Whorf didn't actually speak Hopi, so the mere fact that certain conceptualizations had to be translated to English disproves linguistic relativity. This criticism is kind of harsh, though, because it's practically impossible for academics to actually speak and understand every language that they study. It should be enough to gain an understanding of how a certain culture thinks without actually having to think that way or speak the language.