Colin McGinn takes on Ray Kurzweil:
Contemporary brain science is thus rife with unwarranted homunculus talk, presented as if it were sober established science. We have discovered that nerve fibers transmit electricity. We have not, in the same way, discovered that they transmit information. We have simply postulated this conclusion by falsely modeling neurons on persons. To put the point a little more formally: states of neurons do not have propositional content in the way states of mind have propositional content. The belief that London is rainy intrinsically and literally contains the propositional content that London is rainy, but no state of neurons contains that content in that way—as opposed to metaphorically or derivatively (this kind of point has been forcibly urged by John Searle for a long time).
See this old post on Silly Talk about the Brain Pleasure Centers for more discussion of such silly talk.
What I think is that this is very tricky business, talking of "states of neurons" vs. "states of mind." I'm inclined to think that we can talk about states of neurons without reference to the world in which an organism lives. But, to talk of states of mind, we must include the world in which the organism lives.