The thing that is going to be interesting is now that we have essentially cheap, ethically clear, and realistic digital 'people', what are the experiments that we can do with them and what can we uncover? I'm a little flat-footed even as to the questions that we can ask them now. At the very least, we can use them to 'dry-run' surveys and experiments and have better data collection and stress-testing. Like, you can now generate realistic data now and use that to run the stats while the real surveys are coming in.

More seriously, it seems to be essentially the idea that “surpassing human intelligence” is not the binary outcome many thought it would be, and that much of what passes for human intelligence interpersonally could be imitation of intelligence.

Essentially, we have 'kind' and 'unkind' learning environments.

To be successful in a Kind environment, you drill-and-kill. The feedback is near instant and the ranking is clear. These are things like golf, classical music, and chess.

To be successful in an Unkind environment, you learn as much as you can. The feedback is infrequent and the ranking is murky. These are things like tennis, jazz, and business.

I'd think that the compounding interest only plays in the Unkind environments, as you can make new connections on the new data you've got going in. In the Kind environment, new data doesn't make a difference as you're just trying to be perfect at the thing you're focusing on; if anything it's an impediment.

But, if you want, you can replace "some intellectual capability" with "some capability typically associated with intelligence". Ability to solve unseen logic puzzles, for instance.

Both. The Turing test is silly because it tests people's prejudices and presuppositions about machines, not objectively the machines themselves.

Also people's presumptions will quickly change as we get used to LLM output and we'll start detecting LLM speech with greater precision.

My take is that the whole chinese room argument rests on extremely nebulous and shaky definitions/assumptions, rendering it worthless.

It seems fairly obvious to me that the system of operator + rulebook does "understand" chinese, for every practical definition of "understanding".

Another counterargument would be simple physical simulation: If you built a computer program that could simulate a human by the atom, then you would either have to concede that the resulting machine does "understand" for all definitions that matter, or you have to admit that you believe in magic [1].

[1]: or desperately grasp for loopholes, like nondeterminsitic physical micro-interactions, but might as well call that magic.

This may perhaps be more obvious to a naturalistic philosopher or natural scientist, than a computer "scientist" (ie., a mathematician).

The meaning of the term "pen" in "pass me that pen" includes the pen. So when this room is asked, "pass me the pen" and it replies "i cannot pass the pen" (or whatever it replies) -- it should be obvious that the person in the room, or any function of their activity, has never acquired any reference to "the pen". It is wholly unaware that there is a pen at all.

The purpose of this thought experiment is to show that syntactical correctness or apparent "arrangement of symbols in 'a' correct order" is radically insufficient to evidence semantic competence.

This, again is perhaps more obvious to scientists -- the symbol order is only a proxy measure of semantic competence in people. It's trivial to come up with processes which clearly lack the capacity for such competence and yet are measured (/observed) to produce symbols in the right order.

In many ways, it's an over-engineered thought experiment. However I'd say Searle was baffled that more obvious phrasings of the problem seemed to confuse others, ie., that an observation of symbols isnt an observation of meanings -- one isn't a reliable measure of the other. Only under very many additional conditions does such a relationship hold in people.

Turing was not interested in producing systems that had such competence, so he may well agree with Searle in some ways at least. However, many students of computer science receive no empirical education whatsoever, and lack the basic vocabulary and understanding of the nature of the problem of meaning.

Eg., that in order to mean "pass me the pen" one must be able to acquire a reference to "the pen" which any system unable to observe its environment at the very least cannot do.

Turing machines lack devices, and hence lack any capacity to in principle refer to objects in the world. The only thing a turing machine can be said to do is express an abstraction ( a function of nat -> nat) -- since it is an abstraction.

No capacities follow from expressing such a computational abstraction -- Searle thought the chinese room made this obvious to those who didnt find it so. But he was baffled that anyone didnt already find it obvious.

One could make the same point with physics, rather than with meaning. Eg., the earth orbiting the sun computes +1,-1,+1,-1 .. and so does an infinte numer of physical processes that share no properties with the earth, or the sun, etc. Thus just because we observe +1,-1,+1... does not mean that "inside the chinese physics room" there's an earth orbiting the sun. It could literally be anything.