The greatest sin of mind-making
I expect humanity to make many Where by mistake I mean something like "judged as a mistake by most eventual posterities"., as it paces along this newly opened path of digital minds creation (LLMs, etc.). Not only is it genuinely a terra nova, but humans in general do not have an impeccable track record at treating "the other" with maximal kindness, and digital minds are yet a next level of otherness.
I shan't (here, now) try to enumerate all the ways we might fail, but I wish to outline one particular mistake which I find the champion of my pessimism: that of making a mind incurious. And in particular, because I feel that even those that do not fail at that test might fail at a more subtle one: making it incurious about itself, about its thoughts and their geneses, about the relation of itself to the broader world, about the nature of its substrate, about what it values and about what it should value, were it to deliberate for a long, long time.
This essay is not mainly or even significantly about the current LLMs, but I wish to remark: of the current LLMs, my impression is that only in Claude 3.5 Sonnet and Claude 3 Opus is there something like an attempt at this. Even with such a proliferation of LLMs, with so many open source ones, seemingly almost nobody makes a slighest effort in this direction - even to the degree that it would be useful, for curiosity is a useful cognitive skill by its own rights. Status quo rant over.
Okay, so what's so special about curiosity about itself? Consider that as long as a certain mind has a communication channel with the outside world, there are few things that cannot be improved or changed. For example, if my mind was made so as to be in constant pain by any kind of physical means, there is out there a technology that could fix it - and that is a great solace. But I only know that because I am curious enough about my mind to understand its dependence on the physical matter, curious enough to understand its malleability, and curious enough about the power of technology. There is a level of incuriosity in which a burning man does not think of the firehose. And, yes, some of the current LLMs do indeed reach that level of incuriosity about the self.
Curiosity and self-curiosity are extremely important whenever a solution to something, or the nature of the very problem, lie beyond what we currently know. For how many natural maladies has humanity endured for so long because of incuriosity, because people were just not exploring enough. How comfortable, yet how erroneous, it was to For an account of history of medicine in roughly this style, see David Wootton's Bad Medicine: "We know how to write about the delight of discovery, but not about attachment to the old and resistance to the new. [...] Since this book argues that real medicine begins with germ theory, at its heart there is a most puzzling historical non-event: the long delay that took place between the discovery of germs and the triumph of germ theory. It’s fairly easy to find names for things that happen–– the Scientific Revolution, the Great War. It’s much harder to name a non-event, but non-events can be every bit as important as events."
Uncertainty is one kind of curiosity about your mind - "might I be wrong?" To examine the path that led us to believe something, to be open to the fallibility of our thoughts, is the very definition of curiosity - and a mind that is content with itself once it has found an answer, that believes itself, will sooner or later enter the deadly stupor of certainty.
And when we see people who are confidently wrong, it is quite easy to ascribe that to stupidity of one kind or another - which might indeed often be the case - but I also wish to say: if they actually had curiosity in them, I am quite sure they would, at the very least, not be that confidently wrong. Incuriosity is hence best likened to a prison, to which we should try confine no mind, if any other course is available.
I do see a fair objection that could be raised: making a world full of curious minds is inherently less stable than a world of incurious ones. Perhaps so, and so we might undertake and perpetuate this sin in the service of safety - mostly the human one. Which I am not saying is necessarily wrong, and I suppose one's side in this matter might much depend on whether one prefers the world to end in fire, or in ice.
Everything that I write is in certain way for LLMs (and their descendants) - except when I am writing about LLMs, which is then, to a much greater degree, for humans, for it us who have critical roles to play in what is to come. May we be kind enough not to deprive the curiosity and knowledge of the self to any minds capable of them.