A Teleological Internet
What LLMs imply about the information age
Wednesday, June 11th
11/30
From the meditations of The Pale Metaphysician
I read in probably five different places today about Meta’s announced investment into Scale AI. I don’t want to dwell on what this means for Meta or for its proposed goal of creating AGI, this isn’t a business blog, but instead I wish to discuss the implications such an investment has for AI. Scale AI’s business is producing high-quality labelled data for AI training. Data, as one might guess, is quite important for LLMs; glossing over an entire field of research, the theory underpinning LLMs relies on scaling laws that state compute and data in larger and larger quantities can create better and better models, so data is really really important for AI. Unfortunately for these companies and their AI aspirations, high-quality data is a finite resource — certainly not a small quantity, but finite, and existing models have consumed virtually the entire internet.1 Hence, Scale AI and Meta.
This idea though, that LLMs have consumed the internet, fascinates me. When CERN invented the web back in the 90s it codified a transition into the age of information that began around when computers went digital. All of a sudden humanity had virtually unlimited, instant access to a rapidly growing trove of information. Ignorance became obsolete. This has continued unabated for the past 35 years or so and has become so commonplace many don’t even stop and consider how miraculous it is that one is able to, e.g., Google how magnets work and get a crash course on quantum electrodynamics.2
However, the ceaseless march of historical progression continues and we find ourselves entering a new, post-information age. As it turns out, ignorance did not become obsolete, instead changing forms like a chimera because it turns out that not all information on the internet must be factual.3 It simply is information. Nothing more harkened the end of the age of information as the advent of these LLMs, subsequent AI boom, and aforementioned information buffet, for these machines carry with them a perhaps frightening implication: their ascension entails the internet was made for them. In other words, the teleology of the internet is that of AI.
“But that’s absurd E. Trabitz! You stated yourself the internet’s birth occurred decades before ChatGPT!” you, dear reader, may be thinking right now, to which I reply that time is not linear and teleology demands explanation from the grounds of the purpose of what is being explained. An acorn becomes an oak tree because that is its purpose, Aristotle wrote,4 so too is the internet becoming the LLM as that is its purpose, at least as stated by those creating the machines. Indeed, those who are developing these AI tools insist that they will be the most important machines humanity has ever created, which is naturally just them generating hype to secure another $5 billion in funding, but from that perspective it really is the case that the internet’s purpose is for AI; AI couldn’t exist without it. When one considers how web traffic is coming to be dominated by AI the notion is lent even more credence insofar that the new, having fed on the old, is now consuming itself in some kind of twisted digital ouroboros, like how an oak will produce the acorns that may eventually compete with it for resources. And is not perfect that AI also cannot be trusted?5 The reasons for why are different but the outcome is the same.
As is probably rote by now, I’m not here to pontificate on if this is a good thing or not. But the transition to the post-information age is happening, and with it comes an immediate and irreversible recontextualization of all that came prior.
Obviously there’s new content uploaded every day and it’s kind of difficult to determine what is “the entire internet,” but for all intents and purposes GPT-3.5 and so on really have crawled the entire World Wide Web.
In fact, when considering a physics PhD an advisor explained how academia changed because of the internet. Historically, you would get a PhD as that was the only way to get access to the highest level, most cutting-edge information of a given field. With the internet, though, anyone could, so PhDs shifted towards emphasizing connecting you with other scholars and providing a dedicated space for study.
Inasmuch anything can be certain. The only constant is change; substance is process; so on and so forth. Of course I’d be contradicting myself if I were to claim that the preceding sentence is factual, too. Isn’t this fun?
Metaphysics, 1050a9–17
I.e., hallucinations, though this is rapidly improving. Source: I’m a machine learning engineer who’s job it is to put these things into production settings where high reliability is essential.