Illustration: James Yarema on Unsplash
This essay is my attempt to convey my understanding, and provide a summary, of the key ideas around Virtual Reality presented by David Deutsch in his book, The Fabric of Reality. All errors of interpretation are mine. I highly recommend you read the book.
First, a definition of virtual reality. Virtual reality (VR) is any situation in which a person is artificially given the experience of being in a specified environment (such as a flight simulator).
Before we explore VR further, it’s important to understand that VR is a branch of computation. And we know that computers are physical objects and computations are physical processes. What computers can and can’t do is determined by the laws of physics alone. So the same laws of physics apply to VR.
One of the most important concepts of the theory of computation is universality. A universal computer is usually defined as an abstract machine that can mimic the computations of any other abstract machine in a certain well-defined class.
At this point, you may ask, why does universality in computation matter with respect to virtual reality? The significance lies in the fact that universal computers, or at least good approximations to them, can actually be built, and can be used to compute not just each other’s behaviour but the behaviour of interesting physical and abstract entities. The fact that this is possible is part of the self-similarity of physical reality (more on this self-similarity property of reality in a future post).
So, by understanding the universality of computation, we can now begin to uncover virtual reality’s significance.
Second, VR’s key functionality lies in the fact that it can be programmed. And since we experience our (physical) environment through our senses, any VR generator must be able to manipulate our senses, over-riding their normal functioning so that we can experience the specified environment instead of our actual one.
At the time of writing (1997), Deutsch stated that we can in fact manipulate our senses fairly accurately, using movies and sound recordings, though still not accurately enough for the simulated environment to be mistaken for the original. I suppose, in the 25 years since Deutsch wrote this, modern virtual reality hardware machines (such as the Quest 2) are quite possibly even better, in terms of manipulating our senses, at rendering those simulated environments.
Okay, let’s talk specifics: when a user interacts with a VR machine (in contemporary situations by strapping a pair of goggles), the intention is for that device to generate specifiable sensory input to the user; that is, it would provide specific pictures, sounds, odours. In turn, the user would interact with the simulated environment in specified ways.
How does it work? The VR machine would capture a user’s information and pass it to a computer. The computer would then calculate what the user should be seeing and feeling. It then responds by sending appropriate signals to the image generators. And it feels real because the device provides ‘tactile feedback’ to the user.
So, in the scenario of a user sitting or experiencing a VR rendering of a flight simulator, that user would respond to the throttle of an engine in virtual reality, in a similar manner as they might have, if they were placed in the cockpit of an airplane flying the actual plane. This is an extension of the ‘kickback test’. The ‘kickback test’ is derived from the story of Dr. Samuel Johnson whom, in order to demonstrate the falsity of solipsism, kicked a rock and said “I refute this”. In short, the user’s response would be complex and autonomous.
Virtual reality is significant because it demonstrates that the human capacity to understand the world is inherently unlimited. And this Deutsch argues, is a property of the multiverse at large (more on this later).
What are virtual reality’s limits? And what sorts of environments can be rendered artificially, and with what accuracy? Deutsch argues that the only two limits are physics and the principles of logic. The limitations of technology are merely transitory so we won’t explore them here.
Deutsch focuses on the user’s external experiences, not internal ones (like feeling nervous on your first solo landing of an aircraft) because internal experiences are indirect and they cannot be artificially manufactured using a VR generator. One more quick note on internal experiences before we focus our attention back to external experiences. We can conceive of a technology beyond virtual reality, which could induce specified internal experiences; they would have to override the normal functioning of the user’s mind as well as the senses. In other words, they would replace the user with a ‘different person’. Deutsch also excludes logically impossible experiences as they cannot be rendered artificially. Remember that a virtual reality still has to be programmed, which means it needs to adhere to logical rules (i.e factorising the number 181, as that is logically impossible - it is a prime number; or unconsciousness because when one is unconscious one is not actually experiencing anything.
So, excluding internal experiences and logically impossible experiences, we are left with large classes of logically possible external experiences. These experiences are the group of logically possible experiences, some of which may not be physically possible. Those ‘laws of physics’ referred to by Deutsch are a “yet unknown rule determining the initial state or other supplementary data necessary to give, in principle, a complete description of the multiverse.” Simply put, the environment is physically possible if it actually exists somewhere in the multiverse (at least in some universes). And logically the inverse is true: something is physically impossible if it does not happen anywhere in the multiverse (more on the multiverse in a future post).
Alright back to what a virtual reality machine can do. Its repertoire is the set of real or imaginary environments that the machine can be programmed to give the user the experience of. And its limits are the limitations placed on it by the laws of physics.
I mentioned image generation as the key activity a VR undertakes, so let’s start there. Let’s explore what properties an image generator must have to render a true virtual reality experience to the user: first, it must be able to override the normal functioning of the sense organs; second, those images must resemble those that would be produced by the environment being simulated (it must feel authentic); and there needs to be a mechanism to intercept and suppress the (normal) images produced by the user’s actual environment (i.e if the user was in a flight simulator, the VR needs to override the user’s physical experience of being in that cockpit simulator machine). In a sense, these image renderings are manipulations. They are designed to create artificial sensory experiences.
It’s important to note that all of the sensory manipulations I have described above are physical operations and can only be performed by processes available in the real physical world. It has often been argued that the whole of what we think of as reality is merely virtual reality - a Great Simulator. But that can’t be true. Here’s why: if our world were composed entirely of software, then it must be composed of a physical piece of hardware. And that hardware is grounded in physical reality. It must be programmed in some physical manner to artificially render a virtual reality. In short, Deutsch shows that the explanation of reality being a part of VR is an infinite regress (The Beginning of Infinity, p191).
In terms of sensory experiences that can be manipulated, light and sound can be easily physically absorbed and replaced fairly easily. But that is not true for gravity, as the laws of physics do not permit it. But whilst the physical experience of defying gravity may not be possible, we can conceive of a future where a technology ‘tricks’ the mind by bypassing the sense organs entirely to stimulate a particular signal such as a smell or scent, or the authentic sensation of weightlessness, despite the virtual reality still being physically occurring under normal gravity. In this way, Deutsch explains that the laws of physics do not actually impose a limit on the range and accuracy of image generations and thus every possible sensation can be artificially rendered and experienced by a user.
Earlier, I had flagged the significance of the universality of computation. Its significance with respect to virtual reality is that it, in effect means that it will one day be possible to build a single machine that can render any possible sensation. It would, in VR terms, be a universal image generator. The possibility of a universal image generator unlocks much. For that we will need to understand the exact codes used by our sense organs and develop sufficiently delicate techniques for stimulating nerves. And using this information and techniques, we’ll need to be able to artificially generate nerve signals accurately enough that the brain will not be able to distinguish between the artificially rendered signals and the biological ones our sense organs would send.
So, how should we judge the accuracy of an image generator you may ask. Well, if an image generator is playing a recording taken from life, its accuracy may be defined as the closeness of the rendered images to the ones that a person in the original situation would have perceived. And what do we mean by ‘closeness’? What we simply mean here is those images that are perceived by the user as being relatively indistinguishable from what was intended. If it is truly indistinguishable from the intended image, then the virtual reality has achieved its mission of being accurate.
Earlier, I touched on the significance of the universal generator. Let’s define the properties of such a universal generator: what makes it universal is that, given a recording of any possible image, it evokes the corresponding sensation in the user. How remarkable! For example, with a universal auditory sensation generator - the ultimate hi-fi-system - the recording might be given in the form of a compact disc. Now, initially it seems like we run into a storage problem. To accommodate auditory sensations that last longer than the disc’s storage capacity, we must incorporate a mechanism that can feed any number of discs consecutively into the machine. We already have mechanisms for storing an infinite amount of information on external storage devices such as hard discs, USBs, data centres (what we refer to as the ‘cloud’) etc. The same proviso holds for all other universal image generators, for strictly speaking an image generator is not universal unless it includes a mechanism for playing recordings of unlimited duration (memory and storage). Another condition on the universal generator, the machine will require maintenance or else, with time, the images it generates will become degraded or may cease altogether. So there is another physical condition on its universality. In short, a universal image generator is only universal in a certain context, which it is assumed to be provided with such things as an energy supply, a cooling mechanism and periodic maintenance.
Importantly, the aforementioned conditions do not preclude the machine from being disqualified as a truly single, universal machine provided the laws of physics do not forbid these needs from being met, and provided that meeting those needs does not necessitate changing the machine’s design.
What other qualities does a true virtual reality need to be comprised of? A true VR needs to be interactive. That is, the images it renders cannot be a static set of images (like a movie), but in fact need to reflect, at least in part, what the user chooses to do. In short, it is an information processor as well as an image generator. So, by logical extension, it ultimately needs to keep track of everything the user does that could affect the subjective appearance of the simulated environment.
Interestingly, because it is the human mind that affects the body and therefore by extension the outside world, through the emitting of nerve impulses, this means that in principle a VR generator could theoretically obtain all the information it needs about what the user is doing by intercepting nerve signals travelling from the user’s brain to the rest of the body. The signals in effect would be intercepted and detoured to a computer and decoded to determine exactly how the user’s body would have moved. This also means the simulated body could react differently from the user’s real body, in order for it to survive in simulations of environments that would (ordinarily) kill a real human body, or where simulations would otherwise cause severe malfunctions of a user’s real body.
Let’s turn to what a VR generator must have. Deutsch describes three key components as necessary:
A set of sensors (which may be the aforementioned nerve impulse detectors) to detect what the user (their mind, not necessarily their body) is doing;
a set of image generators (which may nerve-stimulatory devices); and
a computer in control.
Image generators merely provide the interface - the ‘connecting cable’ - between the user and the true VR generator, the computer. For the computer creates the simulated environment.
It is the computer that provides the complex and autonomous ‘kicking back’ that justifies the world ‘reality’ in ‘virtual reality’. The connecting cable contributes nothing to the user’s perceived environment, being from the user’s point of view ‘transparent’, just as we naturally do not perceive our own nerves as being part of our environment. Thus VR generators of the future would be better described, Deutsch writes, as having only one principal component, a (super) computer, together with some trivial peripheral devices. Deutsch notes the practical challenges of actually intercepting all the nerve signals passing into and out of the human brain, and the difficulties from a technical view, to cracking the various codes. However, as I wrote in my earlier post, all problems are soluble. So this set of problems are merely transitory and finite. They will only need to be solved once. After that, the focus of VR will shift once and for all to the computer, to the problem of programming it to render various environments.
The set of environments that we shall be able to render will arise from what environments we can specify; they will at some point no longer depend on what sensors and image generators we can build.
Now to the core of it: specifying an environment will mean supplying a program for the computer, which is at the heart of the VR generator. In VR, there are no particular images intended: what is intended is a certain environment for the user to experience. Specifying a VR environment does not mean specifying what the user will experience, rather it will be about specifying how the environment would respond to each of the user’s possible actions. Here’s a practical example Deutsch uses to illustrate this VR capability: In a game of tennis, a programmer will need to specify in advance the appearance of the court, the weather the demeanor of the audience and how well the opponent should play. But how the game actually plays out depends on the stream of decisions the user makes during the game. So, as you can see, the number of possible tennis games is very large. And this is related to the fact that in VR at least, the number of possible pathways (or tennis games in this example) is simply that the environment occurs somewhere in the multiverse.
And logically, the environment a program renders (for a given type of user, with a physical connecting cable) is a logical property of the program, independent of whether the program is executed. And a rendered environment is accurate in so far as it would respond in the intended way to every possible action of the user. Thus its accuracy depends not only on experiences which users of it actually have, but also on experiences they did not have, but would have had if they had chosen to behave differently during the rendering. This may sound paradoxical, but this is a result of the fact that VR is interactive. This gives an importance difference between image generation and VR generation. The accuracy of an image generator’s rendering can in principle be experienced, measured and certified by the user, but the accuracy of a VR rendering never can be (as they cannot simultaneously experience all the possible experiences in the rendering).
Writing VR programs will become the dominant activity over time. The reason is that to render a given environment for a user with given types of sense organs, a VR generator must be physically adapted to such sense organs and its computer must be programmed with their characteristics. However, the modifications that have to be made to accommodate a given species of user are finite, and need only be completed once. Thereon in, as we consider environments of ever greater complexity, the task of rendering environments for a given type of user becomes dominated by writing the programs for calculating what those environments will do. The species-specific part of the task, in short, is only of fixed complexity and becomes negligible by comparison.
So let’s turn to what the ultimate limits of VR are. Its complexity stems from the fact that the VR needs to be ‘experienceable’ for the largest user base, and therefore VR becomes about ‘rending a given environment’ that many users can experience in their own unique ways, based on how they choose to react within the VR. This is exceedingly difficult as the accuracy of the VR needs to be close, as far as is perceptible, for every possible way the user might behave. In the tennis game scenario, that would take the form of every micro-decision the player makes changing the VR. That would entail a seemingly infinite number of different virtual reality possibilities within the VR rendering (it can be useful to think of them as forks branching out countless times for every decision the user makes). This complexity of VR logically leads to another non-obvious observation: that no matter how observant one is when experiencing a given environment within virtual reality, one cannot certify that it is accurate (or probably accurate). Although a user can probabilistically assess whether a given rendering is inaccurate.
Like scientific reason, we can never validate a theory or experience as true but we can eliminate false (probabilistically) ones. Accurately rendering a physically possible environment depends on understanding its physics; the inverse is also true: discovering the physics of an environment depends on creating a VR rendering of it.
And here, a computer’s description evokes in a reader not just a single image or sequence of images, but a general method of creating many different images, corresponding to the many ways in which the reader may contemplate making observations. The connection between the physical world and the world and the worlds that are renderable in VR is far closer than it looks. There is no such thing as a VR environment that the user would be compelled to interpret as physically impossible. All ‘physically impossible’ renderings are merely interpretations. Thus, if the laws we are using are as close as we can make them to real ones, given the constraints under which we are operating, we may call these renderings ‘applied mathematics’ or ‘computing’. And if the rendered objects are very different from physically possible ones, we may call the rendering ‘pure mathematics’. If a physically impossible environment is rendered for fun we call it a ‘video game’ or ‘computer art’. All the renderings I have described above have an an alternative interpretation, namely that it depicts some physically possible environment.
Imagination is a straightforward form of VR. Our direct experience of the world is VR too. For our external experience is never direct; nor do we experience the signals in our nerves directly. What we experience directly is a VR rendering, generated for us by our unconscious minds from sensory data plus complex inborn and acquired theories (programs) about how to interpret them.
Lastly, reality is objective, physical and independent of what we believe about it. This is an important fact about the nature of reality.
Every last scrap of our knowledge, including mathematics and philosophy, and of imagination, fiction, art and fantasy - is encoded in the form of programs for the rendering of those world’s on our brain’s own VR. All reasoning, all thinking and all experiences are forms of VR. All living processes involve VR too, but humans have a special relationship with it. Biologically speaking, the VR rendering of their environment is the characteristic means by which human beings survive. It is the reason why human beings exist. The ecological niche that human beings occupy depends on VR.
To sum up:
VR is not just a technology in which computers simulate the behaviour of physical environments.
The fact that VR is possible is an important fact about that fabric of reality.
It is the basis not only of computation, but of human imagination and external experience, science and mathematics, art and fiction.
The full scope of VR is unlimited, but in another regard it is also circumscribed.