Supermodelling

As my final high-level post for a while on AGI, I’d like to give my opinions about “Friendly AI”.

For those who don’t know what that means, consider this: Suppose I were to have a series of conceptual breakthroughs and write a boatload of code and the result is an AGI that is “smarter” than me. Call it AGI-1. Next, suppose that it learned enough to make it more knowledgeable than me about AGI theory and about building software systems. Then, supposing that there is an AGI design significantly better than AGI-1 available within the capabilities of AGI-1, it should be able to build a smarter AGI, call it AGI-2. Then, if there is yet another better AGI design within the grasp of AGI-2, it could produce AGI-3. And so on. If this “recursive self-improvement” process proceeds rapidly (that is, if the development time for each cycle is small), the result would be an AGI system that is a LOT LOT smarter than me.

Alternatively, it could be that AGI-1 by itself might be a LOT LOT smarter than me.

Either way — one has to worry whether we could stop this superintelligent AGI from doing things we don’t want it to do. Like kill us, for example.

Certainly this idea is not obscure. The “Terminator” science fiction franchise (along with many other scifi stories) illustrates exactly this scenario. Friendly-AI cognoscenti tend to frown on that point because they disagree with the technical details and the nature of the future that results, but I think that’s irrelevant. The point is that the masses of humanity are quite aware of the potential danger. They don’t think it’s worth worrying about, because it’s a bizarre scifi thing.

My personal view is that this quite likely will be a real and possibly big problem someday. In fact, soon a very minor version of the problem will be getting a lot of attention. More and more, robotic systems will have the ability to harm people (because they will be more common and will have the ability to control more powerful and mobile physical devices). In the immediate future the issue will not be whether they become homicidal maniacs because they won’t be smart enough for that to even be possible — no way to even have the necessary concepts. First the issue will just be whether their software might be buggy in other more mundane ways. What process should we use to validate software that drives cars? How can we minimize the number of accidents caused by poor decisions in household cleaning robots?

I only mention this case because I think it will naturally bring the question (and the inevitable quips about Asimov’s three laws of robotics) into public discourse.

Now, although I think this will be a problem someday, I don’t think it’s going to be a problem soon. We simply aren’t that close to building an AGI with the appropriate level of smartness. It is possible that the solution will be simple and somebody will find it soon but it seems extremely unlikely to me. It’s true that I cannot draw a definite conclusion from the poor results produced by people who are trying very hard right now to build AGI, but it is relevant. For me to give any significant probability to this kind of scenario with multiple revolutionary inventions comprising many huge leaps of understanding, I need to see something that looks like progress in some direction.

Further, for this to be a problem with any sort of suddenness, the AGI would have to be astonishingly more intelligent than a human; able to quickly make multiple technical breakthroughs in many different fields and rapidly master every field of human expertise. To me, it is a huge stretch to posit that near-term commonly-available computer hardware will have the ability to host such a program.

Suppose I’m wrong. Suppose that some secret group somewhere is solving the problems — or more generally the time is almost right and parts of the puzzle start falling into place quickly. And suppose that it turns out that the core of intelligence is amazingly simple and hardware isn’t a limitation.

It would be desirable if this system were programmed with an “ethical system” that keeps it from harming us (or, better yet, from wanting to harm us) — no matter how many times it redesigns and improves itself, no matter how its code drifts and changes over the entire length of the indefinite future. Figuring out how that could work is the “Friendly AI” problem. Here’s a reference. It could turn out somehow that Friendliness is inherent in any superintelligence, but the fact that humans are not Friendly rather dashes that hope as far as I’m concerned.

It becomes even trickier because solving the problem isn’t enough. The first successful AGI project has to correctly include the solution in the implementation. And, all subsequent AGI projects also have to do so. Even though it seems like a good idea, how can we guarantee that? Or, lacking a guarantee, how can we at least make it very likely?

• We can prevent any AGI from ever being built. In the case where it is extremely difficult to develop AGI and/or it requires large supercomputers to execute at dangerous speeds, an intelligence/military effort similar to how we deal with nuclear proliferation might be successful for a while, until a better solution can be found. But the “easy code, hard takeoff” scenario favored by Friendly AI folks is much more difficult to handle. It would require a terrifying level of surveillance, presumably by all governments under some sort of treaty, or imposed by one or more superpowers. Unfortunately, it might be the case that such a surveillance society is coming anyway to combat more mundane threats like bioterror. The technical capabilities for building a total-surveillance infrastructure might only be a few decades away. I don’t really want to speculate on the details because it’s too depressing. If several significant terrorist attacks occur using WMDs, I could imagine the political will to use the technology becoming real.
• The first AGI can be built to be Friendly, then helped and encouraged to “take over the world”. After that, the AGI can take on the surveillance tasks described above. The one AGI becomes humanity’s partner. In this scenario, we do have to develop Friendliness first, then maximize the probability that it is done right. An open process (or perhaps a closed but very heavily funded process) seems much safer than having a small group work on Friendliness theory and implementation. Even so, the effort should definitely contain a rigorous mathematical/logical framework for proving the correctness of the design.

This is all very scary and unreal-sounding. It is not comfortable to think that the two above bulleted scenarios are the only ways to secure ourselves from extinction in the near future. And, as I said before, I don’t personally think superpowerful AGI will arise soon. And, even more important, I don’t think it will arise suddenly, which means that if the draconian measures listed above cannot be decided on, we might nevertheless survive a slower rise of AGI.

It might turn out that “superintelligence” is impossible… that, for some reason, no AGI can be very much smarter than the human race as a whole. If that’s true we probably don’t need to worry that much about it. But I don’t see a good reason to think that such an intelligence cap exists, and it certainly doesn’t seem prudent to bet on it being true.

So, no matter what, it seems as if solving the Friendliness problem is a good idea, and the sooner it can be solved the better. So far, not much progress is apparently being made, although few people if any are actually working hard on the problem.

The ardent believers in the near-future hard-takeoff scenario come across as rather fanatical and alarmist; to be fair, if they are right then I guess they should be. I’m surprised that they don’t seem to be working very hard on an actual solution beyond just “awareness raising”. As I noted before, the public is quite aware of the issue. The only thing the public needs to be shown is evidence of imminence, but no such case is ever attempted.

I am completely perplexed that the believers don’t have some sort of forum for discussing approaches to solving the Friendliness problem, especially technical issues and underlying concepts.

I have thought about starting such a forum myself, but it’s a lot of work to attempt serious community building, and there is no guarantee of success, especially given the idiosyncratic nuttiness of the interested parties and the apparent intractability of the problem.

Still, I might create such a forum just to see what happens.

Beyond that, though, since I’m out of the AGI game, there’s no need to worry about whether I’ll destroy the planet by letting loose a rogue UnFriendly AGI. I’m not even thinking about AGI.

I have some more tangible things in mind.

At least it should be.

Aside from raw materials in their natural location and state, every single thing of value — physical or abstract — is the product of intelligent action. Applied intelligence creates all wealth. A natural core exists in each of us, of course, and that essence is a large part of what we love and cherish in each other and our world, but all else is the product of industrious mind.

Artificial minds will be just as potent a source of value as our natural ones — actually a great deal more. Their economic impact will be incalculable, many times greater than the sum total value of everything ever created.

On a vastly smaller scale, almost trivial, Bill Gates once said: “If you invent a breakthrough in artificial intelligence, so machines can learn, that is worth 10 Microsofts.” Literally, a couple trillion dollars, though the point of the comment was not to attempt an accurate valuation.

You get the point. Money isn’t the only way of measuring worth, but it is one interesting way.

Now, here’s the puzzle: How come AGI development has produced basically zero dollars profit so far, and why can’t AGI efforts attract even small amounts of capital investment?

I have a couple of possible answers for this.

• Maybe nobody yet knows how to make significant progress toward AGI. Or at least, nobody can convince investors that they know how.
• Maybe it is beyond our ability. Humans, even in groups, are only so capable. As finite creatures, there are limits to what we can do. Maybe building AGI is just too hard.
• Maybe we are making money. There are many different overlapping and even contradictory ways of thinking about intelligence. Over the last few months I have moved toward the viewpoint that, roughly speaking, intelligence == modelling ability. Starting from there, it makes sense to think that most of the entire history of the computer industry comprises the first steps toward artificial intelligence, generating quite a lot of value and wealth in the process as we develop the means for modelling the universe on our machines. The computer industry moves forward every day, they just don’t call themselves AGI.
• What do you think the answer is?

If there was actual progress being made toward AGI, the field would not consist of a marginal fringe club of futurophiles debating consciousness, it would be an economic juggernaut. In our modern world, more and more of what we do is touched in some way by computer software, and that software is unbearably stupid. All of it. Not only does it malfunction with alarming frequency, but even when it does work it is completely clueless about the needs of its users, displays almost no fluency with the subject matter it is supposed to be about, and never learns. If we can make that software knowledgeable, smarter, more adaptable, more robust — even small steps would be huge. And that’s not even taking into account coming technological gold mines like robotics, massive recordings of video streams, ubiquitous networking, immersive virtual reality, microbilling, scientific simulation, and on and on.

Typically, the excuse given for lack of progress toward anything tangible is that all “general” intelligent tasks are AGI-Complete — meaning that the whole problem has to be solved in order to solve a piece of it.

That can’t be right. It has to be a consequence of thinking about the problem in the wrong way. So what’s a right way? I am not certain, but there are many possibilities. Here are a few:

• AGI practitioners by and large think that by breaking up intelligence into “narrow” issues and problem domains, mainstream AI research has lost the dream. But maybe that’s wrong. Maybe general intelligence really is just relatively straightforward combinations of “narrow” intelligences. If so, AGI should be all about the process of rapidly developing narrow AI technologies and making them work together to solve problems in real-world task domains. Yes, it’s hard. So make it easier!
• Maybe focusing on a particular application with large economic potential — e.g. natural language question answering, robotic control systems, or forecasting — from an AGI perspective would provide the right leverage for producing self-sustaining progress. Rather than starting with a system that does absolutely nothing (but does it in a completely general way) and try to make it do something from there, it might be better to focus single-mindedly on gradually increasing the generality of a system that does only one thing.
• What do you think is the right way to think about the problem?

Earlier I mentioned that I have been gradually coming to hold the viewpoint that intelligence == modelling ability, and I have touched on that in other blog postings about concepts being models of things in the universe. My bet on the best approach to being real, relevant, and successful is to move forward from that premise. So that’s exactly what I’m going to start to do, though I am not yet certain how to best proceed so the path will be long and winding.

I will be writing more about this anon.

Author’s note:  If possible, you should play the song Spiraling Shape, by They Might Be Giants, while reading this blog entry.

Listening to people complain is rarely entertaining and I did feel a little bit guilty for complaining about the state of AGI a few days ago.  Nevertheless, I’m going to complain some more in this blog post.  After this, having purged my mental digestive system of accumulated toxic waste, I will be able to move forward with cheerful optimism.

I’m interested in figuring out how to build AGI systems, and so if some claim or topic of discussion gets bandwidth, it is worthwhile if (and to the extent that) it imposes specific requirements on AGI implementation.  If a topic of study or discussion leads to no such requirements, or if the imposed constraints are too fuzzy to pin down specifically, the result is not helpful.  Unhelpful topics which are so interesting that they recurringly or continually suck up large amounts of mental energy and conversation I call Mind Traps.

There’s nothing wrong with playful or speculative forays into potential dead ends, but nasty Mind Traps sap our insight, time, and sanity.

Unfortunately, those stuck in Mind Traps do not agree that the subject of their attention is a sinkhole of futility, so most AGI folk will strongly disagree with me about some or all of my list of Mind Traps.

Rather than belligerently rail against these topics, I am simply going to list them.  My hope is that some reader someday when considering one of these topics as they think about building AGI will ask:  “What specific constraint does this place on an AGI implementation?  Is it really helping me understand, design, or build an AGI?  If there is a specific impact, is it really likely to be the best way to look at the issue at hand?”

Mind Traps (when you see these words or phrases, Run Away):

• Turing Machine
• The Halting Problem
• Computability Theory
• Godel’s Theorem
• Kolmogorov Complexity
• Model Theory
• Mind As Evolution
• Mind As Economy
• Mind As … (hint:  Mind is Mind, not something else)
• Consciousness
• Qualia
• Meaning
• Identity
• Game Theory
• Evolutionary Psychology

Thank you for your attention.  Which useless dead ends did I miss?

This post is the first in a brief series where I plan to touch on some general points about AGI, in preparation for re-abandoning AGI as the self-described focus of my playtime research.  My interest in AGI has not waned… I am signed up to attend AGI-09, which is the next conference dedicated to the topic, and I’m really looking forward to it.  I’ll continue to follow some of the online AGI discussions and blogs, though I skim most of it now.  I’ll keep reading books from authors in the many “AGI-related” academic fields.  There’s a huge library of brilliant books out there.  Right now I’m reading Steven Pinker’s The Stuff of Thought.

Problem is, AGI is still too blurry for my taste.  I like to get my hands dirty, building things and experimenting, making specific arguments, looking for answers to relatively precise questions.  Unfortunately I have not been able to find justified ways to do any of these things applied to “AGI” as a general topic.  It is possible to write code, but the relevance of the code is very doubtful.  It’s possible to debate principles, but the debates almost always end up being about the definitions of words, which wouldn’t be so bad if they ever got pinned down as a result.  It’s possible to prove theorems, but there’s no math that actually captures intelligence in anything but trivial ways (that is, it’s almost entirely math for its own sake, rather than math that usefully models general intelligence).

So to make myself happy I need to pick some plausibly AGI-related topics and focus on those instead of more general issues.  For a while, at least.

It seems clear to me that we are a long ways away from “solving” AGI anyway.

Before refocusing this blog, I do have a couple of things to muse about, though.  The first is my opinions about the current state of AGI as a field of study and research.

The biggest problem, and the underlying source of my discontent, is that there are no clear questions to be answered and no clear goals to be pursued; this reflects a complete lack of consensus about the fundamental issues that should define the field.  In a way, that makes AGI a bleeding-edge field — but in another way, it makes AGI not a coherent field of study at all, when there is scant agreement even about basic things like what the G and I mean.

Given this incoherence, the only way to assess “progress” toward the vague goal is just to look at projects underway, and there are only a few of them really.  The 800 pound gorilla at the moment is Ben Goertzel’s Novamente project (and its open-source offspring OpenCog).  Goertzel has written a great deal of material about his ideas and his model of intelligence, but amazingly it still isn’t enough for me to grok exactly what he’s up to.  So far to me it looks like a large pile of technology choices tied very loosely together with an underlying philosophy of intelligence but I cannot see specifically how any of it addresses the fundamental things about mind that interest me.  Also, the main underlying knowledge representation (probabilistic logic networks) are a really nifty expert-system-style modelling formalism but in a deep sense shares the “suggestively named graph node” criticism that all other such knowledge representation schemes fall prey to — since there is no specific story about concept formation, modal reasoning, semantic grounding, and so on.  I imagine that Goertzel would disagree with me on that point and say that the specifics are coming as the implementation proceeds.  I hope that’s right, and I look forward to seeing what comes from the effort, but at the moment I don’t see any reason to think that the grab bag of algorithms and data structures will add up to emergent dynamics capturing general intelligence.

Venerable old AGI-like projects (SOAR and CYC in particular) are still alive and moving forward after decades of work with no particular end in sight as far as I can see.  I can’t think of any reason that the next few decades will lead to fundamental breakthroughs but as usual maybe I just can’t see the big picture.

Other interesting and attention-grabbing projects such as Blue Brain and Numenta may have AGI-related implications down the road but at present they don’t seem to really count as AGI research.

A number of individuals have interesting AGI projects going…  Steve Reed’s Texai project appears to have a clarity of purpose that is impressive.  Vladimir Nesov appears to be building some experimental foundations based on ideas that seem interesting, although most of the time I can’t really understand his explanations; perhaps his thoughts are too deep for me to follow.  Pei Wang’s NARS system is highly regarded.

None of these efforts appear to me to be anywhere near the stage where a convincing impressive demonstration of general intelligence is on the horizon.  I don’t see how another decade is going to make much difference either… but I do wish them well and will follow their progress with great interest.

Some other writers have very interesting ideas with potential (such as Richard Loosemore and J Storrs Hall) but they don’t write very much about what they are up to.

I respect all this stuff and think it’s really great.

The “AGI community” is another matter.  The publications and conference are more or less passable, but unfortunately the less formal fora are victims of their own subject matter.  It’s a rather distressing mix of tedious self-styled critics, delusional wingnuts with theories and prototypes that are clearly not anywhere near the target, people who mistake databases or simple data structures for knowledge and work diligently on batches of code with no discernable purpose or depth, people who apparently cannot even tell the difference between plot devices dreamed up by science fiction writers to move stories forward and actual technologies, deranged lunatic fringe paranoiacs trying to save humanity from imminent destruction by “superintelligent” rogue AI systems (despite absolutely no evidence of actual progress in the field)…. and plenty of other idiosyncratic frustrating viewpoints.  Adding yet another ridiculous idiosyncratic frustrating viewpoint (mine) to the mix doesn’t seem particularly necessary, noble, or fun.

This is all inevetable I suppose given the hyperbolic end goal that all of us AGI folk share… but the disconnect between what is actually real and important today vs the chaotic noise of uncritical (or supercritical) fantasyland is fun in small batches as an occasional diversion, but quickly becomes ugly and tiresome.

So I’m going to stop discussing and thinking about general AGI issues pretty soon in favor of more tangible related things, which I will write about.

Given the formal purity of mathematics and thus (one would think) its perfect suitability for machine representation and manipulation, why are there no decent Automated Mathematicians?

Mathematicians, it turns out, do a lot more than use rules of inference to produce true statements from a database of axioms and other true statements.

It appears to me as if the formal mechanics of mathematical proof constitute only a small proportion of the activity of mathematics as practiced by general intelligences. Constructing informal frameworks in which mathematical constructs can be developed and expressed is much more important. More specifically:

• invention of terms and relationships between them
• generalizing or otherwise modifying mathematical ideas
• development of methods, processes and arguments (e.g. mathematical induction, numerical approximation techniques)

Trouble is, these tasks would seem to be hardly different for mathematics than for any other area of intellectual endeavor. With the exception of the mechanical details of proofs, mathematical concepts are  exemplars for at least some of the things that concepts in general must be and do. This may make math an interesting lens for viewing general intelligence — but mathematics would seem to be an “AGI Complete” subject overall. No surprise that it remains “solved” in only the most trivial and superficial senses.

Math is interesting, though! As far as I can tell, what we call “progress” in mathematics comes from two different approaches:

• Build a mathematical model of some phenomena. Perhaps later generalize the model or modelling technique to cover more phenomena. I guess this would count as applied mathematics.
• Build an abstract structure with interesting properties. What exactly are the properties that make the structure interesting? This counts as pure mathematics or maybe abstract mathematics.

AM and similar efforts have always acknowledged these kinds of points, at least in part. They are often referred to as conjecture generators or something similar. Unfortunately, this strikes me as a rather shallow metamathematical approach, resulting in surface imitation at best. I have the same reaction to simple formal approaches to mimicking analogy, such as Douglas Hofstadter’s Copycat. We must dig deeper for the source of mathematics and mind.

One question that always nags at me is what exactly should an AGI do? I think others are puzzled by my confusion on this subject. Surely the answer is everything! That’s not very helpful in a practical way though. So in an earlier post I tried to sketch out what I thought an AGI should be built to do. Even so, it is too much for me to wrap my mind around.

How can progress toward AGI be measured? It’s tricky: defining a task on which performance can be measured causes a focus on something specific when generality is what’s needed. Even the famous Turing Test (in its current laughingstock incarnation, the Loebner Prize) falls prey to this… clever trickery is so far more effective than generality.

After much thought, I have started to focus my thinking on two different tasks. Both of them are inspired from early work in AI and more recently by J Storrs Hall’s excellent book Beyond AI. Both tasks seem like reasonable steps along a pathway toward the previously described end goal for an AGI; both encourage a certain degree of generality and are open-ended enough to have no simple solution. Both lead to interesting demos even at early stages of progress, and both are immediately understandable by other researchers.

Task 1: Super-SHRDLU. Given an environment with objects in it and an AI-controlled manipulator, the goal is to learn how to interact with and model that environment, and discuss it with a human partner. This task will go beyond Winograd’s original SHRDLU (one of the most impressive early AI projects) in the following ways:

• The environment is more complex than Blocks World. The AI interface to the environment will be via normal sensors (cameras, maybe audio, haptic and proprioceptive sensors on the actuator). I envision a first environment being a tabletop with various objects… although blocks may be used, I also imagine things like towels, keys, stuffed animals, string, and so on. Eventually I would like to use a kitchen as the environment, with a fully mobile robot as the manipulator.
• As little knowledge about the environment as possible should be hard-coded into the AI. As an ideal, “everything” would be learned — how to control the manipulator, sensor data interpretation, and the whole ladder of concepts that human infants climb on their way to effectiveness.
• Similarly, as little as possible should be built in regarding language and communication. Certainly no words for objects should be pre-specified. Best if no specific grammar rules are built in either. The AI should learn to communicate with its human partner solely through interaction.

It is not hard to imagine different levels of impressiveness from such a setup. I don’t see a good way to make a “test” that cannot be gamed through special-purpose cheats, but I think that fellow AGI researchers, if shown the results and the code/data built into the system beforehand, could make reasonable judgments about whether the accomplishments of the system are impressive or not.

A final interesting point about this task: It could be done with a robot and a real physical environment, or it could be done in a virtual environment. If a virtual environment is chosen, I would like to try as hard as possible to make it a good virtual environment — the most detailed possible models of objects, lighting, physics, and so on.

Task 2: Super-AM. A different way of looking at the path to the scientist/inventor/engineer AGI I’d like to see built starts by noting that even though formal methods appear to be useful only in certain unusual circumstances, those circumstances revolve around science and engineering. That is, detailed formal models (or maybe “semi-formal models” is a better phrase) are the way advanced science and engineering gets done. The most formal of all things is mathematics. And yet, computers pretty well suck at being mathematicians! I think there’s a good chance that figuring out why will lead to interesting progress.

There’s nothing new about this idea, of course. Doug Lenat wrote a celebrated program called AM in the 1970’s to explore exactly this task.

Three obvious extensions to the task come to mind:

1. Add a language component similar to the Super-SHRDLU task so that conversations can be held about the mathematics, and the system could be taught known mathematics.
2. Add the ability for the system to apply mathematics to the physical world, and focus mathematical efforts on formal models useful for science and engineering.
3. Extend the formal systems of interest to include computer programs, resulting in an automated programming system.

I had always assumed that I would focus on something like the first task as a lens for studying AGI issues — although the exact form of the task I wanted was not clear to me until I read Beyond AI.

At the moment I’m intrigued by the second task, though. While I continue the project of thinking about concepts and models in general terms I’d like to spend some time musing on math.

Question for any readers out there:  is there a better task than these? In an ideal world a substantial number of AGI thinkers would agree that some particular task is interesting enough to be worked on from multiple angles; I think such a focus of attention would help discover the strengths and weaknesses of different efforts. But AGI researchers are kind of like cats, and herding them in this way is probably impossible!

A memory: splashing down from the boat into warm turqoise water off the coast of Belize, hand holding my regulator in place, a chaos of refracted sunlight and the familiar sound of going under the surface.

Our lives seem bound by these events; we guard them jealously — even the painful ones… they arise unbidden to remind us of who we are, who we were.

Perhaps these episodic memories are flash-frozen snapshots of concept activations, chained together in sequence, formed during times of novelty or stress… neural pathways recording grist for the mill of metaphor. Causal traces stored against future need.

I’m not sure we can be human minds without them.

But: can there be minds without them? Minds in general? Minds in the abstract?

There is a tension between thinking about our own minds (the only minds we know of), and trying to puzzle out exactly what the necessary core of minds must be. Perhaps there are many different approaches to mind, different ways of collating data and putting it to use in predicting and controlling the world.

Shall we be heavily inspired by human minds, hopefully grasping at sufficiency, or shall we discard all but reasoned necessity?

Is metaphor just a convenient mechanism? A shortcut mapping of similarities taking advantage of evolution’s empirical observation that form often does imply function? That crude similarities lead to better than random guesses? Are such tricks good ideas for computer minds? Are they necessary for effective communication with human beings? If we scrape away all the tricks, is anything left?

There is very little work available on the abstract approach to mind. I’d like to find more, but most of the reading material at hand involves vague descriptions of human psychological phenomena. We take what we can get I suppose.

Here’s something that bugs me and that I need to study along with these root puzzles about concepts and models: why are automatic theorem provers and automatic computer program generators so pathetic? It seems as if the distilled rationality of formal logic (probabilistic, nonmonotonic, purple, whatever) should capture the essence of abstract mind… but we apparently have not or cannot codify physics in a useful generative way. In fact, it seems we cannot even do much of a job of formalizing that most formal of subjects: mathematics itself — at least not in such a way that anything interesting comes from the effort.

Why is that? I wonder.

Having discussed how the physical and abstract universe contains things (and abstractions and events and relations, which are also things), it seems clear to me that the core funciton of mind is modelling these things.

But what is modelling? Searching for an answer to that question is part of what I’m trying to do, but basically by modelling I mean representation and the manipulation of representations.

A representation is a partial mind state that refers to the thing being represented. Partial mind state is a rather vague term whose meaning depends on the mind substrate; it might refer to properties of biological neural networks, it might refer to data structures in a computer memory. Reference is the key deep issue. Coming up with a clean way of thinking about reference is one of my major long-term goals. It seems certain that the degree to which a representation refers to a thing depends on the usefulness and accuracy of the modelling manipulations performed on the representation. As a quick example, a representation for “flower” refers to physical flowers (in part) to the degree that perceptions of the world allow accurate recognition of flowers.

In practical terms it seems as if a causal relationship between the things being represented and the representation itself is necessary, but it might be that such a relationship is only extremely helpful. That is, the details of a flower category representation could have come from some process independent of flowers themselves (say, from Deep Thought about the implications of the laws of physics, or pure random representation generation) – but observations, descriptions, or other causal links to actual flowers seem more practical.

Following what I think is fairly normal terminology, I will refer to these representations as concepts from now on.

Note that concepts are themselves things. An AGI should be able to work with pieces of itself just like pieces of the external universe, and constructing concepts about concepts is a big part of that. The details, however, are murky to me at this point. What are the “parts” of a concept? What does it mean for one concept to be related to another, and where does that relationship come from? I hope that eventually, after thinking long enough about concepts, ways of thinking about these questions will appear.

The important nuts and bolts place to start, I think, is to focus on the “manipulations” rather than the representations — the point of concepts is that they are the way minds deal with the universe, and we should be able to develop a set of requirements for that.

Some obvious examples: Concepts have to be created somehow. They are used in some way to predict the behavior of the universe. They are used for logical and metaphorical reasoning (although whether minds must use metaphor is unclear). They must be used for recognition. They may be given labels for use in linguistic communication. And so on.

The next phase of my research will involve coming up with this requirements list in a more detailed form. I really don’t want to end up with fifty unrelated bullet points; I am hoping that many of these requirements will end up being different ways of describing a small set of underlying mechanisms… but it’s too soon to start inventing mechanisms without knowing what they need to do.

Besides elaborating on a list I have developed myself from thought and general reading, I want to dive into an analysis of several different works related to this subject. My initial reading list is:

• Minsky: The Society of Mind
• Lakoff and Nunez: Where Mathematics Comes From
• Lakoff: Women, Fire, and Dangerous Things
• Yudkowsky: Levels of Organization in General Intelligence
• Fauconnier: The Way We Think

I expect this requirements phase to take quite some time. In my earlier research I was too quick to jump to hopefully-cool representation mechanisms, but I’m in no particular hurry.  A particular interest of mine is trying to simplify the apparent complications inherent in incorporating time into a conceptual framework.  I am greatly puzzled by this subject.

Besides this work I will continue to think about interesting AGI-related topics.

Why do I want to help build a thinking machine?

Because death is bad.

We bravely talk about death being natural; about mortality charging up our limited hours with meaning; about meeting God in the afterlife… but until the time comes when our suffering becomes unbearable, we do not want to die. We do not want our loved ones to die, we do not want the stranger on the street to die.

And there are other bad things. Hunger is bad. Excessive pain is bad. Lack of freedom is bad. And so on.

In theory, in a grand theory born of audacious yearning, these things can be drastically reduced because they only require moving stuff around. The trick is knowing which stuff to move, building the stuff to precise specifications, and moving it around properly. Stuff like food. Stuff like chemicals. Stuff like cellular repair robots.

Alas, not all bad things can be so conveniently dispatched. People seek power over others, act on hatred and fear, cause harm to achieve desired ends. Just moving some stuff around in a consensual way does not help with that, though motivations for harming others may be reduced somewhat when material benefit is always more easily obtained in other ways.

Still, ending physical suffering, hunger, poverty, and even death is desirable. If only we could move that stuff around properly….

As smart and agile and organized as we can be at our best, we are not that smart, not that agile, not that organized. We need help, and since we’re not going to find it sitting around, we need to build it. We need AGI.

Looking at it purely from this perspective, what I want to build is a computer system that fills the following roles in a “superintelligent” way:

• Scientist, to answer factual questions about the universe and the things in it.
• Inventor, to find creative solutions to problems and new applications we don’t even know we want yet.
• Engineer, to design and build things.
• Manager, to coordinate resources.
• Process Controller, to manipulate the physical world.

In all of these roles, our AGI must be able to fluently communicate with us and understand what it should do (and what it should not do).

Hopefully, this pragmatic view of a thinking machine lets us put aside certain tangential issues, such as consciousness, qualia, and so on. However, for our AGI to effectively communicate with us and understand us it cannot be completely independent of human nature.

If we are going to give an AGI the ability to act, we need to be pretty sure it acts in a good way, and will continue to do so. This so-called Friendliness problem will be the subject of some posts as time goes by. For the moment it is not a large concern although it will be someday. I can muse on the nature of cognition without endangering anybody.

If this is the vision for the far-off desired future, we have to figure out how to get there from here, so I want to start thinking about an intermediate step which is in the right direction and provides sufficient challenges to focus thinking about cognitive architecture and act as a testing ground for theories.

The previous observations about the universe were fairly straightforward, but the universe is more than just physical objects.  In particular, the universe includes nonphysical things.  The first type of nonphysical thing that interests me is a category — an abstraction that refers to groups of things, making a new thing not through spatiotemporal proximity or other physical relation but rather through abstract relationships.

What exactly is an abstract relationship?  I don’t think I even have the right modes of thought to begin thinking about an answer to that question but there seem to be lots of different important aspects to it.  For example, we could say that a planet is “round”.  Round might be thought of as a “property” of a thing or we could say that the planet is categorized as a “round thing”.

But where does “round” come from?  What is the existential status of “round”?

I wish I had a useful answer to these questions.  I do know that this abstraction process, the invention of categorical and other abstract relationships (and their manipulation) is the most important core thing done by minds.  My obsession at this point in my musings on AGI could easily be summarized as:  Where does “round” come from? Wherever the true nature of these category-things may be, it is clear that just like other things discussed previously, they are in general fuzzy in nature.  This likely derives at least in part from the inherently fuzzy nature of the things being categorized.

Whereas physical systems obey the laws of physics, abstract systems obey their own arbitrary rules.  It seems to me that there is an infinity of possible abstract systems, many of which in their elaboration doubtless reveal amazing thinginess.  Conway’s Life is one trivial example, and chess is another, but I imagine superintelligent creatures of the future will take great delight in pondering abstract systems to uncover the beautiful things they contain.  Perhaps there will be a general theory of abstractions, a level of meta-musing about conceivable universes that I cannot even imagine.

Homo Sapiens as individuals and as a species have some limited ability to deal with abstractions — a degree beyond other animals, which also have some such ability.  For practical reasons, these abstractions tend to be closely related to the physical nature of our universe (and thus ourselves) — low-level category formation from sensory input, a type of abstraction built into creatures by evolution, is probably the simplest such ability, and many “hardwired” elaborations on that plan seem to be part of brain function.  Humans have found some hard-won abstractions that are more sophisticated, though — good examples are truth and number.

Truth leads to logic and statistics, number leads to mathematics.  In all of such cases, complicated things are built from simpler things in astonishing ways, using abstract relationships and properties instead of physical ones.  The many many human lifetimes spent slowly expanding these ideas were worthwhile primarily because these abstractions map very well onto the physical world.  We even build abstractions out of the process of building world-modelling abstractions, to arrive at science and rationality.

One point that colors my thinking about building AGI:  Abstract mathematics include interesting categories like “prime numbers” about which astonishing things can be proved and subsequent simple knots of thinginess emerge.  These categories are different than most — “prime number” is not vague.  It is precise.  While that allows long chains of perfect reasoning, most of the things we as thinking beings living in our physical world deal with are vague and not suitable to similar methods.  Despite the apparent truth of reductionism in the sense that all physical reality arises from some root precise formal system, things at larger scales do not have effective formal descriptions (pure reductive descriptions are not effective because of both their unknowability and unwieldy size).  For this reason I am skeptical of approaches to AGI that are based on formal logic and deduction; such a method for defining and referencing things is not expressive enough to work with actual things and seems bound to leak meaning like a sieve.

Nevertheless, when trying to come to grips with the questions “where does ’round’ come from?“, “where does ‘knot’ come from?“, “where does ‘democracy’ come from?” the question “where does ‘prime number’ come from?” does appear as if it could be relevant.

Well, this blog post concludes my brief notes on the nature of the universe, although not in a really satisfactory way.  It’s rather vague.  This may be inevitable to a degree given that the things of our universe appear to be inherently vague, but apprehending things (both physical and abstract) and then working with the results is the core thing that minds do.  It’s what thinking is, and to the extent that we can build machines that do it effectively, we will have built machines that think.

So then, how to go about it?