Although nowdays the phrase "AI" invariably means "huge generative models, especially
language models, trained on lots of data", the field of artificial intelligence is
actually wider, deeper, and indeed much older, then most people realise.
Over the last four decades my work has spanned all the principle areas of AI, its
junior brother data analytics and predictive analytics, as well as the associated areas
(by virtue of complexity) of compilers and operating systems.
I have categorised this research into seven AI or AI-related sectors:
From a pragmatic perspective we can regard AI as the "acquisition and manipulation of
knowledge". The form that knowledge takes is important, and can be:
The following diagram illustrates my involvement in the development of the field of AI.
Definitions are dashed-underlined . Sadly some of the
work of which I am most proud is subject to strict non-disclosure restrictions, and not
discussed here.
In the 1980s, Artificial Intelligence was very much an unusual,
almost eccentric, discipline. Applying it to industrial problems, whilst not unheard of,
was rare but exciting.
Transformer Expert System
In the 80s I built an expert system to determine when critical equipment -
electrical transformers in the UK power grid - would need servicing, using measurements
which could be obtained without taking them out of use. (This was typical of the
pre-privatised Central Electricity Generating Board, a well-managed chartered industry
whose metier was to produce reliable electricity for the country, and was always
developing ways of working efficiently.) It was programed in BASIC!, and
ran on a PC with 640k address space or less, which at the time was considered a high-spec
machine.
In these times, many "routine" problems, such as near-optimal
route-finding were the preserve of artificial intelligence, and their solution -
especially on "modest" machines, was far from obvious. Problems such as chess were
studied and even sponsored as test-beds to develop tools, techniques, and understanding.
In a large part, this approach was succesful.
Chess
Whilst studying Computer Science at London University I wrote a chess
program in Pascal, as an exercise in recursion. Chess lives on as
Imp Chess .
Chess and similar programs exemplify the import AI problem of search-space traversal: how,
in an unfeasibly large domain of possible solutions, do we find the answer to this
problem?
(As an exercise in complex programming, every aspiring programmer should write a
chess program, a compiler , or an
operating system .)
In the early '90s it was not clear that computer
programs could do anything useful at a semantic or understanding level. And a document
could easily be too big to load into RAM. None the less,
we had a go...
Multi-Lingual Summariser
I believe I wrote the first industrial-grade multi-lingual text
and html summariser. It:
used a technique I developed called Discussion Flow Analysis ("DFA") on a
tree-representation of the parsed text. DFA endeavours to produce a summary of n
sentences which, when read together, have a discussion flow as similar as possible to the
original document, so it is better than just choosing the “best” n sentences. (A
heuristic avoids the combinational explosion the pure technique would produce.)
can even evaluate summarise tables (keeping only the relevant rows or columns)
understood images from their context
was extremely fast, so that summaries could be shrunk and grown in real time by means
of buttons. It achieved this partly by having two internal phases: an analysis phase,
performed once, and a synthesis phase, performed when one of the buttons was pressed
worked in an environment of scarse RAM: it was typically not possible to load an
entire document into memory, and this was another reason for structuring the analysis
into passes, like a compiler
won an industry award, and was licenced by IBM. (Who were, by the way, a delight to
work with, and always behaved with impecable integrity.)
is multi-lingual, even understanding languages such as Japanese, which according to
Japanese speakers were perfectly respectable summaries. Watching a
program you wrote summarise a language you don’t speak remains a surreal experience even
now, let alone in the 1990s.
was able to focus on a particular query, which was particularly useful when used
alongside text retrieval
A client tested the summariser, asking both it and ten people to summarise ten
documents. Each person then scored each of the other summaries. They discovered that
where people could agree on what constituted a good summary, the summariser did well; but
where people could not agree, the summariser did not do so well. This interesting
result hints that some documents are intrinsically summarisable, whilst others are not.
Unlike Large Language Models, this technique is incapable of "hallucinating"
(aka "not working properly", "generating errors", or "just making it up") and therefore
is still preferable in any environment where welfare is at issue. A
python implementation exists today.
One of our developments was to add knowledge
to language understanding. There was no vast repository of internet knowledge available -
there was no internet(!) - so explicitly captured knowledge was used. Arguably this is
better anyway, since this knowledge would reflect the user's perspective.
Semantic Text Retrieval
Inquisitor was a multi-lingual knowledge-based text retrieval engine,
also licensed by IBM. As with the summariser, it incrementally grew its knowledge base,
so that its understanding of the meaning of significant terms was the same as the reader's,
significantly boosting its performance. By allowing those explanations of terms to span
languages, Inquisitor could search across language.
Classification is a fundamental task of AI, which
is now routine - you probably use it as a spam filter.
News Classification
Similar programs, using inducted or curated knowledge, could analyse and then classify
or filter text.
A quantitative golden-set study revealed that, at least on the domain of
news stories, an AI trained with supervised learning - ie given a set of classified stories
from which to learn - that whilst the AI is more accurate than people it is less flexible,
unable to recognise exceptions, and more likely when wrong to be dramatically wrong.
This characteristic of trained AI is still apparent in recent large language models,
which are often confidently wrong.
An innovative application of AI is to reduce work
load, applying human knowledge to human interactions.
Automatic e-mail responder: AutoAnswer
I built AutoAnswer for automatic e-mail responding for help-desks. When it received an
email it would:
See if this e-mail was a response to an earlier of its own answers, in which case
it would forward it on to a human being, on the basis that its own answer had not
been fully satisfactory.
Use sentiment analysis to see if the sender was angry, and also in this case send
it on to a human being, on the basis that accidentally upsetting this sender
might loose a customer.
Use its knowledge base and natural language understanding to see if this was a
question it understood and could answer, in which case it would go ahead and
answer. In a major deployment, this outcome was about 80% of cases, so it reduced
the work-load by a factor of five.
The final case is where it doesn't recognise the question, so it would be sent on
to a human being, who would not only answer the original correspondant but, since
this question and answer is clearly missing from the knowledge base, add it
to the knowledge base.
The challenge was not the sentiment analysis (which was working well in the
2000s), but in finding a knowledge representation which was both powerful (and so could be
applied to a wide range of queries) and accurate, yet simple enough to be written by
domain experts who were not knowledge engineers. This challenge was successfully met, as
AutoAnswer handled 80% of incoming queries for a large help-desk organisation. This
is an effective deployment approach for AI: don't ask it to do everything, just let it
reduce the work-load, in this case by a factor of five.
In the 20th century almost all natural language
representation was in the form of text, whether it be inverted text indexes or n-gram
statistics. A few companies were pioneering alternative, non-textual, language
representations .
Non textual language knowledge: Bullets
Bullets is a form of semantic language representation amenable to extremely fast
manipulation:
Representation: A bullet is a one-way mappings from text (and potentially images) to
overloaded bit-signatures, where that mapping is itself a learnt optimisation. Becuase
the representation is one-way - text to bullet is possible, but bullet to text is
ambiguous - they are intrinsically secure.
Manipulation: Bullets can be 'added' to summarise a set of documents, or multiplied to
yield the cross-relevance of one bullet (perhaps representing a query) to an index of
bullets. These manipulations are extremely fast, partly because of the optimisation
during the learning-the-mapping process. It was one of the few circumstances where
dropping from Pascal or C to assembler was worth-while, because in that way an
'irrelevant' candidate bullet (when doing a search) could be disposed of in as few as
four machine instructions.
I have recently been using a bullets-approach in calculating a form of
word-embeddings used in Large Language Models. Because they are
optimised, they are extremely fast, saving CPU. Because they are lossy, they are
extremely compact, saving RAM; but as that optimisation is a semantic one, the
"information" lost is anyway redundant and not helpful in calculating the embeddings. In
other words, useless "information" is dropped early on in the pipeline.
Biology can inspire AI .
In the pre-spacecraft but photographic era it was thought that earlier drawings of
'spokes' in Saturn's rings or craters on the surface of Mars were imagined. But when
spacecraft eventually visited those planets, it turned out that the human eye had been
doing an excellent job of capturing the fragmentary moments of diffraction limited
'seeing', when the atmosphere momentarily cooperated.
Active Imaging
In the image domain I wrote an astro-photography program, Active
Imaging, which produces sharp photos from the fuzzy video of a telescope web-cam.
Deliberately modelled on the human eye-brain system, by selecting momentary “sharp”
fragments, it assembles an over-all sharp image. Active Imaging can also assemble a
bigger picture, for example if a telescope is left in a fixed position, so the image of
the moon scans across the camera frame due to the rotation of the earth.
The image on the left is one of the input images, from a video stream, with the
relative strengths of each channel superimposed.
On the right is the active image, which shows far more detail.
(For astronomers: an active image is not simply a stacked image, even though
stacking does indeed reduce noise by the square root of the number of frames. An active
image stacks only the 'best' parts of each image, in the same way as an eye, and thus
can capture the momentary diffraction-limited fragments that appear from time to time.)
Active Imaging is still an effective way of constructing diffraction-limited
images and unlike other techniques in use, needs neither a bright guide-star nor an
artificial laser-star. I would still welcome a professional observatory contacting me,
as I believe Active Imaging would be a powerful tool to add to their inventory.
Agentic AI acts as an agent on behalf
of the user, typically ferriting out useful information, and distilling it. We developed
and deployed this approach at the turn of the century.
Aegentic AI: Intelligent Directed Spiders
The "Concept Engine" must have been one of the very first AI agents or intelligent
directed spiders (hence the picture!).
The user would start it by carefully defining the research he or she wanted carried out,
by means of a series of natural language prompts.
The Concept Engine would then research the web, traversing from one web page to another
by means of the hyper-link, climbing the hill of relevance to more and more useful
content. It ran multiple search queues, each with slightly different but of course
related semantic emphasis, to avoid being trapped in local minima. It was able to
interrogate web sites, understanding query syntax on the fly, and could therefore access
the dark web as well.
It would then assemble the information it had found, summarising relevant content (with
the summariser) to produce a comprehensive report which linked back to the original
sources. (A draw-back with LLMs is of course that where they got their knowledge from is
of course so diluted that they cannot validly explain where.) The Concept Engine was
therefore capable of explanation, a valuable characteristic in AI.
We pushed the barriers of AI into analysing and
understanding what it could see using semantic image segmentation .
IOP 360
Intelligent Observation Point 360 can analyse, describe, or take
action on images or video. It reduces the image to a set of components, whose
inter-relationships combined with perspective can then be used to deduce what is being
looked at, or from video frame differences, what is happening.
Under the hood, it uses a fast method of segmenting the image to meaningful components
(ie they can be annotated and would be verified as meaningful by a person) from which
it can deduce a hierarchical description of the scene and from that can deduce what is
being looked at. I do not believe this technique is used anywhere else.
Back in 2000 few would think of putting AI in a Mars lander ,
yet we did, and at the time it was some of the most advanced AI around, and
its unique language analysis technique arguably still is...
Virtual Astronaut
At the turn of the century at Aerospace Scientific we developed the
Virtual Astronaut, which would have been uploaded to Beagle-II on Mars had it landed
successfully. The aim was for people to e-mail the lander which would understand what it
could see and sense through cameras and instruments, read their e-mail, and reply
meaningfully.
The Virtual Astronaut works by reducing all incoming information – including camera
images and language from the e-mail – to sets of assertions, and the architecture is
illustrated on the right. Specialist recognisers were
used for different media types, with language being understood by, primarily, an
Inferencing system. A multi-stage forward-chaining inference engine deduced both
intermediate assertions (of different lifetimes), frames, and second stage rules, which
were then invoked to produce output assertions, which a generative grammar used to write
the reply.
This method of unifying image, data, and language to a common representation
is very effective. It still works and is a fascinating demo.
I re-implemented the Virtual Astronaut in a micro-controller for model rockets,
aircraft, and drones so that people can fly and communicate with a version of themselves,
which will react as they would.
"Explainability is the AI equivalence of
accountability." It makes no sense to hold "AI" responsible for its actions, but it does
make sense to hold either the developers or the user's to account for mis-use, and to do
this we need explainability. Not just in applications where welfare
can be at stake, but also where we need to understand why just to be confident in
results.
Satellite Image Understanding
We subsequently built a Satellite Image Understanding proof of concept to quickly
analyse satellite images, intended for slow space-based hardware.
This functional requirements were it should:
run in the very constrained CPU and RAM of space-hardened hardware, AND
run quickly (within 14 seconds over station constrained by a low orbit)
apply rules which are both powerful and easy for a domain expert to write
detect and distinguish between (for example) clouds, snow, and sea pollution
be able to explain both textually and visually why it thinks this part of the
image is an oil slick
We (Aerospace Scientific) achieved this with a layered knowledge base, which reads a
bit like FORTH programming, with a similar 'incrementally build up the
concepts' paradigm. The lowest level was couched in terms of image processing, the mid
level mapped those concepts to high level image concepts, and the top level (written by
the domain expert) expressed the conditions under which snow, cloud, or pollution would
be seen.
We also investigated a real-time image analysing
system to find safe places to land on Mercury, which at the time meant landing on unknown
territory. This is hard because there is no
atmosphere, so its "rockets all the way down", so the heavy weight of a power supply big
enough to run a radar altimeter is out. We solved this (in the lab) by using image
understanding to calculate altitude and so forth by rate of change of rate of change of
view with rocket firing, along with image analysis of a "safe place to land", whilst
optimising the trajectory to find a route with the highest probability of safe landing.
Simples.
Although "ElizaBots" had existed for generations,
these natural language systems used explicit knowledge. We realised that it would be
good for a language system to learn how to be like a particular human, or set of humans.
This was an early Large Language Model .
Conversation Analysis
Conversation Analysis was the first "Large Langage Model" that I have heard of, albeit
a small large language model. This is how I described it soon after...
Conversation analysis uses a dual hidden-Markov model to learn how to engage in
conversations. It can learn to imitate eccentric politicians by reading their speeches,
and is very amusing, but its performance when learning from social media is poor,
reflecting the sparse knowledge density found.
The dual-model separated the tasks of conversation flow - what needs to be said - from
the task of generating it. To do so, of course, it needed a way of describing the
conversation flow, which I achieved with a concept I termed "pivots". The same technique
now works nicely on, say, Wikipedia. Unlike most LLMs, its architecture means it always
knows where it learnt something, and can therefore evidence its opinions. The dual-level
model also means that it can be multi-lingual, because of the seperation of intent
and expression into separate layers.
Many useful data analytics algorithms are hard to
run in parallel and become less efficient (in terms of running time) with data size.
At the tipping point of "big data" is reached, some problems will become uneconomic to
analyse. Even if we ignore the dubious ethics of building power stations simply
to power the insatiable demands of the "just make it up and most people will neither
notice nor care when its wrong" Large Language Models the world is in thrall to.
Optimisation
Many algorithms and heuristics, including those used to train today's
large language models, can suffer from a "computational explosion" as the amount of data
to process grows. In computer science, this is referred to as the "Order" of an. For
example, if the amount of CPU required is in direct proportion to the size of the data,
so that doubling the data only doubles the CPU needed, this is called "Order(N)".
Unfortunately, most useful algorithms grow much faster. Many are Order(N squared)
which means as the size of the data doubles, the amount of CPU required quadruples, and
this quickly gets out of hand: a thousand times the data requires a million times the CPU.
This is one of the reasons large AI is now gobbling up so much RAM and electricity, and
significantly contributing to carbon emissions.
To mitigate this, I have developed a general-purpose meta-heuristics to reduce the
Order (i.e. running time) of algorithms
so that they become efficient again. It is a meta-heuristic, since it is a wide-scope
method for taking a high order algorithm (eg n-squared), and converting it to a lower
order (eg order n) heuristic .
The technique also estimates the (low) error thus introduced, which tends to be less
than the implicit error of most AI training data sets.
This technique is generally applicable, and I have used it recently in a
word embedding technique. It could be used to reduce the CO2 footprint of many
of today's large AI systems.
Can AI yield insights into fundamental computer
science problems? Yes.
The Halting Problem
Alan Turing proved that no Turing Machine - a formal model of
algorithms - could be guaranteed to predict if another program would ever finish.
Naturally I had to write a program which could do this, and indeed it did, and could even
do so with self-modifying programs.
But all Turing Machines are unlimited in size, whereas all real machines are constrained.
Although the Turing Machine is widely regarded as the 'propper model of computing', I
regard it as the 'propper model of algorithms', since there are precisely zero computers
with the unbounded size of a theoretical Turing Machine.
If we introduce the concept of 'magnitude' - loosely machine size - I discovered that
a program in a 'bigger magnitude' machine (or environment) can predict if a a program in
a 'smaller magnitude' machine' will halt.
It is therefore entirely feasible for compilers to detect if a subroutine which has
constrained resource (RAM, execution instructions, numeric precision) could halt, which
would be very useful for safety-critical systems.
Large AI is responsible for considerable harm, by repeating the opinions
learnt from those with the loudest (internet) voices, and displacing jobs. From giving
dangerous advice, through invasion of privacy, to encouraging suicide.
Prosthetic Senses: Hearing
Many people are, or become, deaf or hard of hearing. I am developing devices which
render the spoken voice as patterns (which can be learnt to correspond to language),
either visual or vibrations. It works by visually (or through vibrations, which
calculations indicate are just feasible) presenting the formants of which speech syllables
are composed, extracting them digitally through the Hartley transform or in analog through
resonant circuits.
Prosthetic Senses: Sight
Similarly many people are, or become, blind or partially so. I am developing a series
of wearable AI devices which give a blind person more information about their environment.
These combine:
sensors (i) ultra-sound or (ii) machine vision with cameras or (iii) light
sensors; with
AI recognisers of the environment from those senses, such as image recognition and
optical flow in the case of cameras; and with
renderers which portray the detected environment through (a) voice descriptions
(b) sound or (c) vibration, which is particularly helpful to those who are also
deaf.
The most promising approach is, in effect, to give people the echo location used by
dolphins, which are, after all, close-ish relatives. It is amazing how quickly the human
brain can integrate a new sense. In this approach the AI is working in harmony with
people, rather than instead of.
LLMs represent "words" as vectors of approximately 1,000 numbers.
By subtracting the vector for "man" from the vector for "king" and adding the vector for
"woman" you get the vector for "queen". However, each dimension (or position in the
vector) does not actually correspond to any meaning. Given that some human languages,
albeit artificial ones, can get by with only 200 or so words, it seems plausible that
there are two many dimensions, and they are simply too big.
Embeddings
The embeddings approach I am developing, where each embedding is an unsigned integer,
has several interesting propterties...
words which have similar meanings have similar values
as bits are dropped (off the right hand side), the now smaller number identifies the
cluster to which that pair, or quad, or oct, ... of words belongs to. In other words,
the semantic relationships of a word is progressively generalised as the bit representation
is truncated.
This can be computed, even for a large corpus, relatively quickly by using two earlier
techniques discussed here:
A Bullets like approach, so that data structures which, whilst
still capturing all the needed information, are comparatively very small;
A technique similar to deterministic clustering to determine
relationships (preserved in those data structures);
Using the Order N optimisation for speed;
The upshot is... fast computation of semantically rich embeddings.
I am also working on a technique for more "traditional" word embeddings in which
each dimension of the vector is semantically meaningful to a person.
Until recently, all general-purpose programming languages, imperative,
functional, OO, or multi-paradigm, instruct the computer as to the operations required to
solve the problem. Most languages transform higher-level abstractions to low-level
operations, but the algorithm was always specified.
declarative programming language with exact specifications.
General Purpose Declarative Programming Language
The research objective is to implement a general purpose declarative language, or
extension to an existing language, so that the programmer only has to specify what needs
to be accomplished: the system would invent the algorithm and steps required.
For example, suppose our C library lacked cube roots. A declarative extension, with new
keyword such, might read:
float CubeRoot ( float x )
{ Return ( y such y*y*y == x ) }
Here CubeRoot has an input (it takes a float x) and an output (it returns float y, such
that y3 = x) specification. The declarative language pre-processor would
invent the function body (using selection, iteration, and recursion as needed), so that it
does indeed return 3 √x. The code necessary to make an input→output
transformation (here numbers to their cube root) is referred to as an invented solution.
Conceivably, invented solutions could themselves be parameterised (perhaps using
meta-programming or template techniques), and present a family of solutions.
The fundamental AI technique has been de-risked: for a specific case, it has
successfully invented an algorithm to solve a real-world problem. (In fact, it provided
an optimum solution where I thought no solution existed at all.) It used general-purpose
inventive techniques, without special-case knowledge. An early enhancement would be the
ability to automatically factor a problem into sub-problems, each also described by the
same input→output specification.
It may be desirable to include target resource limits that, ideally, a solution should
have. (Eg maximum RAM use or running time; or minimum "accuracy".) The system could know
that until these targets were hit, it should not regard the function as sufficiently
solved.
A subroutine library would be used to avoid re-solving the same sub-problem for each
cycle of development, or save CPU resources across a team. As each subroutine must have
an input→output description, it would be easy to index, retrieve and re-use solutions.
This library could store:-
System invented solutions to programmer specified problems;
System invented solutions to system factored sub-problems;
Programmer solutions to fully input→output specified problems (ie sub-routines);
Fully specified sub-routines embodying known solutions, (eg trig or Fourier
transforms)
The main disadvantages are:
the invention phase is likely to be extremely CPU intensive, though not on the
scale of training LLMs;
how the invented code works will often be unintelligible although
what it does will be clear - it will accomplish the specification.
An out-of-scope extension might be to employ natural language generation techniques to
describe the invented algorithm.
Potential advantages include:
substantial programmer productivity gain;
increased reliability through generating verification and test code from the
input→output specifications;
the ability to mix with “hand-written” code; and
some intractable problems may prove to have solutions.
