PowerPoint Assistant: Augmenting End-User Software through Natural Language Interaction

This links to a set of notes that I first wrote up in 2003 or 2004, somewhere in there, but didn't post to the web  until 2015. It calls for a natural language interface with end-user software that could be easily extended and modified through natural language. The abstract talks of hand-coding a basic language capability. I wrote these notes before machine learning had become so successful. These days hand-coding might be dispensed with – notice that I said "might." In any event, a young company called Adept has recently started up with something like that in mind:

True general intelligence requires models that can not only read and write, but act in a way that is helpful to users. That’s why we’re starting Adept: we’re training a neural network to use every software tool and API in the world, building on the vast amount of existing capabilities that people have already created.

In practice, we’re building a general system that helps people get things done in front of their computer: a universal collaborator for every knowledge worker. Think of it as an overlay within your computer that works hand-in-hand with you, using the same tools that you do.

* * * * *

Another working paper at Academia.edu: https://www.academia.edu/14329022/PowerPoint_Assistant_Augmenting_End_User_Software_through_Natural_Language_Interaction

Abstract, contents, and introduction below, as usual.

* * * * *

Abstract: This document sketches a natural language interface for end user software, such as PowerPoint. Such programs are basically worlds that exist entirely within a computer. Thus the interface is dealing with a world constructed with a finite number of primitive elements. You hand-code a basic language capability into the system, then give it the ability to ‘learn’ from its interactions with the user, and you have your basic PPA.

C O N T E N T S

Introduction: Powerpoint Assistant, 12 Years Later 1
Metagramming Revisited 3
PowerPoint Assistant 5
Plausibiity 5
PPA In Action 6
The User Community 12
Generalization: A new Paradigm for Computing 13
Appendix: Time Line Calibrated against Space Exploration 15

Introduction: Powerpoint Assistant, 12 Years Later

I drafted this document over a decade ago, after I’d been through an intense period in which I re-visited work I’d done in the mid-to-late 1970s and early 1980s and reconstructed it in new terms derived jointly from Sydney Lamb’s stratificational linguistics and Walter Freeman’s neurodynamics. The point of that work was to provide a framework – I called it “attractor nets” – that would accommodate both the propositional operations of ‘good old artificial intelligence’ (aka GOAFI) and the more recent statistical style. Whether or not or just how attractor nets would do that, well, I don’t really know.

But I was excited about it and wondered what the practical benefit might be. So I ran up a sketch of a natural language front-end for Microsoft PowerPoint, a program I use quite a bit. That sketch took the form of a set of hypothetical interactions between a use, named Jasmine, and the PowerPoint Assistant (PPA) along with some discussion.

The important point is that software programs like PowerPoint are basically closed worlds that exist entirely within a computer. Your ‘intelligent’ system is dealing with a world constructed with a finite number of primitive elements. So you hand-code a basic language capability into the system, then give it the ability to ‘learn’ from its interactions with the user, and you have your basic PPA.

That’s what’s in this sketch, along with some remarks about networked PPAs sharing capabilities with one another. And that’s as far as I took matters. That is to say, that is all I had the capability to do.

For what it’s worth, I showed the document to Syd Lamb in 2003 or 2004 and he agreed with me that something like that should be possible. We were stumped as to just why no one had done it. Perhaps it simply hadn’t occurred to anyone with the means to do the work. Attention was focused elsewhere.

Since then a lot has changed. IBM’s Watson won at Jeopardy and more importantly is being rolled out in commercial use. Siri ‘chats’ with you on your iPhone.

And some things haven’t changed. Still no PPA, nor a Photoshop Assistant either. Is it really that hard to do? Are the Big Boys and Girls just distracted by other things? It’s not as though programs like PowerPoint and Photoshop serve niche markets too small to support the recoupment of development costs.

Am I missing something, or are they?

So I’m putting this document out there on the web. Maybe someone will see it and do something about it. Gimme’ a call when you’re ready.