Tag Archives: Intelligent Personal Assistant

Experimenting with Intelligent Personal Assistant Platform

Traditionally the term Intelligent Personal Assistant is associated with Siri, Google Now or Cortana and voice based communication. For me, this term means something different – extendable open modular platform doing useful things for the user with user privacy at the centre. I have been experimenting with various ways to build something like this for the last few years in my spare time. I consider it as a research hobby, although I would love to see some Personal Assistant Platforms in existence and usage. In this post I would like share some ideas and observations.

My approach is more close to the approach used in platforms which became quite popular in resent years because of the ChatOps movement. The idea is to use chat-based user interface and “bots” which listen for patterns and commands in chat rooms and react on messages using coded “skills”. ChatOps solutions typically concentrate on coordination of human activities and task automation with bots. The same ideas (and software) can be used for building Personal Assistants. I have been interested in exploring core assistant architecture so I decided (after trying few different bot frameworks) to use JASON [1]. JASON is not a typical bot platform, it is a framework for building multi-agent systems. JASON is written in Java but agent behaviours and declarative knowledge are coded in AgentSpeak [2] – high level language with some similarity to Prolog.

Current architecture is presented below.

Slide02

I use Slack as a chat platform. Slack provides user apps for various devices, it allows messages with rich content and has APIs for bot integration. There are couple variations of API, but the most attractive from the Personal Assistant perspective is the WebSocket based Realtime API. With this API, bot (assistant) can run in a cloud, on an appliance at your home, or just on your desktop computer (if you still have one). Slack group capabilities are also helpful, it is possible to communicate with various versions and types of the assistants. I use node.js-based gateway to decouple Slack and JASON based infrastructure. I also use Redis as a simple messaging medium. There is nothing special about node.js and Redis in this architecture except that both are easy to work with for implementing “plumbing”. JASON AgentSpeak does not have capabilities to talk to external services directly but it can be extended through Java components. I coded basic integration with Redis pubsub inside of my JASON solution, and for advanced scenarios (when I need services with lots of plumbing) I use node.js or other service frameworks.

Personal Assistant is implemented as a multi-agent system with additional components in node.js (and other frameworks as needed) communicating through Redis. At the centre of the multi-agent system is the Assistant Agent, User Agent and various Expert agents. User Agent is responsible for communicating with the user, Assistant agent plays the role of the coordinator and Expert agents implement specific skills. Expert agents can delegate low level “plumbing” details to services. The system is quite concurrent and runs well on multicore computers. JASON allows also to run multi-agent system on several computers but this is outside of my current experiments. Agents can create other agents, can use direct message based communication (sync and async) or pubsub based message broadcasting. Agents have local storage implemented as prolog-like associative term memory, logical rules and scripts. AgentSpeak follows BDI (Belief–Desire–Intention) software model[3] which allows to code quite sophisticated behaviours in a compact way. Prior to JASON I tried to implement the same agents with traditional software stacks such as node.js, Akka Actors. I also looked quickly at Elixir, Azure Actors. It is all good, it just takes more time and lines of code to get to the essence of interesting behaviours. I actually tried JASON earlier but made a mistake. I started with implementing Personal Assistant as a “very smart” singleton agent. Bad idea! Currently JASON-based assistant is a multi-agent system with many very specialized “experts”.

Let’s look at the “Greeting” expert agent, for example. This agent can send variations of the “Hi” message to the user and will wait for greeting response (for some time). The same agent listens for variations of the “Hi” message from the user. Greeting agent implements bi-directional communication with the user with mixed initiative. It means that the agent can initiate greeting and wait for the user response or it can respond to the user greeting. In addition, greeting agent reacts on user on-line presence change, remembers last greeting exchange. After successful greeting exchange it broadcasts “successful greeting” message which can initiate additional micro conversations.

Reactivity, autonomy, and expertise are the primary properties of the Assistant multi-agent system. Various expert agents run in parallel, they have goals, react on events, develop and execute plans, notify other agents about events, compete for the user attention. Although many agents run simultaneously, there is a concept of the “focus agent” that drives communication with the user. Even without requests from the user, Assistant is active, it may generate notifications to the user, ask questions, suggest to initiate or continue micro conversations.

Multi-agent platform JASON directly supports implementation of manually encoded reactive and goal oriented behaviours (quite sophisticated). It can be also extended with more advanced approaches such as machine learning, simulation of emotions, ethical decision making, self modelling. My current interest is mostly in experimenting with various expert agents, coordination of activities and user experience, but I am looking forward to other topics.

  1. JASON: https://en.wikipedia.org/wiki/Jason_(multi-agent_systems_development_platform)
  2. AgentSpeak: https://en.wikipedia.org/wiki/AgentSpeak
  3. Belief–Desire–Intention software model: https://en.wikipedia.org/wiki/Belief–desire–intention_software_model

Home Automation and Intelligent Personal Assistants

Home automation became a hot topic again. We have more sensors, automation devices, more connectivity options, and better managing apps.

Legacy approach to home automation typically includes using several mobile controlling apps each dedicated to a specific device. This approach is not sustainable with the increasing number of devices. It does not provide optimal user experience: we cannot see the “big picture” – status of a “home” at any moment of time. We are forced to jump between various controlling apps with different user interfaces. There is no simple way to see correlation between device performance. Optimization, predictive modelling are limited by device scope.

More advanced approach includes the idea of a “home automation hub”. In this case, user can have only one app which controls all home automation devices. Hub unifies various physical connectivity protocols, creates integrated “big picture”, allows advanced optimization. Many consumers will be probably quite happy with this level of automation and integration (at least for now).

As the natural next step, I see potential for integration of “home automation hub” into Intelligent Personal Assistants. “Home” is an important concern for many, but not the only one. The same type of consolidation is going on in other areas. We may soon deal with “personal health hub”, “personal finance hub”, “personal investment hub”, “car hub”, “transportation hub”, “travel hub” etc. Each of these hubs will be active, monitoring current situation, identifying possible/recommended actions, “fighting” for our attention. Some mediation will be required and this is one of the main roles that can be played by Intelligent Personal Assistants: optimization of user experience based on integrated view of highly automated world.

Google Glass and Intelligent Personal Assistants

I have been investigating Google Glass Mirror API and this investigation has generated some thoughts about Google Glass and Personal Intelligent Assistants that I would like to discuss.

I am quite enthusiastic about Google Glass, mostly because it creates a framework for context-aware real-time user centric services. I am very interested in the Mirror API service created and maintained by Google and capabilities for developers to deliver services based on the Glass platform.

The Mirror API allows developers to manipulate Glass timeline, react on changes in timeline and other events, it allows also controlled information sharing between apps/services. In principle, it is not that different from the App centric model that we have already with smartphones. We just do not need to check our smartphones from time to time, important information is always available. But with Glass, there is a big difference from my perspective: various services become integrated in one unified timeline with unified interface and user experience. Glass implements basic information and service integration – literally “on the glass” . I also consider Glass as a new generation of notification centre with information delivered proactively to Glass owners

Mirror API service

How can we incorporate Intelligent Personal Assistants into this picture? Personal Assistant can be viewed as a specialized service with one of its interfaces being Glass-aware.

Mirror API and Intelligent Personal Assistant

Intelligent Personal Assistant provides mediated communication between its owner and various service providers. It manages personal data/event clouds and provides integrated view on “things” and events important for its user. Personal Assistants will have multiple user interfaces including smart glasses, watches, phones, tablets, car dashboards, TVs, etc.

There are currently examples of closed services called “Personal Assistants” tightly coupled with specific vendor solutions, but I am more interested in an open extensible platform with various components and options for deployment. I am looking forward to something like “WordPress for Personal Assistants”. My understanding of Intelligent Personal Assistants is very close to classic FIPA Personal Assistants [1] and the idea of “Personal Data Locker” introduced by David Siegel in his book “Pull” [2] and his vision video [3]

References:

  1. FIPA Personal Assistant Specification
  2. Pull: The Power of the Semantic Web to Transform Your Business by David Siegel
  3. Personal Data Locker vision video by David Siegel