Lynx Analytics logo

Neural Network That
Learns From A Huge Graph

Presented at Spark Summit East, 2017.

Lynx Analytics develops a big graph analysis engine on top of Apache Spark. One of our recent developments is a recurrent neural network library that learns from the structure of the graph in order to predict missing features of vertices.

A real-life use case is demographic estimation where the task is to predict the age of different customers of a telco by exploring their connections to other people, the age of those people and other classical features like internet or phone usage patterns.

One of the main challenges we faced was to develop a training process for our purposes. The usual way of training a supervised learning algorithm considers each vertex as an independent prediction problem. But due to the use of connections between the vertices in our algorithm we cannot treat vertices independently. On the other hand, if you consider the whole graph as one problem, then you do not have any separate training data at all. In this talk we will show some tricks that we used in order to perform the prediction and the training process on the same graph.

The other main challenge is to handle graphs so big that they do not fit into the memory of a single machine and perform really resource-intensive computations on them. To tackle this problem it is necessary to store and make computations on the graph distributedly. The difficulty of this is that we cannot just simply cut the graph into smaller pieces since we need to propagate data via the edges for the training process.

In the talk we will show core algorithmic ideas to tackle the above-mentioned problems and present some experimental results.