{"id":7347,"date":"2019-05-02T08:00:11","date_gmt":"2019-05-02T15:00:11","guid":{"rendered":"http:\/\/softwareengineeringdaily.com\/?p=7347"},"modified":"2019-05-01T20:38:56","modified_gmt":"2019-05-02T03:38:56","slug":"online-learning","status":"publish","type":"post","link":"https:\/\/softwareengineeringdaily.com\/2019\/05\/02\/online-learning\/","title":{"rendered":"Online Learning"},"content":{"rendered":"

Machine learning and AI are concepts that are ever-growing and are continually being integrated into the workloads of more enterprises. If you are in the software field, you probably have at least an idea on how machine learning actually happens – a certain model is trained on the training data, perhaps with cross-validation. The trained model is then measured in effectiveness using the test data. If the model is sufficiently successful on the metric of your choice, then it gets deployed to be used for any new, unseen data for future predictions.<\/span><\/p>\n

This process, well-known to many, is actually offline or batch machine learning. In offline learning, certain assumptions are made: it is assumed that there exist relevant and reasonably-sized data that the model can be trained on, and more importantly, the future data on which the predictions will be performed are sampled from the same distribution as the training data. This is also known as the <\/span>independent and identically distributed random variables (i.i.d.)<\/span><\/a> assumption. <\/span><\/p>\n

\"\"<\/p>\n

Online machine learning versus batch learning. (a) Batch machine learning workflow; (b) Online machine learning workflow, from <\/span>Zheng et.al. Effective Information Extraction Framework for Heterogeneous Clinical Reports Using Online Machine Learning and Controlled Vocabularies<\/span><\/a><\/p>\n

These assumptions can be valid for practical settings. When analyzing the works of an author, it might be assumed that the author has a consistent writing style, and all their work inherits, in some way, this style. When building a machine learning model to predict whether a given text can be from this particular author, it can be valid to hold to the i.i.d. assumption, and train a large model on the works of the author. <\/span><\/p>\n

However, i.i.d. assumption does not always hold, and is not always practical. A classic example is adversarial spam filtering.<\/span><\/p>\n

 <\/p>\n

Spam Email Filtering<\/b><\/h1>\n

One basic example where the i.i.d. assumption doesn\u2019t hold comes in the form of an adversarially adaptive setting: spam filtering. This is a binary classification problem, where each data point, e.g. an email, is classified either as \u2018spam\u2019 or \u2018not spam\u2019. <\/span><\/p>\n

\"\"<\/p>\n

Consider the following scenario: we have a machine learning model for spam filtering, that has been trained on a large data set, and achieves 95% accuracy. Since this accuracy is sufficiently well, the model is deployed to production. However, in time, the model\u2019s strength goes down, and it lets spam email get through. Why? <\/span><\/p>\n

Because people that generate spam email are playing it smart – when they notice that their emails are not being opened, or the ads are not being clicked, they tweak their content to fool the model we deployed. The future data is no longer sampled from the same distribution that we trained the model on!<\/span><\/p>\n

This is an adversarial setting – the spammers are adversaries to our model. What is the solution? For this simple example, the solution can also be simple: the developer can manually re-train the model again as new data come in, and with a clever choice of loss function and success metric, they can put greater weight on the most recent data. Another option is to automate this process, and have the model re-trained in defined intervals, e.g. once every week, and then automatically deployed. This way, the overall performance stays relatively stable. <\/span><\/p>\n

For the straightforward example of email spam filtering, offline learning with automated re-training can work well. The worst that can happen is, the adversaries play it cunningly and right after a model deployment, they start sending smarter spam. Some spam emails manage to get through the model, landing on the user\u2019s inbox, until the next training phase where these new spam emails will be known to be spam. <\/span><\/p>\n

Illustration of exploratory attacks on a machine learning based spam filtering system,<\/span><\/p>\n

from <\/span>Tegjyot Singh Sethi and Mehmed Kantardzic, A Dynamic-Adversarial Approach to the Security of Machine Learning<\/span><\/a><\/p>\n

\"\"<\/p>\n

A natural follow-up is to circumvent this problem is shortening the time intervals between training the model. The shorter the interval, the less tweaking time the adversaries will have. When taken to the extreme, this approach converges to real-time updates to the model, as time intervals approach virtually 0. This practice then becomes known as \u2018online learning\u2019.<\/span><\/p>\n

 <\/p>\n

Online Learning<\/b><\/h1>\n

Formally, <\/span>online learning is the process of answering <\/span>a sequence of questions<\/b> given knowledge of the correct answer to previous questions<\/span><\/a>. Opposed to the classic offline machine learning, where all data points are available at training time, in online learning, data points come in sequences – we are only exposed to one data point at a time. <\/span><\/p>\n

Online learning is performed in consecutive rounds denoted by <\/span>t, <\/span>where in each round the learner receives a question <\/span>x<\/span>t<\/span>,<\/span> and subsequently provides an answer <\/span>p<\/span>t<\/span>.<\/span> When the true answer is received, the learner suffers a loss, and the learner can update the model accordingly. <\/span><\/p>\n

\"\"<\/p>\n

The notion of risk and loss is inherently different between offline and online learning. In offline learning, the risk is the expected loss over all the data points, since the data points are not sequential. In the online setting, data are seen as a sequence: the risk of the model <\/span>changes<\/b> with each new observation. At any point in time, <\/span>T<\/span><\/i>, the risk is <\/span><\/p>\n

\"\"<\/p>\n

where <\/span>p<\/span>t <\/span>and <\/span>y<\/span>t <\/span>are as described above. The overall goal is to minimize <\/span>regret<\/b>, a different notion of cumulative loss. Regret is defined as<\/span><\/p>\n

\"\"<\/p>\n

What this means is that the regret of an algorithm is the difference between the risk of the given algorithm and the risk of the algorithm that can achieve the minimum risk in hindsight. This formulation makes intuitive sense: the data is coming in discrete points, so we cannot make a holistic guess. Our model can be judged by the past predictions, and we can only <\/span>regret<\/b> the actions that we have not taken. <\/span><\/p>\n

An intuitive example to this online learning setting is very well-known in the neural network area: stochastic or online gradient descent. Typically used during the optimization step of neural networks, gradient descent is an algorithm that aims to minimize a target function <\/span>by iteratively moving in the direction of steepest descent as defined by the negative of the gradient<\/span><\/a>. <\/span><\/p>\n

Online gradient descent executes gradient descent by only taking a single sample, i.e. a batch size of 1, for each iteration. After obtaining a loss for this single example, the parameters, i.e. weights in neural networks, are updated according to the negative gradient of this loss. The similarity between the general pseudocode description of online learning and the algorithm for online gradient descent is apparent. <\/span><\/p>\n

\"\"<\/p>\n

[From the slides of Wojciech Kotlowski, Institute of Computing Science, Pozna<\/span>\u0144<\/span> University of Technology] <\/span><\/p>\n

<\/h1>\n

When is online learning useful? <\/b><\/h1>\n

One of the main settings that require an online learning solution are when the data are too big to train a static offline model efficiently. Out-of-core training, where the data set does not wholly fit into the main memory, is not feasible in environments with limited resources. Then, the data can be treated as an incoming stream, and processed in an online fashion. Treating each data point as a new observation and iteratively tuning the learning model makes this possible. As can be seen from this setting, online learning does not always have to be \u2018online\u2019 in the common sense. Most of the current offline learning algorithms actually deploy online learning in the optimization step, e.g. online gradient descent.<\/span><\/p>\n

Another setting where online learning is useful is when the underlying distribution of the data drifts over time. This occurrence is also known as <\/span>concept drift<\/b>. The changes in the distribution of the data can be attributed to numerous causes, one of the most prominent case being the adversarial setting like spam filtering. Other causes can be <\/span>changes in population, changes in personal interests, or the complex nature of the environment<\/span><\/a>. <\/span><\/p>\n

\"\"<\/p>\n

Illustration of the four structural types of concept drift, <\/span><\/p>\n

from <\/span>Indr\u0117 \u017dliobait\u0117, Learning under Concept Drift: an Overview<\/span><\/a><\/p>\n

Concept drift happens in various real-life applications. <\/span>\u017dliobait\u0117<\/span><\/a> divides them into four categories: monitoring control, personal assistance, decision making, and artificial intelligence. Some examples include fraudulent transaction detection in banking, customer profiling and dividing customers into non-stationary segments, forecasting in finance, and stock price prediction.<\/span><\/p>\n

 <\/p>\n

In Practice<\/b><\/h1>\n

Algorithms for online machine learning on streaming data is an active research area. Machine learning is a very broad area, and online machine learning can be integrated into existing paradigms, such as building <\/span>online deep learning<\/span><\/a> models. <\/span><\/p>\n

Currently, there are numerous open source approaches to building machine learning models for online settings:<\/span><\/p>\n