Machine Learning Trends 2018
We figure out how technologies and approaches to work in machine learning have changed in the last 5 years, using the example of the study of Andrej Karpathy.
The head of the machine learning department at Tesla, Andrej Karpathy, decided to find out how ML trends develop in recent years. To do this, he used the database of documents on machine learning for the past five years (about 28 thousand) and analyzed them. Andrew shared his findings with Medium
Features of the archive of documents
Let's first consider the distribution of the total number of downloaded documents for all categories (cs.AI, cs.LG, cs.CV, cs.CL, cs.NE, stat.ML) for a period of time. We get the following:
It can be seen that in March 2017 almost 2,000 documents were downloaded. The peaks that appear on the graph are probably due to conference dates associated with machine learning (NIPS / ICML, for example).
The total number of papers will serve as a denominator. We can see which part of the documents contains interesting keywords.
Fundamentals of Deep Learning
First, we define the most commonly used frameworks in Deep Learning. To do this, we find papers that mention the frameworks in any place of work (even if it's a list of used literature).
For March 2017 the following picture is obtained:
Thus, 10% of all documents downloaded during this period contain references to TensorFlow. Of course, not every article will mention the environment used, but assuming that such references occur in a document with a certain fixed probability, it turns out that about 40% of members of the machine learning community use TensorFlow.
And here's a picture of how some of the most popular frameworks evolved over time:
You can see that the popularity of Theano has slowed. Caffe quickly took off in 2014, but lost in recent years on the popularity of TensorFlow. Torch and PyTorch are slowly but surely gaining popularity.For beginners its always good practice to get through Journal of Machine Learning Research.
Top 10 Publications for Foundations and Trends in Machine Learning
1. Machine learning rules: best practices from Google developers
Top of the list of the best publications on machine learning is the publication of Google's developers, designed to help those who already have basic knowledge of machine learning, but there is not enough experience to understand the benefits of certain practices.
The idea of the manual is similar to the similar Google guide to the C ++ style and other popular guidelines for practical programming. The manual consists of 43 clearly described rules-recommendations.
2. Lessons learned from the reproduction of the materials of the article on training with reinforcement
Guided by the recommendation that the detailed reproduction of the results of scientific publications on machine learning is one of the most effective ways to improve the quality of their skills, the author details in detail about such experience gained during the development of the project, which is devoted to reinforcement learning.
3. On the way to the virtual stuntman
The problems of controlling traffic dynamics have recently become part of the standard tasks of training with reinforcement. Methods of in-depth training have shown here high efficiency for a wide range of problems.
However, characters, whose movement pattern was found as a result of training with reinforcement, observe undesirable artifacts: trembling, asymmetric gait, excessive limb mobility. The publication discusses the possibilities of teaching models more natural behavior.
4. Annotated Transformer
The idea of the Transformer architecture from the popular article " Attention is All You Need " last year has attracted the attention of many researchers in the field of computer linguistics.
In addition to improving the quality of translation, this approach provides a new architecture for many other natural language processing tasks. Although the source article is written in clear language, the very idea is rather difficult to implement correctly.
5. Differentiable plasticity: a new method of machine learning
In the middle of the selection of the best publications on machine learning for April 2018 was published the publication of the laboratory of artificial intelligence Uber about developments in the field of neural networks and an attempt to transfer the concept of plasticity of biological neural networks. The plasticity of real neurons lies in the ability to constantly interact between neurons throughout the entire existence of a neural network, which allows animals to adapt to changing conditions throughout their lives.
The article considers one of the possible approaches for such "learning" of artificial neural networks. Scientific publication of the laboratory Uber, which served as the source of the above post, can be read on arXiv .
6. Deep training to improve the quality of medical imaging
The difficulty in working with medical imaging archives is that, in their mass, they are represented by clinical assumptions. This means that when you want to extract an image (for example, a front x-ray of the chest), often instead you get a folder of many different images: with horizontal and vertical reflections, inverted pixel values, rotations at some angle.
7. Why do companies stop using RNN and LSTM?
Interest in recurrent neural networks and networks based on long short-term memory has increased dramatically in 2014, and over the next few years, these methods have become some of the best ways to solve problems of sequential learning and sequential (seq2seq) translation, which led to surprising results in improving recognition quality speech and the corresponding development of Siri, Cortana and Google's voice assistant, improving the quality of machine translation of documents, converting images into text, and so on.
But now, in 2018, the tools of successive models are no longer the best solutions, and more and more companies are moving to neural networks based on Attention based networks. The author explains the advantages of this approach and why many companies have moved away from the use of recurrent neural networks.
8. Keras and convolutional neural networks
This article presents the second publication of a three-part series on the construction of a complete complex classification of images based on in-depth training. The author, accompanying the story with examples of code, shows how to implement, train and evaluate the result of the convolutional neural network on its own data set. We recommend reading all three parts: the final one demonstrates how to deploy the pre-model Keras in a mobile application. For the sake of the fan as a task, the author realizes the childhood dream of creating the Pokedex - a device for recognizing the Pokémon.
9. How to implement the YOLO object detector (v3) from scratch in PyTorch
Detection of objects is an area that has greatly benefited from the latest developments in the field of in-depth training. As mentioned above, the best way to get acquainted with an algorithm, in particular an object detection algorithm, is to implement itself.
10. From viewing to listening: audio-visual separation of speech
Closes the top ten best machine learning publications for April 2018 post from Google's blog on artificial intelligence . It is well known that people even in a noisy environment know how to focus their attention on a particular person, mentally "drowning out" all other voices and sounds. However, the same task still represents a topic for machine learning. In the post we describe an audiovisual model that allows, in particular, to choose the person on the video in whose speech we want to focus, to distinguish their voices from the general noise.
