In the last few decades of the past century and the first decade of the current century, neural networks were considered as just another tool in the machine learning and artificial intelligence toolbox. However in the past decade a transition was made and the deep learning revolution began. Many attribute the transition point to the AlexNet 2012 ImageNet victory and equally important is the availability of quality large datasets. To get a feel for this, read this report about ImageNet and the ImageNet challenge
Many of these advances are driven by the efficient design of systems. This includes the ability to collect, store, and distribute huge datasets; as well as the ability to compute efficiently using GPUs. That is, the mathematics (and mathematical engineering methods) have been around for a long time, but the advancements have to do more with efficient system design than with innovative mathematics. Nevertheless, the field of deep learning integrates mathematical thinking with engineering in a very tightly coupled manner. That is, in many ways, the deep learning professional is a mathematical engineer dealing primarily with algorithms, models, functions, matrices, tensors, convolutions, and similar objects - often through the use of a programming language such as Python, R, or Julia.
Now in 2021, a decade into the deep learning revolution, the training and use of deep neural networks has become extremely accessible. The popular deep learning frameworks Tensor Flow and PyTorch have become the standard (as for for now) and one can find hundreds of tutorials and examples for how to use these frameworks for a variety of applications. There are also multiple online platforms aimed at providing (often free) deep learning education for non-mathematicians such as fast.ai and deeplearning.ai. Our focus with this course was slightly different. We presented an overview of deep learning aimed at audiences that feel comfortable with mathematical language.
Deep learning is moving fast so by “recent” we mean trends from the past 0-3 years. Much work is now focused on further advancements in natural language processing and natural language understanding. For example the GPT-3 language model from OpenAI was introduced this year:
Other work is in the domain of scientific machine learning (see also this page). For example recently, DeepMind announced a breakthrough with respect to protein folding. See this:
Other recent trends include further advancements using generative adversarial networks and similarly with reinforcement learning.
It is hard to predict where the field of deep learning will go in the coming years. Many await the arrival of Artificial General Intelligence (AGI) where a machine will have capacity to understand and learn any intellectual task that a human being can. However, with the current mathematical ideas and technologies, it may be safe to say that AGI is still not attainable.
Much more realistically we should consider the integration of deep learning in almost all aspects of industry and technology. This is happening today. The productive fields of medicine, transportation, agriculture, and education are continuously enhanced with deep learning innovations. Similarly with more debatable domains such as defense (offense) and social networks.
See for example the Waymo self-driving taxi,
Further see robots (often using deep learning) in agriculture:
Mathematics is a broad field that has always formed a foundation for science and technology. The “Mathematical Engineering of Deep Learning” refers to the application of mathematics in deep learning. Deep learning professionals and researchers have made use of the mathematics from the go. However, mathematics researchers and mathematics educators are more often than not, unaware of the details of deep learning. This is not surprising because deep learning is an emerging and rapidly evolving field while mathematics (and mathematics education) often moves at a slower pace.
One of our key purposes in this course was to introduce mathematical audiences with the mathematical engineering details of deep learning. Understanding of deep learning is not just important for machine learning engineers and data scientists. It is also important for applied mathematics modellers, statisticians, and even pure mathematicians. In fact, we believe that education of the general mathematical community about the concepts of deep learning can go a long way not just for the sake of deep learning, but also for the purpuse of keeping mathematics relevant and exciting in all other domains.
For example, we hope that in the future, even high school (secondary school) teachers teaching students about linear equations, parabolas, or basic calculus will posses such general knowledge so as to demystify “artificial intelligence” and be able to convey to their students how simple mathematical ideas, coupled with efficient computation, can go very far.
Page built: 2021-03-04 using R version 4.0.3 (2020-10-10)