Julia for optimization and learning

Semestr: Winter

Range: 1P+3C

Completion:

Credits: 4

Programme type: Bachelor

Study form: Fulltime

Course language: Czech

Time table at FEE

Summary:

Julia programming language is increasingly known by the community for its suitability in the field of numerical calculations. The course consists of two parts. The first part presents the basics of Julia. The second part introduces mathematical optimization and its application in machine learning, statistics and optimal control of differential equations. While the first part shows the individual concepts of Julia, the second part combines them into longer logical sections of code. We explain each application theoretically. Students are encouraged both to write simple functions by themselves and compare them with already existing packages. The course ends with a final project. Students can either choose a topic connected to their theses or join a Kaggle competition with real data. This course is also part of the inter-university programme prg.ai Minor. It pools the best of AI education in Prague to provide students with a deeper and broader insight into the field of artificial intelligence. More information is available at https://prg.ai/minor.

Keywords:

Julia, programming, optimization, machine learning.

Course syllabus:

8. Introduction to optimization. Optimality conditions for optimization with and without constraints.
9. Introduction to linear regression and classification problems. Closed-form solutions and iterative methods.
10. Introduction to neural networks. Types of neural networks. Individual layers. Overfitting.
11. Convolutional layers, neural network structure, stochastic gradient descent.
12. Selected parts of statistics. Regularized linear regression. Spectral decomposition. Monte Carlo sampling. Hypothesis testing. Generalized linear models.
13. Introduction to ordinary differential equations. Relation with spectral decomposition. Optimal control as a combination of ODEs and optimization.

Seminar syllabus:

1. Introduction to Julia, advantages and disadvantages compared to Matlab, Python and R. Declarations of variables, basic numerical types, numerical and logical operators. Text strings.
2. Vectors, matrices and multidimensional arrays: basic properties, indexing, application of functions by elements. Other data structures and their properties: Tuple, NamedTuple, Dict.
3. If-elseif-else statements and ternary operator. Simple and nested for/while loops. Basic use of iterators and generators. Overview of the most used iterators: zip, eachcol, eachrow, enumerate. Scopes in loops.
4. Function declarations, mandatory, positional and keyword arguments. Functions as a set of methods. Calling methods according to the number of arguments and their type (multiple-dispatch). Local and global workspace.
5. Overview of the most used standard libraries: LinearAgebra, Statistics, Random. Useful packages: Plots.jl for creating graphs, DataFrames.jl for working with tabular data. Packages for interaction with other languages: PyCall, RCall.
6. Compound types and creating a logical hierarchy using abstract types. Parametric types. Internal and external constructors. Definition of specialized methods for complex and abstract types.
7. Organization of code into scripts and modules. Function import and extending existing code. Organization of installed packages into separate environments. Julia package creation.
8. Gradient visualization. Gradient descent method and step length selection. Projected gradients.
9. Data preparation and visualization. Linear and logistic regression training. Predictions.
10. Data preparation. Implementation of a simple neural network, including gradient calculation and training.
11. Creating neural networks using the Flux.jl package. Working on GPU.
12. Implementation of LASSO and ridge regression according to iterative methods for optimization. Analysis of the required number of samples for good approximation quality. The Distributions.jl, HypothesisTests.jl, and GLM.jl packages.
13. Hand-written implementation of the wave equation. Package ODE.jl.

Literature:

1. Online skripts https://bit.ly/JuliaML
2. Julia documentation. https://docs.julialang.org/en/v1/manual/documentation/index.html
3. Kochenderfer, M. J. and Wheeler, T. A. Algorithms for optimization. MIT Press, 2019. https://algorithmsbook.com/optimization/
4. Lauwens, B. and Downey, A. B. Think Julia: how to think like a computer scientist. O'Reilly Media, 2019. https://benlauwens.github.io/ThinkJulia.jl/latest/book.html
5. Goodfellow, I. and Bengio, Y. and Courville, A. Deep Learning. MIT Press, 2016. https://www.deeplearningbook.org/

Examiners:

Lecturers:

Instructors: