Primjene teorije informacija na probleme dubokog učenja

master's thesis

• Title (english): Applications of information theory on deep learning problems
• Author: Jakob Tot
• Mentor: Matej Mihelčić (mentor)
• Committee member: Matej Mihelčić (predsjednik povjerenstva)
• Committee member: Hrvoje Planinić (član povjerenstva)
• Committee member: Miljenko Huzak (član povjerenstva)
• Committee member: Marko Horvat (član povjerenstva)
• Granter: University of Zagreb; Faculty of Science; (Department of Mathematics); Zagreb
• Defense date and country: 2024-03-06, Croatia
• Scientific / art field, discipline and subdiscipline: NATURAL SCIENCES; Mathematics

Abstract

U ovom radu dajemo pregled osnovnih pojmova i koncepata teorije informacija i dubokog učenja. Rad započinjemo definiranjem najvažnijih mjera teorije informacija u diskretnom slučaju. Nakon toga, obrađujemo dva najvažnija teorema čitavog područja: teorem o kodiranju izvora i teorem o kodiranju kanala. Nakon toga uvodimo mjere u neprekidnom slučaju i pokazujemo poveznicu s diskretnim slučajem. U drugom dijelu rada bavimo se dubokim učenjem. Dajemo kratki povijesni pregled počevši od prvih neuronskih modela. Ilustriramo ekvivalentnost nekih klasičnih modela strojnog učenja s modelima s jednim neuronom. Posebnu pažnju posvećujemo unaprijednim i konvolucijskim neuronskim mrežama te prolazimo kroz algoritam unazadne propagacije. Za kraj, uspoređujemo točnost različitih modela na problemima klasifikacije slika. Tu naglašavamo da kao funkciju gubitka neuronskih modela koristimo unakrsnu entropiju, veličinu iz teorije informacija

Abstract (english)

In this paper, we provide an overview of the basic concepts and principles of information theory and deep learning. We begin by defining the most important measures of information theory in the discrete case. After that, we discuss two of the most important theorems in the entire field: the source coding theorem and the channel coding theorem. We then introduce measures in the continuous case and demonstrate the connection with the discrete case. In the second part of the paper, we focus on deep learning. We provide a brief historical overview starting from the earliest neural models. We illustrate the equivalence of some classical machine learning models with single-neuron models. We pay special attention to feedforward and convolutional neural networks and go through the backpropagation algorithm. Finally, we compare the accuracy of different models on image classification problems. Here we emphasize that the loss function used by the neural models is crossentropy, a quantity from information theory.

• Keywords: teorem o kodiranju izvora; teorem o kodiranju kanala; neuronski modeli; modeli strojnog učenja
• Language: croatian
• URN:NBN: urn:nbn:hr:217:644603
• Study programme: Title: Mathematical Statistics; Study programme type: university; Study level: graduate; Academic / professional title: sveučilišni magistar matematike (sveučilišni magistar matematike)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2024-05-31 10:26:48