Analiza velikih količina podataka korištenjem Apache Spark platforme

master's thesis

• Title (english): Analysis of large amounts of data using the Apache Spark platform
• Author: Vanja Nevajdić
• Mentor: Ognjen Orel (mentor)
• Committee member: Ognjen Orel (predsjednik povjerenstva)
• Committee member: Robert Manger (član povjerenstva)
• Committee member: Marko Tadić (član povjerenstva)
• Committee member: Filip Najman (član povjerenstva)
• Granter: University of Zagreb; Faculty of Science; (Department of Mathematics); Zagreb
• Defense date and country: 2021-09-23, Croatia
• Scientific / art field, discipline and subdiscipline: NATURAL SCIENCES; Mathematics

Abstract

U današnjem digitalnom svijetu potrebno je analizirati ogromne količine podataka. Da bi to bilo moguće, najprije su se razvili distribuirani računalni sustavi i paralelno procesiranje podataka, a s vremenom i mnoštvo tehnologija za obradu i analizu podataka, jedna od kojih je Apache Spark. Apache Spark je platforma osmišljena za distribuirano procesiranje velike količine podataka. Ona se odlikuje svojom brzinom, širinom primjene i jednostavnosti korištenja. Brzina izvođenja ostvaruje se čuvanjem podataka u memoriji i internom optimizacijom koda. U Sparku se može raditi sa strukturiranim i graf podacima te tokovima podataka, a mogu se provoditi razne analize, uključujući strojno učenje. U praktičnom dijelu radu napravljena je analiza javno dostupnog skupa podataka Yelp platforme koristeći Apache Spark. Radi se o podacima o poduzećima, njihovim korisnicima i recenzijama koje oni ostavljaju. Analizirala se kvaliteta tih podataka tako što su se proučavale vremenske razlike između dvije uzastopne recenzije istog korisnika. Fokus analize bila je detekcija recenzija koje su napisane od strane robota.

Abstract (english)

In today’s digital world there is a need for analysing vast amounts of data. To achieve that, distributed computer systems and parallel processing were invented. In the following years, a number of technologies for processing and analysing data were created, with Apache Spark being one of them. Apache Spark is a platform designed for large-scale distributed data processing. Its main characteristics are speed, broad application and ease of use. Its speed is achieved with in-memory storage and code optimisation. Spark can be used on structured, graph and streaming data to perform different types of analyses, including machine learning. As part of this thesis, data quality analysis was performed on Yelp’s open dataset using Apache Spark. The data consisted of information about businesses, their customers and the reviews that they write. The focus of the analysis was on bot detection, which was performed by exploring the time between two consecutive reviews of a customer.

• Keywords: distribuirano procesiranje podataka; strojno učenje; Yelp platforma
• Keywords (english): distributed data processing; machine learning; Yelp’s open dataset
• Language: croatian
• URN:NBN: urn:nbn:hr:217:597888
• Study programme: Title: Mathematical Statistics; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra matematike (magistar/magistra matematike)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2021-11-02 13:59:50