Performance Evaluation of Five Machine Learning Algorithms with Smote for Sentiment Classification of YouTube Comments on Purbaya’s Policy
Keywords:
Support Vector Machine , Sentiment Classification, Machine Learning Algorithms , YouTube Comments, Public Policy AnalysisAbstract
The evolution of public policy communication in the digital age has encouraged social media to become a deliberative space that openly reflects the dynamics of public perception and sentiment. This paper aims to conduct a comparative analysis of the performance of several machine learning algorithms in classifying the sentiment of YouTube user comments as a representation of public response to policy communication delivered by Minister Purbaya. The research was conducted using web scraping techniques with the YouTube Data API v3 on three public policy videos, with a total of 6,466 comments. The text data were processed using preprocessing, lexicon-based sentiment labeling, TF-IDF feature weighting, and class imbalance handling via the Synthetic Minority Over-sampling Technique (SMOTE). Five machine learning algorithms were applied, comprising SVM, KNN, RF, NB, and LR. Model performance was evaluated using accuracy, precision, recall, and F1-score metrics. The results showed that SVM produced the most optimal performance with an accuracy value of 0.78 and a consistent F1-score balance across all sentiment classes. These findings highlight the importance of algorithm suitability, feature representation, and data balancing strategies in developing reliable sentiment analysis models for evaluating data-driven public policy communication.
Downloads
References
[1] R. R. E. Nainggolan, “Analisis Penggunaan Website dan Media Sosial Pemerintah untuk Pelayanan Publik,” JTKP, vol. 6, no. 1, pp. 1–21, Jun. 2024, doi: 10.33701/jtkp.v6i1.4221.
[2] I. R. Agustina, “Implementasi Open Government Indonesia melalui Saluran Youtube Resmi Sekretariat Presiden,” JKP, vol. 7, no. 1, pp. 134–151, Mar. 2023, doi: 10.25139/jkp.v7i1.5674.
[3] M. F. A. Hakim, S. Subhan, and P. Listiaji, “Social Media Comments For Government Institution Video Classification Using Machine Learning,” jitk, vol. 10, no. 2, pp. 433–440, Nov. 2024, doi: 10.33480/jitk.v10i2.5187.
[4] A. S. W. Darmawan, A. Degaf, and N. F. Anggrisia, “Expressive Speech Acts and Public Sentiments in Netizen Responses to Political Posts on X,” JOLLT, vol. 13, no. 2, p. 960, Apr. 2025, doi: 10.33394/jollt.v13i2.13167.
[5] S. Kondolele, M. I. Sultan, M. Akbar, and B. A. Putra, “The nexus between public communication and policy implementation revisited: insights from the Population and Civil Registration Service of South Sulawesi, Indonesia,” Front. Commun., vol. 10, p. 1556747, May 2025, doi: 10.3389/fcomm.2025.1556747.
[6] E. S. Negara et al., “Sentiment Analysis Of Public Opinion On Presidential Advisory Appointments Using Naive Bayes Classification,” Jur. Ilmh. Ilm. Ter. Un. Ja, vol. 8, no. 2, pp. 452–466, Nov. 2024, doi: 10.22437/jiituj.v8i2.35254.
[7] R. H. Muhammadi, T. G. Laksana, and A. B. Arifa, “Combination of Support Vector Machine and Lexicon-Based Algorithm in Twitter Sentiment Analysis,” khif, vol. 8, no. 1, pp. 59–71, Mar. 2022, doi: 10.23917/khif.v8i1.15213.
[8] E. Erlin, Y. Desnelita, N. Nasution, L. Suryati, and F. Zoromi, “Dampak SMOTE terhadap Kinerja Random Forest Classifier berdasarkan Data Tidak seimbang,” matrik, vol. 21, no. 3, pp. 677–690, Jul. 2022, doi: 10.30812/matrik.v21i3.1726.
[9] L. L. Van Fc, M. K. Anam, M. B. Firdaus, Y. Yunefri, and N. A. Rahmi, “Enhancing Machine Learning Model Performance in Addressing Class Imbalance,” CogITo Smart Journal, vol. 10, no. 1, pp. 478–490, Jun. 2024, doi: 10.31154/cogito.v10i1.626.478-490.
[10] M. K. Anam, T. A. Fitri, A. Agustin, L. Lusiana, M. B. Firdaus, and A. T. Nurhuda, “Sentiment Analysis for Online Learning using The Lexicon-Based Method and The Support Vector Machine Algorithm,” Ilk. J. Ilm., vol. 15, no. 2, pp. 290–302, Aug. 2023, doi: 10.33096/ilkom.v15i2.1590.290-302.
[11] E. Susanti, M. Maimunah, and S. Nugroho, “Sentiment Analysis of YouTube Comments Using Machine Learning Models,” PIKSEL, vol. 13, no. 1, pp. 103–114, Mar. 2025, doi: 10.33558/piksel.v13i1.10743.
[12] A. F. Anjani, D. Anggraeni, and I. M. Tirta, “Implementasi Random Forest Menggunakan SMOTE untuk Analisis Sentimen Ulasan Aplikasi Sister for Students UNEJ,” TEKNOSI, vol. 9, no. 2, pp. 163–172, Sep. 2023, doi: 10.25077/TEKNOSI.v9i2.2023.163-172.
[13] M. A. Al-Shabi, “Evaluating the performance of the most important Lexicons used to Sentiment analysis and opinions Mining,” International Journal of Computer Science and Network Security, vol. 20, no. 1, 2020.
[14] R. Chandra and E. M. Sipayung, “Analisis Sentimen Ulasan Aplikasi Samsat Digital Nasional Menggunakan Algoritma Naive Bayes Classifier,” TEKNOSI, vol. 10, no. 3, pp. 156–164, Jan. 2025, doi: 10.25077/TEKNOSI.v10i3.2024.156-164.
[15] H. Leidiyana, T. Misriati, and R. Aryanti, “Klasifikasi Sentimen Terhadap Kebijakan Tapera Menggunakan Komparasi Machine Learning dan SMOTE,” JKKI, vol. 8, no. 2, pp. 125–135, Nov. 2024, doi: 10.31603/komtika.v8i2.12595.
[16] W. Priatna and J. S. Hidayat, “Implementasi Term Frequency – Inverse Document Frequency (Tf-Idf) Dan Vector Space Model (Vsm) Untuk Pencarian Berita Bahasa Indonesia,” Jurnal Ilmiah Informatika, Arsitektur dan Lingkungan, vol. 14, no. 2, pp. 119–133, 2019.
[17] S. Qaiser and R. Ali, “Text Mining: Use of TF-IDF to Examine the Relevance of Words to Documents,” IJCA, vol. 181, no. 1, pp. 25–29, Jul. 2018, doi: 10.5120/ijca2018917395.
[18] D. N. Agustia and R. R. Suryono, “Comparison of Naïve Bayes, Random Forest, and Logistic Regression Algorithms for Sentiment Analysis Online Gambling,” ISI, vol. 10, no. 1, pp. 284–295, Jan. 2025, doi: 10.35314/prk93630.
[19] A. Yaman and M. A. Cengiz, “The Effects of Kernel Functions and Optimal Hyperparameter Selection on Support Vector Machines,” Journal of New Theory, vol. 34, pp. 64–71, 2021.
[20] M. Muttakin, N. R. Rusmana, and D. Ramadhani, “Analisis Perbandingan Algoritma Decision Tree, Random Forest, Knn, Dan Svm Dalam Prediksi Penyakit Jantung,” Journal of System & Technology, vol. 1, no. 2, 2025.
[21] F. Rustam, M. Khalid, W. Aslam, V. Rupapara, A. Mehmood, and G. S. Choi, “A performance comparison of supervised machine learning models for Covid-19 tweets sentiment analysis,” PLoS ONE, vol. 16, no. 2, p. e0245909, Feb. 2021, doi: 10.1371/journal.pone.0245909.
[22] Vincent, G. Darian, and N. Surantha, “Performance Evaluation of Convolutional Neural Network (CNN) for Skin Cancer Detection on Edge Computing Devices,” Applied Sciences, vol. 15, no. 6, p. 3077, Mar. 2025, doi: 10.3390/app15063077.
[23] B. Shannaq, O. Ali, S. A. Maqbali, and A. Al-Zeidi, “Advancing user classification models: A comparative analysis of machine learning approaches to enhance faculty password policies at the University of Buraimi,” JIPD, vol. 8, no. 13, p. 9311, Nov. 2024, doi: 10.24294/jipd9311.
[24] M. Fairuzabadi, Mahir Data Science dengan Python Analisis Data & Machine Learning Untuk Pemula. Yogyakarta: Yash Media, 2025.
[25] N. Neshat, S. Gupta, N. Unadkat, K. Jain, and A. Rathwa, “AI-Driven Stroke Classification and Detection: A Retrospective Study Using MRI Imaging,” J Neonatal Surg, vol. 14, no. 15S, pp. 2294–2300, Apr. 2025, doi: 10.63682/jns.v14i15S.4928.
[26] A. M. Van Der Veen and E. Bleich, “The advantages of lexicon-based sentiment analysis in an age of machine learning,” PLoS ONE, vol. 20, no. 1, p. e0313092, Jan. 2025, doi: 10.1371/journal.pone.0313092.
[27] N. S. Sediatmoko, Y. Nataliani, and I. Suryady, “Sentiment Analysis of Customer Review Using Classification Algorithms and SMOTE for Handling Imbalanced Class,” Indonesian J. of Inf. Syst., vol. 7, no. 1, pp. 38–52, Aug. 2024, doi: 10.24002/ijis.v7i1.8879.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 M. Darma Alfero (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
