LLM Group Study Report

Introduction

This study group aims to enhance understanding of Large Language Models (LLMs) and learn related technologies.
Starting with the comparison, introduction, and overview of LLM models, the ultimate goal is to conduct a project.

Target Audience

• Individuals interested in LLMs
• Individuals with a basic understanding of Python programming
• Individuals with a basic understanding of machine learning and deep learning

Methodology

• Online (Discord room)
• Weekly regular meetings (Every Wed. 21:00 ~ 22:00 (UTC +09:00))
• Focus on presentations, discussions, and practical exercises

Schedule

Week	Learning Content	Objectives	Resources
Week 1	LLM Overview	Understand the basic concepts of LLMs	Resource 1
Week 2	LLM Model Comparison (GPT, Claude, etc.)	Compare the characteristics of various LLM models	Resource 2
Week 3	Prompt Injection	Understand prompt injection attacks and learn defense strategies	Resource 3
Week 4	Analysis of LLM processes input into tokens	Experimenting with and analyzing how an LLM processes input text into tokens	Resource 4
Week 5	Embedding words and Cosine Similarity in LLM	Experiencing the relationship between the words via LLM and analyzing mathematically in shallow way	Resource 5
Week 6	Self-Attention and Similarity Calculation	Understand how attention weights are derived from vector similarities	Resource 6
Week 7	Scaled Dot Product Attention (SDPA) Mechanics	Analyze and implement the mathematical steps of Scaled Dot Product Attention	Resource 7
Week 8	Multi-Head Attention	Analyze the mechanics of multi-head attention mathematically	Resource 8
Week 9 (Basic)	Fundamental of Position-wise Feed Forward Network	Explaining the concept of Position-wise Feed Forward Network	Resource 9-1
Week 9 (Advanced)	Experiment with Position-wise Feed Forward Network	Both experimenting and analyzing how position-wise feed forward network works	Resource 9-2
Week 10	Positional Encoding	A basic concept of positional enconding	Resource 10
Week 11	Residual Connection	Explaining how residual is essential for transformer library and residual connection works in fundamental way	Resource 11
Week 12	Epilogue	Skimmed through what transformer is and how it works basically	No Resource

Resource List

• Resource 1: LLM Overview
• Resource 2: LLM Model Comparison (GPT, Claude, etc.)
• Resource 3: Prompt Injection
• Resource 4: Tokenization in LLMs
• Resource 5: Embedding and Cosine Similarity between words
• Resource 6: How attention calculates similarty
• Resource 7: The principle of how scaled dot product attention is calculated
• Resource 8: Rigorous use of mathematics in multi-head attention
• Resource 9-1: Some basic concepts of position-wise feed forward network.
• Resource 9-2: Visualizing how the position-wise feed forward network works.
• Resource 10: Explaining about the positional encoding.
• Resource 11: The importance of residual connection in transformer library.

Progress Report

• Week 1: LLM Overview learning completed (Topic: LLM basic concept)
• Week 2: Comparison of LLM models such as GPT, Claude, etc. completed (Topic: GPT, Claude, or other LLM models comparison)
• Week 3: Prompt Injection learning completed (Topic: Prompt injection attacks and defense strategies)
• Week 4: Discussing about exploring and analyzing how an LLM breaks down input text into individual tokens (Topic: Tokenization in LLMs)
• Week 5: (Code available) Experimenting how words are related in a specific LLM (Topic: Embedding and Cosine Similarity between Words)
• Week 6: Learning how attention is calculated from vector similarity (Topic: Self-Attention and Similarity Calculation)
• Week 7: Mathematical breakdown and implementation of Scaled Dot Product Attention (Topic: Scaled Dot Product Attention Mechanics)
• Week 8: Multi-Head Attention with rigorous mathematical tools. (Topic: Mathematical Analysis of Multi-Head Attention)
• Week 9: Basic concepts and visualizing how the position-wise feed forward network works. (Topic: Position-wise Feed Forward Network)
• Week 10: Fundamental concepts of positional encoding. (Topic: Positional Encoding)
• Week 11: How residual connection works in transformers library. (Topic: Residual Connection)
• Week 12: Epilogue: Explored the wonderland of the transformer library (Next Topic: BERT)