diff --git a/.gitignore b/.gitignore index 15385fc..9701e41 100644 --- a/.gitignore +++ b/.gitignore @@ -67,7 +67,9 @@ Project/CWSNER/model/ # NCTU_DL_HW1 # TaibetanMNIST -Project/NCTU_DL_HW1/TibetanMNIST.npz +Project/NCTU_DL/HW1/TibetanMNIST-DNN/TibetanMNIST.npz +# NCTU_DL_HW2 +Project/NCTU_DL/HW2/*.zip # Byte-compiled / optimized / DLL files __pycache__/ diff --git a/.lfsconfig b/.lfsconfig index 5708500..297b850 100644 --- a/.lfsconfig +++ b/.lfsconfig @@ -1,2 +1,3 @@ [lfs] url = https://gitlab.com/daviddwlee84/DeepLearningPractice.git/info/lfs + \ No newline at end of file diff --git a/Notes/Application/NLP/EnglishPreprocessing.md b/Notes/Application/NLP/EnglishPreprocessing.md new file mode 100644 index 0000000..a5a1d19 --- /dev/null +++ b/Notes/Application/NLP/EnglishPreprocessing.md @@ -0,0 +1,6 @@ + +Tokenizer + +Stop Words + +[All you need to know about NLP Text Preprocessing](https://gdcoder.com/all-you-need-to-know-about-nlp-text-preprocessing/) diff --git a/Notes/Concept/DataSmoothing.md b/Notes/Concept/DataSmoothing.md new file mode 100644 index 0000000..8b7f3b7 --- /dev/null +++ b/Notes/Concept/DataSmoothing.md @@ -0,0 +1,71 @@ +# Data Smoothing + +## Background + +### [N-gram](N-GramModel.md) + +### [Data Sparsity](N-GramModel.md#Data-Sparseness) + +## Overview of Smoothing Technique + +Simple Smoothing + +* Addictive smoothing + * Add-one smoothing +* Held-out Estimation (留存估計) + * Deleted Estimation / Two-way Cross Validation (刪除估計) +* Good Turing smoothing +* ... etc. + +Combination Smoothing + +* Interpolation smoothing (插值) + * Jelinek-Mercer smoothing +* Katz smoothing (backoff) (退回模型) +* Kneser-Ney smoothing + +## Simple Smoothing + +> All the n-gram which didn't appear will have the same probability distribution. + +### Add-one Smoothing + +> Add one to frequency of each n-gram + +#### Addictive Smoothing + +> Add $\delta$ instead of one to frequency of each n-gram. +> (Typically, $0<\delta\leq1$) + +### Held-out Estimation + +> If the corpus is large, it's a good method. +> Since it need to split data into two set. + +#### Deleted Estimation / Two-way Cross Valiation + +> If the corpus is small + +### Good Turing Smoothing + +$$ +p_{GT} (\text{an n-gram occcuring r times}) = \frac{(r+1)N_{r+1}}{N\cdot N_r} +$$ + +* [Wiki - Good–Turing frequency estimation](https://en.wikipedia.org/wiki/Good%E2%80%93Turing_frequency_estimation) + +## Combination Smoothing + +### Interpolation Smoothing + +#### Jelinek-Mercer Smoothing + +### Katz Smoothing (Backoff Model) + +### Kneser-Ney Smoothing + +## Links + +* [**Slides - Standford NLP Lunch Tutorial: Smoothing**](https://nlp.stanford.edu/~wcmac/papers/20050421-smoothing-tutorial.pdf) +* [Wiki - Smoothing](https://en.wikipedia.org/wiki/Smoothing) +* [NLP 筆記 - 平滑方法(Smoothing)小結](http://www.shuang0420.com/2017/03/24/NLP%20%E7%AC%94%E8%AE%B0%20-%20%E5%B9%B3%E6%BB%91%E6%96%B9%E6%B3%95(Smoothing)%E5%B0%8F%E7%BB%93/) diff --git a/Notes/Concept/Dialogue.md b/Notes/Concept/Dialogue.md new file mode 100644 index 0000000..30ca76a --- /dev/null +++ b/Notes/Concept/Dialogue.md @@ -0,0 +1,49 @@ +# Dialogue + +## Overview + +### Catetory + +* **Task-oriented** dialogue: to get something done during conversation + * Assistive + * customer service + * giving recommendations + * question answering + * Co-operative + * two agents solve a task together through dialogue + * Adversarial + * two agents compete in a task through dialogue +* **Social** dialogue: no explicit task + * Chit-chat + * for fun or company + * Therapy / mental wellbeing + +### Approach + +* pre-neural dialogue system + * pre-defined templates + * retrieve an appropriate response from a corpus of responses +* open-ended freeform dialogue system + +## Problems / Solution + +A naive application of standard seq2seq+attention methods thas serious pervasive deficiencies for (chitchat) dialogue + +* Genericness / boring responses +* Irrelevant responses (not sufficiently related to context) +* Repetition +* Lack of context (not remembering conversation history) +* Lack of consistent persona + +### Irrelevant response problem + +* [[1510.03055] A Diversity-Promoting Objective Function for Neural Conversation Models](https://arxiv.org/abs/1510.03055) + +### Genericness / boring response problem + +... cs224n lecture 15 slides + +## Resources + +* [[1506.05869] A Neural Conversational Model](https://arxiv.org/abs/1506.05869) +* [Neural Responding Machine for Short-Text Conversation - ACL Anthology](https://www.aclweb.org/anthology/P15-1152/) diff --git a/Notes/Concept/Embedding.md b/Notes/Concept/Embedding.md new file mode 100644 index 0000000..c3d0c29 --- /dev/null +++ b/Notes/Concept/Embedding.md @@ -0,0 +1,20 @@ +# Embedding + +自然語言高級專題 Lect8 + +> * vs. Feature Engineering in Statistics Machine Learning +> * vs. Dictionary-based Word Representative + + + + + + +同義詞詞林 +CCD +[HowNet](http://www.keenage.com/) + + + + +[Embedding/Chinese-Word-Vectors: 100+ Chinese Word Vectors 上百种预训练中文词向量](https://github.com/Embedding/Chinese-Word-Vectors) diff --git a/Notes/Concept/GenerativeMethod.md b/Notes/Concept/GenerativeMethod.md new file mode 100644 index 0000000..5a2da2c --- /dev/null +++ b/Notes/Concept/GenerativeMethod.md @@ -0,0 +1,8 @@ + + + + + + +[CycleGAN Project Page](https://junyanz.github.io/CycleGAN/) +[junyanz/CycleGAN: Software that can generate photos from paintings, turn horses into zebras, perform style transfer, and more.](https://github.com/junyanz/CycleGAN) \ No newline at end of file diff --git a/Notes/Concept/ImageCaptioning.md b/Notes/Concept/ImageCaptioning.md new file mode 100644 index 0000000..6d92c6a --- /dev/null +++ b/Notes/Concept/ImageCaptioning.md @@ -0,0 +1,9 @@ +# Image Captioning + +cs224n lecture 15... + +## Dataset + +COCO + +[[1805.04833] Hierarchical Neural Story Generation](https://arxiv.org/abs/1805.04833) diff --git a/Notes/Concept/KnowledgeEmbedding.md b/Notes/Concept/KnowledgeEmbedding.md new file mode 100644 index 0000000..840db99 --- /dev/null +++ b/Notes/Concept/KnowledgeEmbedding.md @@ -0,0 +1,5 @@ +# Knowledge Embedding + +## Resources + +* [OpenKE](https://github.com/thunlp/OpenKE) diff --git a/Notes/Concept/MachineTranslation.md b/Notes/Concept/MachineTranslation.md new file mode 100644 index 0000000..2cb0af9 --- /dev/null +++ b/Notes/Concept/MachineTranslation.md @@ -0,0 +1,21 @@ +# Machine Translation + +## Overview + +> * Rule-based Approach +> * Corpus-based Approach + +### History + +* 1950s: Early Machine Translation + * mostly *ruled-based* - using a bilingual dictionary to map words to their counterparts +* 1990s-2010: Statistical Machine Translation + * learn a *probabilistic model* from data + * $\argmax_y P(y|x) = \argmax_y\underbrace{P(x|y)}_{\text{Translation Model}}\underbrace{P(y)}_{\text{Language Model}}$ + * learning alignment: correspondence between particular words in the translated sentence pair +* 2014 after: Neural Machine Translation + * [sequence-to-sequence](../Mechanism/seq-to-seq.md) + +## Resources + +* [Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 8 – Translation, Seq2Seq, Attention - YouTube](https://www.youtube.com/watch?v=XXtpJxZBa2c&feature=youtu.be) diff --git a/Notes/Concept/ModelCompression.md b/Notes/Concept/ModelCompression.md new file mode 100644 index 0000000..31ba562 --- /dev/null +++ b/Notes/Concept/ModelCompression.md @@ -0,0 +1,25 @@ +# Model Compression / Knowledge Distillation + +## Resources + +### Paper + +Classic + +* [Model compression](https://dl.acm.org/citation.cfm?id=1150464) +* [[1503.02531] Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531) + +Survey + +* [FLHonker/Awesome-Knowledge-Distillation: Awesome Knowledge-Distillation.](https://github.com/FLHonker/Awesome-Knowledge-Distillation) +* [[1710.09282] A Survey of Model Compression and Acceleration for Deep Neural Networks](https://arxiv.org/abs/1710.09282) + +### Tools + +* [NervanaSystems/distiller: Neural Network Distiller by Intel AI Lab: a Python package for neural network compression research.](https://github.com/NervanaSystems/distiller) + * [Distiller Documentation](https://nervanasystems.github.io/distiller) +* [GMvandeVen/continual-learning: PyTorch implementation of various methods for continual learning (XdG, EWC, online EWC, SI, LwF, DGR, DGR+distill, RtF, iCaRL).](https://github.com/GMvandeVen/continual-learning) + +Pytorch + +* [peterliht/knowledge-distillation-pytorch: A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility](https://github.com/peterliht/knowledge-distillation-pytorch) diff --git a/Notes/Concept/Storytelling.md b/Notes/Concept/Storytelling.md new file mode 100644 index 0000000..79eb873 --- /dev/null +++ b/Notes/Concept/Storytelling.md @@ -0,0 +1,3 @@ +# Storytelling + +[Storytelling Workshop 2019](http://www.visionandlanguage.net/workshop2019/) diff --git a/Notes/Concept/SubwordsModel.md b/Notes/Concept/SubwordsModel.md new file mode 100644 index 0000000..deef9fd --- /dev/null +++ b/Notes/Concept/SubwordsModel.md @@ -0,0 +1,17 @@ + + + +sentencepiece + +Byte Pair Encoding (BPE) + +[google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation.](https://github.com/google/sentencepiece) + + +fastText +Aim ... (cs224n lecture12 slides) +An extension of the word2ve skip-gram model with character n-grams + + + +[Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 12 – Subword Models - YouTube](https://www.youtube.com/watch?v=9oTHFx0Gg3Q&feature=youtu.be) \ No newline at end of file diff --git a/Notes/Concept/SyntaxAnalysis.md b/Notes/Concept/SyntaxAnalysis.md new file mode 100644 index 0000000..d90c0b2 --- /dev/null +++ b/Notes/Concept/SyntaxAnalysis.md @@ -0,0 +1,157 @@ +# Context-Free Syntax Analysis + +## Syntax Analysis Overview + +* "Word" as unit + * Segmentation, Tokenization + * Morphological analysis, Lemmatization, Stemming + * Part-of-speech Tagging + * ... +* "Sentence" as unit + * Syntactic parsing +* "Paragraph" as unit + * Discourse parsing + +What is syntax? + + +## Background + +### Chomsky Hierarchy + +![Wiki Chomsky Hierarchy](https://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Chomsky-hierarchy.svg/640px-Chomsky-hierarchy.svg.png) + +| Grammar | Languages | Automaton | Production rules (constraints) | +| ------- | --------------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| Type-0 | **Recursively enumerable** (Natural Language) | Turing machine | $αAβ→β{\displaystyle \alpha A\beta \rightarrow \beta }$ | +| Type-1 | **Context-sensitive** | Linear-bounded non-deterministic Turing machine | $αAβ→αγβ\alpha A \beta \rightarrow \alpha \gamma \beta$ | +| Type-2 | [**Context-free**](#context-free-grammer) (**CFG**) | Non-deterministic pushdown automaton | $A→α{\displaystyle A\rightarrow \alpha }$ | +| Type-3 | **Regular** | Finite state automaton | $A→a{\displaystyle A\rightarrow {\text{a}}}$ and $A→aB{\displaystyle A\rightarrow {\text{a}}B}$ | + +> Production rules's symbols: +> +> * ${\displaystyle {\text{a}}}$ = terminal +> * $A$, $B$ = non-terminal +> * $\alpha$ , $\beta$ , $\gamma$ = string of terminals and/or non-terminals +> * $\alpha$ , $\beta$ = maybe empty +> * $\gamma$ = never empty + +### Context-free Grammer + +* [Wiki - Context-free grammar](https://en.wikipedia.org/wiki/Context-free_grammar) + + + + + +... + +parse 句法樹 + +LL + +LR + +e.g. Yacc + + + + +## Syntax Analysis Algorithm + +* [Wiki - Parasing Algorithms](https://en.wikipedia.org/wiki/List_of_algorithms#Parsing) + +Parsing Algorithm + +* Top-down + * LL +* Bottom-up + * LR + * Simple LR 標準LR + > This can't be used on the language which has ambiguity and order => can't analysis natural language + * Generalized LR 廣義LR + * CYK +* Mixed/other + * Earley + +Big Picture + +1. Generate all the Syntax Tree +2. Syntax Disambiguation, i.e. find the correct Syntax Tree + +### Earley Algorithm + +* [Wiki - Earley parser](https://en.wikipedia.org/wiki/Earley_parser) + +Operations + +* Prediction +* Scanning +* Completion + +### LR ALgorithm + +* [Wiki - LR parser](https://en.wikipedia.org/wiki/LR_parser) + +#### Simple LR Algorithm + +* [Wiki - Simple LR parser](https://en.wikipedia.org/wiki/Simple_LR_parser) + +#### Generalized LR Algorithm + +* [Wiki - GLR parser](https://en.wikipedia.org/wiki/GLR_parser) + +-- + +## Statistics-based Syntax Analysis Algorithm（統計句法分析) + +PCFG vs. CFG + +For a PCFG, G is a quadruple + +$$ +G = (V_N, V_T, S, P) +$$ + +* $V_N$ +* $V_T$ +* $S$ +* $P$ + +### CYK Algorithm + +* [Wiki - CYK algorithm](https://en.wikipedia.org/wiki/CYK_algorithm) + +## Probabilistic CFG as Language Model + +> Compare with **n-gram model**, PCFG-based language model considered the structure information of a sentence. +> N-gram model can be instead considered a sentence as a linear structure + +### Probabilistic Context-Free Grammar in Detail + +> aka. Stochastic CFG + +* [Wiki - Probabilistic context-free grammar](https://en.wikipedia.org/wiki/Probabilistic_context-free_grammar) + +#### Basic questions of PCFG + +### Inside and Outside Variable + +* Inside Variable $\beta$ +* Outside Variable $\alpha$ + +### Inside Algorithm + +### Outside Algorithm + +### Viterbi Algorithm + +### Inside-Outside Algorithm + +-- + +## Dependency Parsing (依存句法分析) + +## Resources + +* CS224n 2019 Lecture 5: Linguistic Structure: Dependency Parsing diff --git a/Notes/Concept/TrainEmbedding.md b/Notes/Concept/TrainEmbedding.md new file mode 100644 index 0000000..ebce063 --- /dev/null +++ b/Notes/Concept/TrainEmbedding.md @@ -0,0 +1,129 @@ +# Train Embedding + +自然語言處理 10_ slides + +## Prediction Learning Model + +Prediction Learning Model Overview + +* Training Data + * Text = $w_1w_2 \dots w_T$ + * Sampling with a window + * $(\text{begin}w_2, w_1) (w_1w_3, w_2), \cdots, (w_{T-2}w_T, w_{T-1}), (w_{T-1}\text{end}, w_T)$ + * Objective function - Average logarithm maximum likilihood function + * Maximum Likilihood Estimation: Find a embedding that can maximum the objective function + +Modeling with Neural Network + +* Input layer +* Non-linear transform layer +* Linear transform layer +* Softmax output layer +* Pairwise rank lost + + + +TBD: formula + + +### Collobert & Weston Model + +* Language segment length (window) is 11 (i.e. l = 5) +* Non-linear activation function: HardTanh +* |V| = 30000 +* Hidden layer dimension: 100 +* Embedding dimension: 50 +* Training corpus: Wikipedia with 631M words + +Paper: [Natural Language Processing (almost) from Scratch](https://arxiv.org/abs/1103.0398) + +### CBOW (Continuous Bag-of-Words) + +> Simplifier: No hidden layer + +Predict objective word with context. + +* Input layer: Use "summation" to represent a bag-of-words +* Output layer + + +TBD + +### SkipGram + +> Even simplifier: No "summation" + +* Input layer +* Sampling with window on unlabeled training data + * $(w_1, w_2)$ +* + +### Negative Sampling Training + +> To simplify the output calucation (calculating softmax is too expensive) +> +> This training method can apply on not only SkipGram model but also any other model that use softmax as output layer (e.g. Collobert & Weston Model, CBOW Model) + +**Negative Sampling** for a given word $w$. + +* Get positive sample from training data sample $(w, c)$ +* Random generate negative sample $(w, c')$ + + + + + + +Construct negative sample + +### Conclusion of CBOW vs. SkipGram + +Predict + +* CBOW: "predicting the word given its context" +* SkipGram: "predicting the context given a word" + +Example "Hi fred how was the pizza?" + +* CBOW with 3-grams: {"Hi fred how", "fred how was", "how was the", ...} +* Skip-gram with 1-skip and 3-grams: {"Hi fred how", "Hi fred was", "fred how was", "fred how the", ...} + +## Matrix Factorization Learning Model + +* Objective word $w$, Word in context $c$ + * $w \in V_w$ + * $c \in V_c$ + * $(w, c) \in D$ Trianing data +* Define Co-occurrence Matrix $M$ + * $|V_w|$ rows + * $|V_c|$ columns + * $M_{ij}$ represent relation between $w_i$ and $c_j$ + +Relation representation + +* Naive: co-occurence count + * $M_{ij} = \#(w_i, c_j)$ +* [PMI Pointwise Mutual Information](https://en.wikipedia.org/wiki/Pointwise_mutual_information) +* PPMI Positive PMI + +### SVD + +* non-symmetric embedding based on SVD + * $W_{SVD} = U_d \cdot \Sigma_d$ + * $C_{SVD} = V_d$ +* symmetric embedding based on SVD + * $W_{SVD} = U_d \cdot \sqrt{\Sigma_d}$ + * $C_{SVD} = V_d \cdot \sqrt{\Sigma_d}$ + +### GloVe + +## Evaluation + +### Synonymous Prediction - Word Analygy Task + +### Similarity Evaluation - Word Similarity Task + +## Resources + +* [What are the continuous bag of words and skip-gram architectures?](https://www.quora.com/What-are-the-continuous-bag-of-words-and-skip-gram-architectures) +* [CS 224D: Deep Learning for NLP Lecture Notes](https://cs224d.stanford.edu/lecture_notes/notes1.pdf) diff --git a/Notes/Concept/VanishingGradient.md b/Notes/Concept/VanishingGradient.md new file mode 100644 index 0000000..64347c2 --- /dev/null +++ b/Notes/Concept/VanishingGradient.md @@ -0,0 +1,32 @@ +# Vanishing/Exploding Gradient + +> this is not just a RNN problem: it can be a problem for all neural architectures especially *deep* ones + +Conclusion + +> (paper "OnOn the difficulty of training recurrent neural networks") + +* Vanishing gradient: if the **largest eigenvalue of $W_h$ < 1**, then the gradient will **shrink** exponentially +* Exploding gradient: if the **largest eigenvalue of $W_h$ > 1**, ... + +Problem might cause + +* RNN-LM + * learning *sequential recency* > *sysyntactic recency* + * the model weight are only updated only with respect to *near effects* => can't learn *long-distance dependencies* (the information (gradient) vanished) +* SGD + * if gradient becomes too big + * bad update => take too large a step and reach a bad parameter configuration + * maybe result in *inf* or *nan* + +Solution + +* Gradient clipping: if the norm of the gradient is greater than some threshold, scale it down before applying SGD update => solution for gradient exploding +* LSTM +* Skip connections (direct connections, shortcut connections, residual connections) + + + +* Residual connection (ResNet) +* Dense connection (DenseNet) +* Highway connection (HighwayNet) \ No newline at end of file diff --git a/Notes/Concept/WordRepresentation.md b/Notes/Concept/WordRepresentation.md new file mode 100644 index 0000000..f003bd7 --- /dev/null +++ b/Notes/Concept/WordRepresentation.md @@ -0,0 +1,131 @@ +# Word Representation + +## Concept + +"Meaning" in linguistic way: (denotational semantics) + +signifier (symbol) <==> signified (idea or thing) + +## Obtain Word Meaning + +* Usable resources + * corpus + * search engine + * wikipedia +* Usable clue + * relationship between words + +Problems with resources like [WordNet](https://wordnet.princeton.edu/): + +* Great as a resource but *missing nuance* (only correct in some contexts) +* Missing new meanings of words +* subjective +* requires human labor to create and adapt +* can't comupute accurate word similarity + +### Vector Representation + +* one-hot vector + * each index represent a word + +> * sparse vector +> * able to explain by human +> * relative words will have some visible relation + +* dense vector + * hard to explain by human + +#### Representing words as discrete symbols + +In traditional NLP, we regard words as discrete symbols - a localist representation + +Words can be represented by **one-hot vector** + +* vector dimension = number of words in vocabulary +* disadvantages: + * vectors are *orthogonal*, that is no natural notion of **similarity** for one-hot vectors + +Possible solution: + +* try to rely on WordNet's list of synonyms +* instead, learn to encode similarity in the vectors themselves + +#### Representing words by their context + +> Distributional semantics: a word's meaning is given by the words that frequently appear close-by + + + +## Word Similarity based on Context + +* Context Vector + +* [brightmart/nlu_sim](https://github.com/brightmart/nlu_sim) - all kinds of baseline models for sentence similarity + +### Distributional Similarity + +### Mutual Information + +Pointwise Mutual Information (PMI) + +* [Wiki - Pointwise mutual information](https://en.wikipedia.org/wiki/Pointwise_mutual_information) + +### Similarity + +* cosine + * [Wiki - Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) +* Jaccard + * [Wiki - Jaccard index](https://en.wikipedia.org/wiki/Jaccard_index) +* Dice + * [Wiki - Sørensen–Dice coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) +* JS + * [Wiki - Jensen–Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) + +#### Word Mover's Distance + +* [WMD_tutorial](https://markroxor.github.io/gensim/static/notebooks/WMD_tutorial.html) +* [Word Mover’s Distance for Text Similarity - Towards Data Science](https://towardsdatascience.com/word-movers-distance-for-text-similarity-7492aeca71b0) + +## Explicit Semantic Analysis + +Wikipedia-based ESA + +* [Wiki - Explicit semantic analysis](https://en.wikipedia.org/wiki/Explicit_semantic_analysis) + +## Latent Semantic Analysis + +> earlier than ESA + +> human usually can't understand the meaning of the vector + +Application => SVD + +* [Wiki - Latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis) + +## Page-count-based Similarity Scores (Cooccurance Measures) + +## Word Sense Disambiguation 詞義消岐 + +## Word Sence Discrimination 詞義區分 + +### Application + +* Machine Translation +* Information Retrieval +* Question Answering +* Knowledge Acquisition + +### Lesk Algorithm + +* [Wiki - Lesk algorithm](https://en.wikipedia.org/wiki/Lesk_algorithm) + +#### Simulated Annealing + +TBD + +pun 雙關 + +homographic pun 語義雙關 + + +[Embedding/Chinese-Word-Vectors: 100+ Chinese Word Vectors 上百种预训练中文词向量](https://github.com/Embedding/Chinese-Word-Vectors) diff --git a/Notes/Mechanism/BRNN.md b/Notes/Mechanism/BRNN.md new file mode 100644 index 0000000..890a370 --- /dev/null +++ b/Notes/Mechanism/BRNN.md @@ -0,0 +1,7 @@ +# Bidirectional Recurrent Neural Network + +## Links + +### Tutorial + +* [Andrew Ng - RNN W1L11: Bidirectional RNN](https://youtu.be/bTXGpATdKRY) diff --git a/Notes/Mechanism/Pooling.md b/Notes/Mechanism/Pooling.md new file mode 100644 index 0000000..726e7fb --- /dev/null +++ b/Notes/Mechanism/Pooling.md @@ -0,0 +1,20 @@ +Global pooling => max + +Average/Mean Pooling + Max Pooling & concatenate + +Overlapped Pooling + +[Max Pooling is a downsampling strategy in Convolutional Neural Networks | Kaggle](https://www.kaggle.com/learn-forum/60325) + +In almost all of the famous CNN architectures, max pooling has been performed with overlapping regions. [Kernel Size, Stride] - + +```txt +AlexNet = [3x3, 2]; +GoogleNet = [3x3, 2] , [3x3, 1]; +VGGCNNS = [3x3,3], [2x2,2]; +VGGCNNM and variants = [3x3, 2]; +VGGCNNF = [3x3, 2]. +``` + + +Current notes are in `Layers.md`... diff --git a/Notes/Mechanism/YOLO.md b/Notes/Mechanism/YOLO.md new file mode 100644 index 0000000..ccb3a90 --- /dev/null +++ b/Notes/Mechanism/YOLO.md @@ -0,0 +1,16 @@ +# You Only Look Once + +## Resource + +### Author - [Joseph Chet Redmon](https://pjreddie.com/) + +* [YOLO: Real-Time Object Detection](https://pjreddie.com/darknet/yolo/) + +### Tutorial + +* [Youtube - YOLO Object Detection (TensorFlow tutorial)](https://youtu.be/4eIBisqx9_g) + * [code](https://github.com/llSourcell/YOLO_Object_Detection) + +### Github + +* [ultralytics/yolov3](https://github.com/ultralytics/yolov3) - YOLOv3 in PyTorch > ONNX > CoreML > iOS diff --git a/Notes/Practice/EmbeddingUsage.md b/Notes/Practice/EmbeddingUsage.md new file mode 100644 index 0000000..f8e97de --- /dev/null +++ b/Notes/Practice/EmbeddingUsage.md @@ -0,0 +1,17 @@ +gensim & glove... +fasttext +bert +elmo + +Tools +load embedding +from gensim.models import KeyedVectors +KeyedVectors.load_word2vec_format + +look up table +[Keras Text Preprocessing](https://keras.io/preprocessing/text/) +keras.preprocessing.text.Tokenizer + +padding +[Keras Sequence Preprocessing](https://keras.io/preprocessing/sequence/) +keras.preprocessing.sequence.pad_sequences diff --git a/Notes/Practice/Evaluation.md b/Notes/Practice/Evaluation.md new file mode 100644 index 0000000..df58ed8 --- /dev/null +++ b/Notes/Practice/Evaluation.md @@ -0,0 +1,5 @@ +Dataset + +GLUE + +SQuAD diff --git a/Notes/Practice/FindResources.md b/Notes/Practice/FindResources.md new file mode 100644 index 0000000..a6bfc8a --- /dev/null +++ b/Notes/Practice/FindResources.md @@ -0,0 +1,62 @@ +# Finding Resources + +> Stanford CS224n Lecture 9 course notes, slides .... + +* ACL anthology +* online preceedings of major ML conferences + * NeurIPS + * ICML + * ILR +* online preprint servers e.g. arxiv +* [Arxiv Sanity Preserver](http://www.arxiv-sanity.com/) +* Paper with Code + +> ### Paper +> +> * [**2019-2020 International Conferences in Artificial Intelligence, Machine Learning, Computer Vision, Data Mining, Natural Language Processing and Robotics**](https://jackietseng.github.io/conference_call_for_paper/) +> * [自然語言處理領域國內外著名會議和期刊](http://deepon.me/2018/10/02/%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86%E9%A2%86%E5%9F%9F%E5%9B%BD%E5%86%85%E5%A4%96%E8%91%97%E5%90%8D%E4%BC%9A%E8%AE%AE%E5%92%8C%E6%9C%9F%E5%88%8A/) +> * [NLP領域國際頂會（ACL/EMNLP/NAACLl等）的難度如何？](https://www.zhihu.com/question/266242639/answer/312713059) +> +> * Level Criteria +> * CCF level: only used in China +> * H-index +> * [h-index - Wikipedia](https://en.wikipedia.org/wiki/H-index) +> * [h-index：簡單易懂的評估指標，呈現出多數文章的被引用表現 | 臺大圖書館參考服務部落格](http://tul.blog.ntu.edu.tw/archives/2485) +> +> #### Conference +> +> NLP +> +> * ACL Association for Computational Linguistics +> * [ACL 2020](https://acl2020.org/) +> * EMNLP +> * [roomylee/EMNLP-2019-Papers](https://github.com/roomylee/EMNLP-2019-Papers) - Statistics and Accepted paper list with arXiv link of EMNLP-IJCNLP 2019 +> * NAACL North American Chapter of the Association for Computational Linguistics +> * [NAACL-HLT 2019](https://naacl2019.org/) +> * COLING +> +> Application Scenario +> +> * WWW The Web Conference +> * [WWW 2020](https://www2020.thewebconf.org/) +> +> General Models +> +> * NPIS Neural Information Processing Systems +> * [NIPS Proceedingsβ](https://papers.nips.cc/) +> +> Not Sure +> +> * CCL +> * AAAI +> * ICLR +> * [AminJun/ICLR2020](https://github.com/AminJun/ICLR2020) - ICLR2020 Downloader & Search Tool + +## Data + +Linguistic Data Consortium + +.... + +[Datasets for Natural Language Processing](https://machinelearningmastery.com/datasets-natural-language-processing/) +[niderhoff/nlp-datasets: Alphabetical list of free/public domain datasets with text data for use in Natural Language Processing (NLP)](https://github.com/niderhoff/nlp-datasets) diff --git a/Notes/Practice/GPUTraining.md b/Notes/Practice/GPUTraining.md new file mode 100644 index 0000000..d6024e3 --- /dev/null +++ b/Notes/Practice/GPUTraining.md @@ -0,0 +1,55 @@ +# GPU Training + +## Environment Setup + +* PyTorch 1.12 with Cuda 10.0.13 + +### Nvidia GPU and Cuda + +Cuda 10.0.13 +https://developer.nvidia.com/cuda-10.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal + +nvidia command + +$ nvcc --version +$ nvidia-smi + +device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu') +.to(device) +https://github.com/pytorch/pytorch/issues/1668 + +CUDA_VISIBLE_DEVICES + +[CUDA Pro Tip: Control GPU Visibility with CUDA_VISIBLE_DEVICES | NVIDIA Developer Blog](https://devblogs.nvidia.com/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/) +[cuda - How do I select which GPU to run a job on? - Stack Overflow](https://stackoverflow.com/questions/39649102/how-do-i-select-which-gpu-to-run-a-job-on) +[Set Default GPU in PyTorch - jdhao's blog](https://jdhao.github.io/2018/04/02/pytorch-gpu-usage/) +[torch.cuda — PyTorch master documentation](https://pytorch.org/docs/stable/cuda.html) +[Syllo/nvtop: NVIDIA GPUs htop like monitoring tool](https://github.com/Syllo/nvtop) + +### Pytorch version + +PyTorch 1.12 + +## Pytorch + +### Single GPU + +### Multiple GPU + +[Multi-GPU Examples — PyTorch Tutorials 1.2.0 documentation](https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html) +[Optional: Data Parallelism — PyTorch Tutorials 1.2.0 documentation](https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html) +[torch.nn — PyTorch master documentation](https://pytorch.org/docs/stable/nn.html#dataparallel-layers-multi-gpu-distributed) + +[Run Pytorch on Multiple GPUs - PyTorch Forums](https://discuss.pytorch.org/t/run-pytorch-on-multiple-gpus/20932/17) + + +RNN warning + +[RNN module weights are not part of single contiguous chunk of memory - PyTorch Forums](https://discuss.pytorch.org/t/rnn-module-weights-are-not-part-of-single-contiguous-chunk-of-memory/6011) + +```py +# add +self.rnn.flatten_parameters() +# before +output, hn = self.rnn(x) +``` diff --git a/Notes/Practice/Metrics.md b/Notes/Practice/Metrics.md new file mode 100644 index 0000000..da47ddf --- /dev/null +++ b/Notes/Practice/Metrics.md @@ -0,0 +1,4 @@ +Sentence Similarity: Accuracy +CWS, NER: F1-score +Machine Translation: BLUE +... \ No newline at end of file diff --git a/Notes/SOTA_Models/Capsule.md b/Notes/SOTA_Models/Capsule.md new file mode 100644 index 0000000..2dc02e9 --- /dev/null +++ b/Notes/SOTA_Models/Capsule.md @@ -0,0 +1,13 @@ +[gram-ai/capsule-networks: A PyTorch implementation of the NIPS 2017 paper "Dynamic Routing Between Capsules".](https://github.com/gram-ai/capsule-networks) + + +[Capsule Networks: The New Deep Learning Network - Towards Data Science](https://towardsdatascience.com/capsule-networks-the-new-deep-learning-network-bd917e6818e8) + +[Capsule Networks (CapsNets) – Tutorial - YouTube](https://www.youtube.com/watch?v=pPN8d0E3900) + +[Capsule Networks: An Improvement to Convolutional Networks - YouTube](https://www.youtube.com/watch?v=VKoLGnq15RM) + +[Multi-Labeled Relation Extraction with Attentive Capsule Network](http://bcmi.sjtu.edu.cn/~zhaohai/pubs/aaai2019-3243-capsule_v5.pdf) + + +[[1710.09829] Dynamic Routing Between Capsules](https://arxiv.org/abs/1710.09829) \ No newline at end of file diff --git a/Notes/SOTA_Models/GAN.md b/Notes/SOTA_Models/GAN.md new file mode 100644 index 0000000..b42a1da --- /dev/null +++ b/Notes/SOTA_Models/GAN.md @@ -0,0 +1,8 @@ +# Generative Adversarial Network + +## Resources + +* [A Beginner's Guide to Generative Adversarial Networks (GANs)](https://skymind.ai/wiki/generative-adversarial-network-gan) +* [GANs from Scratch 1: A deep introduction. With code in PyTorch and TensorFlow](https://medium.com/ai-society/gans-from-scratch-1-a-deep-introduction-with-code-in-pytorch-and-tensorflow-cb03cdcdba0f) +* [diegoalejogm/gans: Generative Adversarial Networks implemented in PyTorch and Tensorflow](https://github.com/diegoalejogm/gans) +* [PyTorch Example - Deep Convolution Generative Adversarial Networks](https://github.com/pytorch/examples/tree/master/dcgan) diff --git a/Notes/SOTA_Models/GPT.md b/Notes/SOTA_Models/GPT.md new file mode 100644 index 0000000..2711f62 --- /dev/null +++ b/Notes/SOTA_Models/GPT.md @@ -0,0 +1,12 @@ + +GPT + +GPT-2 + +[AI Language Models & Transformers - Computerphile - YouTube](https://www.youtube.com/watch?v=rURRYI66E54) + + + +[Morizeyao/GPT2-Chinese: Chinese version of GPT2 training code, using BERT tokenizer.](https://github.com/Morizeyao/GPT2-Chinese) + +[imcaspar/gpt2-ml: GPT2 for Multiple Languages, including pretrained models. GPT2 多语言支持, 15亿参数中文预训练模型](https://github.com/imcaspar/gpt2-ml) \ No newline at end of file diff --git a/Notes/SOTA_Models/NLP/QA/BiDAF.md b/Notes/SOTA_Models/NLP/QA/BiDAF.md new file mode 100644 index 0000000..233b3fd --- /dev/null +++ b/Notes/SOTA_Models/NLP/QA/BiDAF.md @@ -0,0 +1,8 @@ +# BiDAF: Bidirectional Attention Flow for Machine Comprehension + +* [BiDAF](https://allenai.github.io/bi-att-flow/) + * [pdf](https://arxiv.org/pdf/1611.01603.pdf) + + +Machine Comprehension vs. Question Answering + diff --git a/Notes/SOTA_Models/Overview.md b/Notes/SOTA_Models/Overview.md new file mode 100644 index 0000000..e69de29 diff --git a/Notes/SOTA_Models/SMN.md b/Notes/SOTA_Models/SMN.md new file mode 100644 index 0000000..19a9284 --- /dev/null +++ b/Notes/SOTA_Models/SMN.md @@ -0,0 +1,21 @@ +# Sequential Matching Network + +[Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://arxiv.org/abs/1612.01627) + +## Overview + +### Model + +![SMN](https://user-images.githubusercontent.com/166852/37170780-c95b6bca-22d9-11e8-8a63-0f7095f45f10.png) + +1. Word embedding +2. GRU +3. CNN +4. GRU +5. Score + +## Utterance-Response Matching + +## Matching Accumulation + +## Matching Prediction diff --git a/Notes/SOTA_Models/TextCNN.md b/Notes/SOTA_Models/TextCNN.md new file mode 100644 index 0000000..4ccc268 --- /dev/null +++ b/Notes/SOTA_Models/TextCNN.md @@ -0,0 +1,3 @@ +https://github.com/ShawnyXiao/TextClassification-Keras +[bigboNed3/chinese_text_cnn: TextCNN Pytorch实现 中文文本分类 情感分析](https://github.com/bigboNed3/chinese_text_cnn) +[Shawn1993/cnn-text-classification-pytorch: CNNs for Sentence Classification in PyTorch](https://github.com/Shawn1993/cnn-text-classification-pytorch) \ No newline at end of file diff --git a/Notes/SOTA_Models/VAE.md b/Notes/SOTA_Models/VAE.md new file mode 100644 index 0000000..af31ce2 --- /dev/null +++ b/Notes/SOTA_Models/VAE.md @@ -0,0 +1,5 @@ +# Variational Autoencoder + +## Resources + +* [PyTorch Example - VAE](https://github.com/pytorch/examples/tree/master/vae) diff --git a/Notes/SOTA_Models/XLNet.md b/Notes/SOTA_Models/XLNet.md new file mode 100644 index 0000000..0effad7 --- /dev/null +++ b/Notes/SOTA_Models/XLNet.md @@ -0,0 +1,5 @@ +# XLNet + +[zihangdai/xlnet: XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://github.com/zihangdai/xlnet) +[2019-NLP最強模型: XLNet - Taiwan AI Academy - Medium](https://medium.com/ai-academy-taiwan/2019-nlp%E6%9C%80%E5%BC%B7%E6%A8%A1%E5%9E%8B-xlnet-ac728b400de3) +[[1906.08237] XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) diff --git a/Notes/Structure/SiameseNetwork.md b/Notes/Structure/SiameseNetwork.md new file mode 100644 index 0000000..361c565 --- /dev/null +++ b/Notes/Structure/SiameseNetwork.md @@ -0,0 +1,41 @@ +# Siamese Networks + +## Overview + +### Major Application + +* Face Recognition based upon Similarity +* Object Classifier +* Detect minute changes in Documents +* Blood Cell Classification + +## Concepts + +### Contrastive Loss + +### Oneshot Learning + +> Oneshot learning is like extracting the feature vectors of the input images from the trained model to say **how much both of the images are dissimilar** without training the images of large datasets + +### Few Shot Learning + +## Resources + +### Article + +* [One Shot Learning with Siamese Networks using Keras](https://towardsdatascience.com/one-shot-learning-with-siamese-networks-using-keras-17f34e75bb3d) +* [**Siamese Neural Network ( With Pytorch Code Example ) - Innovation Incubator Group of Companies**](https://innovationincubator.com/siamese-neural-network-with-pytorch-code-example/) + * [example code](https://colab.research.google.com/drive/1FsixLon5Zz3_rFA0xIHzc8Tvnnw8FLr8) +* [One Shot Learning and Siamese Networks in Keras – Neural Tinkering](https://sorenbouma.github.io/blog/oneshot/) + +### Example + +* [delijati/pytorch-siamese: Siamese Network implementation using Pytorch](https://github.com/delijati/pytorch-siamese) +* [akshaysharma096/Siamese-Networks: Few Shot Learning by Siamese Networks, using Keras.](https://github.com/akshaysharma096/Siamese-Networks) + * [Few short learning with Siamese Networks](https://github.com/akshaysharma096/Siamese-Networks/blob/master/Few%20Shot%20Learning%20-%20V1.ipynb) +* [likejazz/Siamese-LSTM: Siamese LSTM for evaluating semantic similarity between sentences of the Quora Question Pairs Dataset.](https://github.com/likejazz/Siamese-LSTM) + +### Paper + +* [Siamese Neural Networks for One-shot Image Recognition](http://www.cs.utoronto.ca/~gkoch/files/msc-thesis.pdf) +* [Siamese Neural Networks for One-shot Image Recognition](https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf) diff --git a/Notes/Structure/seq-to-seq.md b/Notes/Structure/seq-to-seq.md new file mode 100644 index 0000000..e273c83 --- /dev/null +++ b/Notes/Structure/seq-to-seq.md @@ -0,0 +1,83 @@ +# Sequence-to-Sequence (Encoder-Decoder) Model + +> an example of Conditional Language Model +> +> * Language Model: the decoder is predicting the next word of the target sentence y +> * Conditional: its predictions are also conditioned on the source sentence x + +* **Encoder**: Input sequence --> Hidden states of input (encoding of the input) +* **Decoder**: Hidden state of output --> Output sequence + * Prediction of an output sequenc conditioned on an input sequence + +Applications + +* Machine translation (source language -> target language) +* Response generation in dialogue models +* Summarization (long text -> short text) +* Answer generation +* Paraphrase generation +* Dialogue (previous utterances -> next utterance) +* Parsing (input text -> output parse as sequence) +* Code generation (neural language -> code) + +## General Concept + +### Decoding + +#### Greedy Decoding + +generate (or "decode") the target sentence by *taking argmax on each step* of decoder + +stopping criterion: keep decoding until the model produces a *\ token* + +problem: there is no way to undo decisions + +#### Exhaustive Search Decoding + +try computing all possible sequences + +> too expensive + +#### Beam Search Decoding + +core idea: on each step of decoder, keep track of the k most probable partial translation (which is called *hypotheses*) + +(k is the *beam size*) + +> in bean search decoding, different hypotheses may produce \ tokens on different timesteps + +stopping criterion: when a hypothesis produces \, that hypothesis is complete => place it aside and continue exploring other hypotheses via beam search + +> usually we contine beam search until +> +> * reach timestep T +> * have at least n completed hypotheses +> +> (where T and n is pre-defined cutoff) + +## Different Structures + +### Neural Recurrent Sequence Models + +### Recurrent Sequence to Sequence + +### Convolutional-Based Sequence Models + +### Transformer-based seq-to-seq + +### Attention (The Encoder-Attention-Decoder) Architecture + +**Attention**: "Pay attention to" different sub-sequences of the input when generating each of the token of the output + +Modeling alignment in machine translation + +### Bidirectional Encode Representations from Transformers (BERT) + +Self-attention, borrowed from NMT, is powerful in representing contexts and responses + +## Recources + +### Tutorial + +* [Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 8 – Translation, Seq2Seq, Attention - YouTube](https://www.youtube.com/watch?v=XXtpJxZBa2c&feature=youtu.be) +* [Andrew Ng - RNN W1L02: Notation (of sequence model)](https://youtu.be/XeQN82D4bCQ) diff --git a/Notes/Technique/GNN.md b/Notes/Technique/GNN.md new file mode 100644 index 0000000..a562ab9 --- /dev/null +++ b/Notes/Technique/GNN.md @@ -0,0 +1,52 @@ +# Graph Neural Network + +## Concept + +### Graph Representations + +#### Algebraic Representation + +* $A/W$ Adjacency Matrix + * $a_{ii} = 0$ $\forall i$ if the graphs has no self-loops + * $a_{ij} = a_{ji}$ if the graph is undirected +* $D$ Degree Matrix + * $D = \operatorname{diag}(A1)$ + * $\operatorname{diag}(x)$ is a diagonal matrix with the entries of $x$ along the diagonal +* Laplacian Matrix: used to develop the concepts of graph frequency + * Combinatorial Graph Laplacian Matrix + * $L = D - A$ + * Random Walk Graph Laplacian Matrix + * $\mathcal{T} = I - D^{-1}A$ + * all nodes have degree 1 + * Symmetric Normalized Graph Laplacian Matrix + * $\mathcal{L} = D^{-1/2}LD^{-1/2} = I - D^{-1/2}AD^{-1/2}$ + * $(\mathcal{L})_{ij} = -a_{i,j}\frac{1}{\sqrt{d_i}\sqrt{d_j}}$, $i \neq j$ + * all nodes have degree 1 + +### Graph Filter + +#### Vertex Domain + +$$ +y = Tx +$$ + +* Linear Graph Filter $T$ + * Simple 1-hop filters: + * $T$ is chosen to be $A$ + * 1-hop operator + * i.e. the sum of neighboring nodes + $$y(i) = \sum_{j\in N_i} a_{ij}x(j)$$ + * can be used to infer unknown neighbor node using the average + * prediction error + * $y^{n(i)} = x(i) - y(i)$ + * $D^{-1}A$ + * $\mathcal{T} = I - D^{-1}A$ + * $L$ + +#### Spatial domain + +## Resources + +* [Must-read papers on GNN](https://github.com/thunlp/GNNPapers) +* [[1312.6203] Spectral Networks and Locally Connected Networks on Graphs](https://arxiv.org/abs/1312.6203) diff --git a/Notes/Technique/GraphConvolutionalNetwork.md b/Notes/Technique/GraphConvolutionalNetwork.md new file mode 100644 index 0000000..21f97d8 --- /dev/null +++ b/Notes/Technique/GraphConvolutionalNetwork.md @@ -0,0 +1,8 @@ +# Graph Convolutional Network + +## Resources + +### Paper + +* [Semi-Supervised Classification with Graph Convolutional Networks]() + * [《Semi-Supervised Classification with Graph Convolutional Networks》閱讀筆記](https://zhuanlan.zhihu.com/p/31067515) diff --git a/Project/ImgClassifierAPI/README.md b/Project/ImgClassifierAPI/README.md new file mode 100644 index 0000000..73338f2 --- /dev/null +++ b/Project/ImgClassifierAPI/README.md @@ -0,0 +1,18 @@ + +## TODO + +1. Model +2. Flask API +3. Test +4. Docker + +## Links + +* [Deploying a Machine Learning Model as a REST API](https://towardsdatascience.com/deploying-a-machine-learning-model-as-a-rest-api-4a03b865c166) +* [HTTP/REST API File Uploads](https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/) + +https://stackoverflow.com/questions/33279153/rest-api-file-ie-images-processing-best-practices + + +https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html +https://github.com/avinassh/pytorch-flask-api \ No newline at end of file diff --git a/Project/ImgClassifierAPI/api.py b/Project/ImgClassifierAPI/api.py new file mode 100644 index 0000000..4f071d7 --- /dev/null +++ b/Project/ImgClassifierAPI/api.py @@ -0,0 +1,35 @@ +from flask import Flask +from flask_restful import reqparse, abort, Api, Resource +import numpy as np + +app = Flask(__name__) +api = Api(app) + +# argument parsing +parser = reqparse.RequestParser() +parser.add_argument('query') + +class PredictSentiment(Resource): + def get(self): + # use parser and find the user's query + args = parser.parse_args() + user_query = args['query'] + + # vectorize the user's query and make a prediction + uq_vectorized = model.vectorizer_transform(np.array([user_query])) + prediction = model.predict(uq_vectorized) + pred_proba = model.predict_proba(uq_vectorized) + + # Output either 'Negative' or 'Positive' along with the score + if prediction == 0: + pred_text = 'Negative' + else: + pred_text = 'Positive' + + # round the predict proba value and set to new variable + confidence = round(pred_proba[0], 3) + + # create JSON object + output = {'prediction': pred_text, 'confidence': confidence} + + return output \ No newline at end of file diff --git a/Project/ImgClassifierAPI/model.py b/Project/ImgClassifierAPI/model.py new file mode 100644 index 0000000..969ef0b --- /dev/null +++ b/Project/ImgClassifierAPI/model.py @@ -0,0 +1,19 @@ +from keras.applications.resnet50 import ResNet50 +from keras.preprocessing import image +from keras.applications.resnet50 import preprocess_input, decode_predictions +import numpy as np + +def getModel() + model = ResNet50(weights='imagenet') + return model + +img = image.load_img(img_path, target_size=(224, 224)) +x = image.img_to_array(img) +x = np.expand_dims(x, axis=0) +x = preprocess_input(x) + +preds = model.predict(x) +# decode the results into a list of tuples (class, description, probability) +# (one such list for each sample in the batch) +print('Predicted:', decode_predictions(preds, top=3)[0]) +# Predicted: [(u'n02504013', u'Indian_elephant', 0.82658225), (u'n01871265', u'tusker', 0.1122357), (u'n02504458', u'African_elephant', 0.061040461)] \ No newline at end of file diff --git a/Project/MachineTranslation/README.md b/Project/MachineTranslation/README.md new file mode 100644 index 0000000..ce2c493 --- /dev/null +++ b/Project/MachineTranslation/README.md @@ -0,0 +1,23 @@ +# Machine Translation + +## Resources + +### Tutorial + +From Scratch + +* [NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.2.0 documentation](https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html) + +Use TorchTextj + +* [Language Translation with TorchText — PyTorch Tutorials 1.2.0 documentation](https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html) + +Transformer + +* [**The Annotated Transformer**](http://nlp.seas.harvard.edu/2018/04/03/attention.html) +* [jadore801120/attention-is-all-you-need-pytorch](https://github.com/jadore801120/attention-is-all-you-need-pytorch) - A PyTorch implementation of the Transformer model in "Attention is All You Need" +* [bojone/attention: some attention implements](https://github.com/bojone/attention) + +### Article + +* [【機器翻譯】transformer | ESON](https://blog.eson.org/pub/664e9bad/) diff --git a/Project/NCTU_DL/HW1/DL_HW1.pdf b/Project/NCTU_DL/HW1/DL_HW1.pdf new file mode 100644 index 0000000..4f3bc4d Binary files /dev/null and b/Project/NCTU_DL/HW1/DL_HW1.pdf differ diff --git a/Project/NCTU_DL/HW1/README.md b/Project/NCTU_DL/HW1/README.md new file mode 100644 index 0000000..9fd0cb0 --- /dev/null +++ b/Project/NCTU_DL/HW1/README.md @@ -0,0 +1,23 @@ +# NCTU Deep Learning 2020 Homework 1 + +## 1. DNN from scratch - Tibetan MNIST + +> More detail about softmax derivation checkout [here](../SoftmaxDerivation) + +Dataset: [MNIST of Tibetan handwriting](https://github.com/bat67/TibetanMNIST) + +```sh +# Download dataset +wget https://github.com/bat67/TibetanMNIST/raw/master/Datasets/TibetanMNIST.npz +``` + +## 2. Convolutional Neural Network for Image Recognition - Medical Mask + + + +## Resources + +TODO + +* http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture4.pdf +* https://github.com/ckyken/DL diff --git a/Project/NCTU_DL/HW1/TibetanMNIST-DNN/TibetanDNN.py b/Project/NCTU_DL/HW1/TibetanMNIST-DNN/TibetanDNN.py new file mode 100644 index 0000000..6cc233c --- /dev/null +++ b/Project/NCTU_DL/HW1/TibetanMNIST-DNN/TibetanDNN.py @@ -0,0 +1,109 @@ +import numpy as np +from sklearn.model_selection import train_test_split + +RAND_SEED = 87 + +def load_data(path='TibetanMNIST.npz'): + data = np.load(path) + image = data['image'] + label = data['label'] + + X_train, y_train, X_test, y_test = train_test_split(image, label, test_size=0.3, random_state=RAND_SEED) + return X_train, y_train, X_test, y_test + + +def Softmax(x): + """ + Why mius np.max(x) is for the numerical stability, and it will be canceled out + https://stackoverflow.com/questions/34968722/how-to-implement-the-softmax-function-in-python + """ + e_x = np.exp(x - np.max(x)) + return e_x / e_x.sum(axis=0) # only difference + +class CrossEntropy(): + def __init__(self, use_for_loop:bool = False): + self.use_for_loop = use_for_loop + + def __call__(self, y_hat, y): + """ + Assume y is one-hot vector + """ + # Avoid division by zero + y_hat = np.clip(y_hat, 1e-15, 1 - 1e-15) + + probs = Softmax(y_hat) + log_likelihood = 0 + if self.use_for_loop: + for n in range(y.shape[0]): + for k in range(y.shape[1]): + if y[n, k] == 1: + log_likelihood += -np.log(y_hat[n, k]) + else: + y = np.argmax(y, axis=1) + log_likelihood = -np.log(probs[range(y.shape[0]), y]) + + loss = np.sum(log_likelihood) / y.shape[0] + + return loss + + def gradient(self, y_hat, y): + return y_hat - y + + +class Sigmoid(): + def __call__(self, x): + # To prevent from overflow encountered in exp + x = np.clip(x, 1e-15, 1 - 1e-15) + + return 1.0 / (1.0 + np.exp(-x)) + + def gradient(self, x): + return self(x) * (1.0 - self(x)) + + +class FCNNLayer: + # TODO: make data to be "row"-based + def __init__(self, node_num: int, last_layer_num: int, activation=None, custom_W=None, custom_b=None): + self.__node_num = node_num # output dimension + self.__last_layer_num = last_layer_num # input dimension + + if custom_W is not None: + self.W_ = custom_W + assert custom_W.shape == (self.__last_layer_num, self.__node_num) + else: + limit = 1 / np.sqrt(self.__last_layer_num) + self.W_ = np.random.uniform(-limit, limit, + (self.__last_layer_num, self.__node_num)) + if custom_b is not None: + self.b_ = custom_b + assert custom_b.shape == (self.__node_num, 1) + else: + self.b_ = np.zeros((self.__node_num, 1)) + + self.activation = activation + self.__layer_input = None + + def forward_propagation(self, last_layer): + self.__layer_input = last_layer + if self.activation: + return self.activation(np.dot(self.W_.T, last_layer) + self.b_) + else: + return np.dot(self.W_.T, last_layer) + self.b_ + + def back_propagation(self, gradient, learning_rate=0.0001): + if self.activation: + gradient = self.activation.gradient(gradient) + + W_temp = self.W_ + + gradient_W = self.__layer_input.dot(gradient.T) + gradient_b = gradient + + # Update parameters + self.W_ = self.W_ - learning_rate * gradient_W + assert self.W_.shape == gradient_W.shape + self.b_ = self.b_ - learning_rate * gradient_b + assert self.b_.shape == gradient_b.shape + + accumulated_gradient = W_temp.dot(gradient) + return accumulated_gradient \ No newline at end of file diff --git a/Project/NCTU_DL/HW1/requirements.txt b/Project/NCTU_DL/HW1/requirements.txt new file mode 100644 index 0000000..654ff28 --- /dev/null +++ b/Project/NCTU_DL/HW1/requirements.txt @@ -0,0 +1,4 @@ +sklearn +numpy +matplotlib +pytorch \ No newline at end of file diff --git a/Project/NCTU_DL/HW2/DL_HW2.pdf b/Project/NCTU_DL/HW2/DL_HW2.pdf new file mode 100644 index 0000000..50d9f16 Binary files /dev/null and b/Project/NCTU_DL/HW2/DL_HW2.pdf differ diff --git a/Project/NCTU_DL/HW2/README.md b/Project/NCTU_DL/HW2/README.md new file mode 100644 index 0000000..d151838 --- /dev/null +++ b/Project/NCTU_DL/HW2/README.md @@ -0,0 +1,9 @@ +# NCTU Deep Learning 2020 Homework 2 + +## 1. Recurrent Neural Network for Classification - COVID 19 + +Dataset: [CSSEGISandData/COVID-19: Novel Coronavirus (COVID-19) Cases, provided by JHU CSSE](https://github.com/CSSEGISandData/COVID-19) + +## 2. Variational Autoencoder for Image Generation - Anime Faces + +Dataset: [Anime Faces | Kaggle](https://www.kaggle.com/soumikrakshit/anime-faces) diff --git a/Project/NCTU_DL/HW2/covid_19.csv b/Project/NCTU_DL/HW2/covid_19.csv new file mode 100644 index 0000000..e025d64 --- /dev/null +++ b/Project/NCTU_DL/HW2/covid_19.csv @@ -0,0 +1,188 @@ +,Lat,Long,1/22/20,1/23/20,1/24/20,1/25/20,1/26/20,1/27/20,1/28/20,1/29/20,1/30/20,1/31/20,2/1/20,2/2/20,2/3/20,2/4/20,2/5/20,2/6/20,2/7/20,2/8/20,2/9/20,2/10/20,2/11/20,2/12/20,2/13/20,2/14/20,2/15/20,2/16/20,2/17/20,2/18/20,2/19/20,2/20/20,2/21/20,2/22/20,2/23/20,2/24/20,2/25/20,2/26/20,2/27/20,2/28/20,2/29/20,3/1/20,3/2/20,3/3/20,3/4/20,3/5/20,3/6/20,3/7/20,3/8/20,3/9/20,3/10/20,3/11/20,3/12/20,3/13/20,3/14/20,3/15/20,3/16/20,3/17/20,3/18/20,3/19/20,3/20/20,3/21/20,3/22/20,3/23/20,3/24/20,3/25/20,3/26/20,3/27/20,3/28/20,3/29/20,3/30/20,3/31/20,4/1/20,4/2/20,4/3/20,4/4/20,4/5/20,4/6/20,4/7/20,4/8/20,4/9/20,4/10/20,4/11/20,4/12/20 +,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum,sum +Country/Region,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, +Afghanistan,33.0,65.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,4,4,5,7,7,7,11,16,21,22,22,22,24,24,40,40,74,84,94,110,110,120,170,174,237,273,281,299,349,367,423,444,484,521,555,607 +Albania,41.1533,20.1683,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,10,12,23,33,38,42,51,55,59,64,70,76,89,104,123,146,174,186,197,212,223,243,259,277,304,333,361,377,383,400,409,416,433,446 +Algeria,28.0339,1.6596,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,3,5,12,12,17,17,19,20,20,20,24,26,37,48,54,60,74,87,90,139,201,230,264,302,367,409,454,511,584,716,847,986,1171,1251,1320,1423,1468,1572,1666,1761,1825,1914 +Andorra,42.5063,1.5218,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,39,39,53,75,88,113,133,164,188,224,267,308,334,370,376,390,428,439,466,501,525,545,564,583,601,601,638 +Angola,-11.2027,17.8739,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,2,3,3,3,4,4,5,7,7,7,8,8,8,10,14,16,17,19,19,19,19,19 +Antigua and Barbuda,17.0608,-61.7964,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,3,3,3,7,7,7,7,7,7,7,9,15,15,15,15,19,19,19,19,21,21 +Argentina,-38.4161,-63.6167,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,2,8,12,12,17,19,19,31,34,45,56,68,79,97,128,158,266,301,387,387,502,589,690,745,820,1054,1054,1133,1265,1451,1451,1554,1628,1715,1795,1975,1975,2142 +Armenia,40.0691,45.0382,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,4,8,18,26,52,78,84,115,136,160,194,235,249,265,290,329,407,424,482,532,571,663,736,770,822,833,853,881,921,937,967,1013 +Australia,-255.9695,1129.8623,0,0,0,0,4,5,5,6,9,9,12,12,12,13,13,14,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,25,27,30,39,52,55,60,63,76,91,107,128,128,200,250,297,377,452,568,681,791,1071,1549,1682,2044,2364,2810,3143,3640,3984,4361,4559,4862,5116,5330,5550,5687,5797,5895,6010,6108,6215,6303,6315 +Austria,47.5162,14.5501,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,3,9,14,18,21,29,41,55,79,104,131,182,246,302,504,655,860,1018,1332,1646,2013,2388,2814,3582,4474,5283,5588,6909,7657,8271,8788,9618,10180,10711,11129,11524,11781,12051,12297,12639,12942,13244,13555,13806,13945 +Azerbaijan,40.1431,47.5769,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,3,6,6,9,9,9,11,11,11,15,15,23,28,28,28,44,44,53,65,72,87,93,122,165,182,209,273,298,359,400,443,521,584,641,717,822,926,991,1058,1098 +Bahamas,25.0343,-77.3963,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,3,3,4,4,4,5,5,9,10,10,11,14,14,21,24,24,28,28,29,33,40,41,42,46,46 +Bahrain,26.0275,50.55,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,23,33,33,36,41,47,49,49,52,55,60,85,85,95,110,195,195,195,210,214,214,228,256,278,285,305,334,377,392,419,458,466,476,499,515,567,569,643,672,688,700,756,811,823,887,925,1040,1136 +Bangladesh,23.685,90.3563,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,3,3,3,3,5,8,10,14,17,20,25,27,33,39,39,44,48,48,48,49,51,54,56,61,70,88,123,164,218,330,424,482,621 +Barbados,13.1939,-59.5432,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,5,5,6,14,17,18,18,18,24,26,33,33,34,34,46,51,52,56,60,63,63,66,67,68,71 +Belarus,53.7098,27.9534,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,6,6,6,6,6,6,9,9,12,27,27,27,36,36,51,51,69,76,76,81,81,86,86,94,94,94,152,152,163,304,351,440,562,700,861,1066,1486,1981,2226,2578 +Belgium,50.8333,4.0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,8,13,23,50,109,169,200,239,267,314,314,559,689,886,1058,1243,1486,1795,2257,2815,3401,3743,4269,4937,6235,7284,9134,10836,11899,12775,13964,15348,16770,18431,19691,20814,22194,23403,24983,26667,28018,29647 +Belize,13.1939,-59.5432,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,2,2,3,3,3,3,4,4,5,7,7,8,9,10,13,14 +Benin,9.3077,2.3158,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,2,2,5,6,6,6,6,6,6,6,9,13,13,16,16,22,26,26,26,26,35,35,35 +Bhutan,27.5142,90.4336,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,3,3,4,4,4,4,5,5,5,5,5,5,5,5,5,5,5 +Bolivia,-16.2902,-63.5887,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,10,10,11,11,12,12,15,19,24,27,29,32,43,61,74,81,97,107,115,123,132,139,157,183,194,210,264,268,275,300 +Bosnia and Herzegovina,43.9159,17.6791,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,3,3,5,7,11,13,18,24,25,26,38,63,89,93,126,136,166,176,191,237,258,323,368,420,459,533,579,624,654,674,764,804,858,901,946,1009 +Botswana,-22.3285,24.6849,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,4,4,4,4,4,6,6,6,6,13,13,13,13 +Brazil,-14.235,-51.9253,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,2,2,2,2,4,4,13,13,20,25,31,38,52,151,151,162,200,321,372,621,793,1021,1546,1924,2247,2554,2985,3417,3904,4256,4579,5717,6836,8044,9056,10360,11130,12161,14034,16170,18092,19638,20727,22192 +Brunei,4.5353,114.7277,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,11,11,37,40,50,54,56,68,75,78,83,88,91,104,109,114,115,120,126,127,129,131,133,134,135,135,135,135,135,135,136,136,136 +Bulgaria,42.7339,25.4858,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,4,7,7,23,41,51,52,67,92,94,127,163,187,201,218,242,264,293,331,346,359,399,422,457,485,503,531,549,577,593,618,635,661,675 +Burkina Faso,12.2383,-1.5616,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,2,2,2,3,15,15,20,33,40,64,75,99,114,146,152,180,207,222,246,261,282,288,302,318,345,364,384,414,443,443,484,497 +Burma,21.9162,95.956,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,8,10,14,15,15,20,20,21,21,22,22,22,23,27,38,41 +Burundi,-3.3731,29.9189,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,3,3,3,3,3,3,3,3,5,5 +Cabo Verde,16.5388,-23.0418,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,3,3,3,4,4,5,5,6,6,6,6,6,6,7,7,7,7,7,7,7,8,8 +Cambodia,11.55,104.9167,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,3,3,5,7,7,7,33,35,37,51,53,84,87,91,96,96,99,99,103,107,109,109,110,114,114,114,114,115,117,119,119,120,122 +Cameroon,3.8480000000000003,11.5021,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,2,2,2,2,2,4,10,10,13,20,27,40,56,66,75,75,91,91,139,139,193,233,306,509,555,650,658,658,730,730,820,820,820 +Canada,671.7607,-1237.6289,0,0,0,0,1,1,2,2,2,4,4,4,4,4,5,5,7,7,7,7,7,7,7,7,7,7,8,8,8,8,9,9,9,10,11,11,13,14,20,24,27,30,33,37,49,54,64,77,79,108,117,193,198,252,415,478,657,800,943,1277,1469,2088,2790,3251,4042,4682,5576,6280,7398,8527,9560,11284,12437,12978,15756,16563,17872,19141,20654,22059,23316,24298 +Central African Republic,6.6111,20.9394,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,3,3,3,3,3,3,3,3,3,3,3,3,3,3,8,8,8,8,8,8,8,8,8,8 +Chad,15.4542,18.7322,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,3,3,3,3,3,3,5,7,7,8,8,9,9,9,10,10,11,11,11,18 +Chile,-35.6751,-71.543,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,4,4,4,8,8,13,23,23,43,61,74,155,201,238,238,434,537,632,746,922,1142,1306,1610,1909,2139,2449,2738,3031,3404,3737,4161,4471,4815,5116,5546,5972,6501,6927,7213 +China,1083.3367,3684.4197000000004,548,643,920,1406,2075,2877,5509,6087,8141,9802,11891,16630,19716,23707,27440,30587,34110,36814,39829,42354,44386,44759,59895,66358,68413,70513,72434,74211,74619,75077,75550,77001,77022,77241,77754,78166,78600,78928,79356,79932,80136,80261,80386,80537,80690,80770,80823,80860,80887,80921,80932,80945,80977,81003,81033,81058,81102,81156,81250,81305,81435,81498,81591,81661,81782,81897,81999,82122,82198,82279,82361,82432,82511,82543,82602,82665,82718,82809,82883,82941,83014,83134 +Colombia,4.5709,-74.2973,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,3,9,9,13,22,34,54,65,93,102,128,196,231,277,378,470,491,539,608,702,798,906,1065,1161,1267,1406,1485,1579,1780,2054,2223,2473,2709,2776 +Congo (Brazzaville),-4.0383,21.7587,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,3,3,3,3,4,4,4,4,4,4,19,19,19,19,22,22,22,45,45,45,45,60,60,60,60 +Congo (Kinshasa),-4.0383,21.7587,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,2,3,4,14,18,23,30,36,45,48,51,51,65,65,81,98,109,134,134,154,154,161,180,180,180,215,223,234 +Costa Rica,9.7489,-83.7534,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,5,9,9,13,22,23,26,27,35,41,50,69,89,117,134,158,177,201,231,263,295,314,330,347,375,396,416,435,454,467,483,502,539,558,577,595 +Cote d'Ivoire,7.54,-5.5471,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,5,6,9,9,14,14,25,73,80,96,101,101,165,168,179,190,194,218,245,261,323,349,384,444,444,533,574 +Croatia,45.1,15.2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,3,5,6,7,7,9,10,10,11,12,12,12,14,19,19,32,38,49,57,65,81,105,128,206,254,315,382,442,495,586,657,713,790,867,963,1011,1079,1126,1182,1222,1282,1343,1407,1495,1534,1600 +Cuba,22.0,-80.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,4,4,4,4,5,7,11,16,21,35,40,48,57,67,80,119,139,170,186,212,233,269,288,320,350,396,457,515,564,620,669 +Cyprus,35.1264,33.4299,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,3,6,6,14,26,26,33,46,49,67,67,84,95,116,124,132,146,162,179,214,230,262,320,356,396,426,446,465,494,526,564,595,616,633 +Czechia,49.8175,15.472999999999999,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,5,8,12,18,19,31,31,41,91,94,141,189,253,298,396,464,694,833,995,1120,1236,1394,1654,1925,2279,2631,2817,3001,3308,3508,3858,4091,4472,4587,4822,5017,5312,5569,5732,5831,5991 +Denmark,189.8634,-40.014300000000006,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,4,4,6,11,11,24,24,37,92,264,444,617,804,836,875,933,1025,1116,1225,1337,1420,1514,1572,1718,1862,2023,2200,2366,2564,2755,3039,3290,3573,3946,4269,4561,4875,5266,5597,5830,6014,6191,6369 +Diamond Princess,0.0,0.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,61,61,64,135,135,175,175,218,285,355,454,542,621,634,634,634,691,691,691,705,705,705,705,705,705,706,706,706,706,706,706,706,706,706,706,706,706,706,706,706,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712,712 +Djibouti,11.8251,42.5903,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,3,3,11,11,12,14,18,18,30,33,40,49,50,59,90,90,135,135,150,187,214 +Dominica,15.415,-61.371,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,2,7,11,11,11,11,11,12,12,12,12,14,14,15,15,15,15,16,16,16 +Dominican Republic,18.7357,-70.1627,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,2,2,5,5,5,5,5,5,11,11,11,21,21,34,72,112,202,245,312,392,488,581,719,859,901,1109,1284,1380,1488,1488,1745,1828,1956,2111,2349,2620,2759,2967 +Ecuador,-1.8312,-78.1834,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,6,7,10,13,13,13,14,15,15,17,17,17,28,28,37,58,111,199,367,506,789,981,1082,1173,1403,1595,1823,1924,1962,2240,2748,3163,3368,3465,3646,3747,3747,4450,4965,7161,7257,7466 +Egypt,26.0,30.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,3,15,15,49,55,59,60,67,80,109,110,150,196,196,256,285,294,327,366,402,456,495,536,576,609,656,710,779,865,985,1070,1173,1322,1450,1560,1699,1794,1939,2065 +El Salvador,13.7942,-88.8965,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,3,3,5,9,13,13,19,24,30,32,32,41,46,56,62,69,78,93,103,117,118,125 +Equatorial Guinea,1.5,10.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,4,6,6,6,6,9,9,9,12,12,12,12,12,12,15,15,16,16,16,16,16,18,18,18,18,21 +Eritrea,15.1794,39.7823,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,4,6,6,6,12,12,15,15,22,22,29,29,31,31,33,33,34,34,34 +Estonia,58.5953,25.0136,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,2,2,3,10,10,10,10,12,16,16,79,115,171,205,225,258,267,283,306,326,352,369,404,538,575,645,679,715,745,779,858,961,1039,1097,1108,1149,1185,1207,1258,1304,1309 +Eswatini,-26.5225,31.4659,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,4,4,4,4,6,9,9,9,9,9,9,9,9,9,9,10,10,12,12,12,12,14 +Ethiopia,9.145,40.4897,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,5,5,6,6,9,9,11,11,12,12,12,16,16,21,23,26,29,29,35,38,43,44,52,55,56,65,69,71 +Fiji,-17.7134,178.065,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,2,3,4,5,5,5,5,5,5,5,5,7,7,12,12,14,15,15,15,16,16,16 +Finland,64.0,26.0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,3,6,6,6,6,12,15,15,23,30,40,59,59,155,225,244,277,321,336,400,450,523,626,700,792,880,958,1041,1167,1240,1352,1418,1446,1518,1615,1882,1927,2176,2308,2487,2605,2769,2905,2974 +France,91.3624,59.71919999999999,0,0,2,3,3,3,4,5,5,5,6,6,6,6,6,6,6,11,11,11,11,11,11,11,12,12,12,12,12,12,12,12,12,12,14,18,38,57,100,130,191,204,288,380,656,959,1136,1219,1794,2293,2293,3681,4496,4532,6683,7715,9124,10970,12758,14463,16243,20123,22622,25600,29551,33402,38105,40708,45170,52827,57749,59929,65202,90848,93773,98963,110065,113959,118781,125931,130727,133670 +Gabon,-0.8037,11.6094,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,3,4,5,5,6,6,7,7,7,7,7,16,18,21,21,21,21,24,30,34,44,44,46,49 +Gambia,13.4432,-15.3101,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,2,3,3,3,3,3,4,4,4,4,4,4,4,4,4,4,4,4,4,9,9 +Georgia,42.3154,43.3569,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,3,3,3,3,4,4,4,13,15,15,24,24,25,30,33,33,34,38,40,43,49,54,61,70,75,79,83,90,91,103,110,117,134,155,162,174,188,196,211,218,234,242,257 +Germany,51.0,9.0,0,0,0,0,0,1,4,4,4,5,8,10,12,12,12,12,13,13,14,14,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,27,46,48,79,130,159,196,262,482,670,799,1040,1176,1457,1908,2078,3675,4585,5795,7272,9257,12327,15320,19848,22213,24873,29056,32986,37323,43938,50871,57695,62095,66885,71808,77872,84794,91159,96092,100123,103374,107663,113296,118181,122171,124908,127854 +Ghana,7.9465,-1.0232,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,6,6,7,7,11,16,19,23,27,53,93,132,137,141,152,152,161,195,204,205,205,214,214,287,313,378,378,408,566 +Greece,39.0742,21.8243,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,4,4,7,7,7,9,31,45,46,73,73,89,99,99,190,228,331,331,387,418,418,495,530,624,695,743,821,892,966,1061,1156,1212,1314,1415,1544,1613,1673,1735,1755,1832,1884,1955,2011,2081,2114 +Grenada,12.1165,-61.678999999999995,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,7,7,7,9,9,9,9,10,12,12,12,12,12,12,12,14,14,14 +Guatemala,15.7835,-90.2308,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,6,6,9,12,17,19,20,21,24,25,28,34,34,36,38,39,47,50,61,61,70,77,87,95,126,137,155 +Guinea,9.9456,-9.6966,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,2,2,4,4,4,4,8,8,16,22,22,30,52,73,111,121,128,144,164,194,212,250,250 +Guinea-Bissau,11.8037,-15.1804,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,2,2,8,8,9,9,15,18,18,18,33,33,36,36,38,38 +Guyana,5.0,-58.75,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,4,4,7,7,7,7,7,19,20,5,5,5,5,8,8,8,12,19,19,23,23,24,31,33,37,37,37,45,45 +Haiti,18.9712,-72.2852,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,6,7,8,8,8,8,15,15,15,16,16,18,20,21,24,25,27,30,31,33,33 +Holy See,41.9029,12.4534,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,4,4,4,4,6,6,6,6,6,7,7,7,7,7,7,8,8,8,8,8 +Honduras,15.2,-86.2419,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,2,3,6,8,9,12,24,24,26,30,30,36,52,68,95,110,139,141,172,219,222,264,268,298,305,312,343,382,392,393 +Hungary,47.1625,19.5033,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,4,7,9,9,13,13,19,30,32,39,50,58,73,85,103,131,167,187,226,261,300,343,408,447,492,525,585,623,678,733,744,817,895,980,1190,1310,1410 +Iceland,64.9631,-19.0208,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,6,11,26,34,43,50,50,58,69,85,103,134,156,171,180,220,250,330,409,473,568,588,648,737,802,890,963,1020,1086,1135,1220,1319,1364,1417,1486,1562,1586,1616,1648,1675,1689,1701 +India,21.0,78.0,0,0,0,0,0,0,0,0,1,1,1,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,5,5,28,30,31,34,39,43,56,62,73,82,102,113,119,142,156,194,244,330,396,499,536,657,727,887,987,1024,1251,1397,1998,2543,2567,3082,3588,4778,5311,5916,6725,7598,8446,9205 +Indonesia,-0.7893,113.9213,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,2,4,4,6,19,27,34,34,69,96,117,134,172,227,311,369,450,514,579,686,790,893,1046,1155,1285,1414,1528,1677,1790,1986,2092,2273,2491,2738,2956,3293,3512,3842,4241 +Iran,32.0,53.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,5,18,28,43,61,95,139,245,388,593,978,1501,2336,2922,3513,4747,5823,6566,7161,8042,9000,10075,11364,12729,13938,14991,16169,17361,18407,19644,20610,21638,23049,24811,27017,29406,32332,35408,38309,41495,44605,47593,50468,53183,55743,58226,60500,62589,64586,66220,68192,70029,71686 +Iraq,33.0,44.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,5,7,7,13,19,26,32,35,35,40,54,60,60,71,71,71,101,110,116,124,154,164,192,208,214,233,266,316,346,382,458,506,547,630,694,728,772,820,878,961,1031,1122,1202,1232,1279,1318,1352 +Ireland,53.1424,-7.6921,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,2,6,6,18,18,19,21,34,43,43,90,129,129,169,223,292,557,683,785,906,1125,1329,1564,1819,2121,2415,2615,2910,3235,3447,3849,4273,4604,4994,5364,5709,6074,6574,8089,8928,9655 +Israel,31.0,35.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,2,3,4,7,10,10,12,15,20,37,43,61,61,75,79,100,126,155,213,218,250,304,427,529,712,883,1071,1238,2369,2693,3035,3619,4247,4695,5358,6092,6857,7428,7851,8430,8904,9248,9404,9968,10408,10743,11145 +Italy,43.0,12.0,0,0,0,0,0,0,0,0,0,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,20,62,155,229,322,453,655,888,1128,1694,2036,2502,3089,3858,4636,5883,7375,9172,10149,12462,12462,17660,21157,24747,27980,31506,35713,41035,47021,53578,59138,63927,69176,74386,80589,86498,92472,97689,101739,105792,110574,115242,119827,124632,128948,132547,135586,139422,143626,147577,152271,156363 +Jamaica,18.1096,-77.2975,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,8,8,10,10,12,13,15,16,16,19,19,21,26,26,26,30,32,36,36,44,47,47,53,58,58,63,63,63,63,65,69 +Japan,36.0,138.0,2,2,2,2,4,4,7,7,11,15,20,20,20,22,22,22,25,25,26,26,26,28,28,29,43,59,66,74,84,94,105,122,147,159,170,189,214,228,241,256,274,293,331,360,420,461,502,511,581,639,639,701,773,839,839,878,889,924,963,1007,1101,1128,1193,1307,1387,1468,1693,1866,1866,1953,2178,2495,2617,3139,3139,3654,3906,4257,4667,5530,6005,6748 +Jordan,31.24,36.51,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,8,17,34,52,69,85,85,112,127,154,172,212,235,246,259,268,274,278,299,310,323,345,349,353,358,372,372,381,389 +Kazakhstan,48.0196,66.9237,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,6,9,10,33,35,44,49,53,60,62,72,81,111,150,228,284,302,343,380,435,464,531,584,662,697,727,781,812,865,951 +Kenya,-0.0236,37.9062,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,3,3,3,7,7,7,15,16,25,28,31,31,38,42,50,59,81,110,122,126,142,158,172,179,184,189,191,197 +"Korea, South",36.0,128.0,1,1,2,2,3,4,4,4,4,11,12,15,15,16,19,23,24,24,25,27,28,28,28,28,28,29,30,31,31,104,204,433,602,833,977,1261,1766,2337,3150,3736,4335,5186,5621,6088,6593,7041,7314,7478,7513,7755,7869,7979,8086,8162,8236,8320,8413,8565,8652,8799,8961,8961,9037,9137,9241,9332,9478,9583,9661,9786,9887,9976,10062,10156,10237,10284,10331,10384,10423,10450,10480,10512 +Kosovo,42.602636,20.902977,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,71,86,91,94,94,112,125,125,126,135,145,145,170,184,184,250,283,283 +Kuwait,29.5,47.75,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,11,26,43,45,45,45,56,56,56,58,58,61,64,64,69,72,80,80,104,112,123,130,142,148,159,176,188,189,191,195,208,225,235,255,266,289,317,342,417,479,556,665,743,855,910,993,1154,1234 +Kyrgyzstan,41.2044,74.7661,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,6,14,14,16,42,44,44,58,58,84,94,107,111,116,130,144,147,216,228,270,280,298,339,377 +Laos,19.856270000000002,102.495496,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,3,6,6,8,8,8,9,10,10,10,10,11,12,14,15,16,16,18,19 +Latvia,56.8796,24.6032,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,2,6,8,10,10,17,26,30,34,49,71,86,111,124,139,180,197,221,244,280,305,347,376,398,446,458,493,509,533,542,548,577,589,612,630,651 +Lebanon,33.8547,35.8623,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,2,2,2,4,10,13,13,13,16,22,22,32,32,41,61,61,77,93,110,110,120,133,157,163,187,248,267,318,333,368,391,412,438,446,470,479,494,508,520,527,541,548,576,582,609,619,630 +Liberia,6.4281,-9.4295,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,3,3,3,3,3,3,3,3,3,3,3,6,6,7,10,13,14,14,31,31,37,48,50 +Libya,26.3351,17.228331,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,3,8,8,10,10,11,11,18,18,19,20,21,24,24,24,25 +Liechtenstein,47.14,9.55,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,4,4,4,7,28,28,28,37,37,51,51,51,56,56,56,56,62,68,68,75,75,77,77,77,78,78,78,79,79,79 +Lithuania,55.1694,23.8813,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,3,3,6,8,12,17,25,27,36,49,83,143,179,209,274,299,358,394,460,491,537,581,649,696,771,811,843,880,912,955,999,1026,1053 +Luxembourg,49.8153,6.1296,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,2,2,3,3,5,7,19,34,51,59,77,140,203,335,484,670,798,875,1099,1333,1453,1605,1831,1950,1988,2178,2319,2487,2612,2729,2804,2843,2970,3034,3115,3223,3270,3281 +MS Zaandam,0.0,0.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,2,9,9,9,9,9,9,9,9,9,9,9,9 +Madagascar,-18.7669,46.8691,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,12,17,19,23,26,26,39,43,57,57,59,70,70,72,82,88,93,93,93,102,106 +Malawi,-13.254307999999998,34.301525,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,4,4,5,8,8,8,9,12,13 +Malaysia,2.5,112.5,0,0,0,3,4,4,4,7,8,8,8,8,8,10,12,12,12,16,16,18,18,18,19,19,22,22,22,22,22,22,22,22,22,22,22,22,23,23,25,29,29,36,50,50,83,93,99,117,129,149,149,197,238,428,566,673,790,900,1030,1183,1306,1518,1624,1796,2031,2161,2320,2470,2626,2766,2908,3116,3333,3483,3662,3793,3963,4119,4228,4346,4530,4683 +Maldives,3.2028,73.2207,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,6,8,8,9,10,13,13,13,13,13,13,13,13,13,13,13,13,16,16,17,17,18,19,19,19,19,19,19,19,19,19,19,19,20 +Mali,17.570692,-3.996166000000001,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,11,18,18,25,28,31,36,39,41,45,47,56,59,74,87,87,105 +Malta,35.9375,14.3754,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,5,6,6,12,18,21,30,38,38,53,64,73,90,107,110,129,134,139,149,151,156,169,188,196,202,213,227,241,293,299,337,350,370,378 +Mauritania,21.0079,10.9408,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,2,2,2,2,2,2,2,3,3,5,5,5,6,6,6,6,6,6,6,6,6,7,7,7,7 +Mauritius,-20.2,57.5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,12,14,28,36,42,48,81,94,102,107,128,143,161,169,186,196,227,244,268,273,314,318,319,324 +Mexico,23.6345,-102.5528,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,5,5,5,5,5,6,6,7,7,7,8,12,12,26,41,53,82,93,118,164,203,251,316,367,405,475,585,717,848,993,1094,1215,1378,1510,1688,1890,2143,2439,2785,3181,3441,3844,4219 +Moldova,47.4116,28.3699,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,3,3,6,12,23,23,30,30,49,66,80,94,109,125,149,177,199,231,263,298,353,423,505,591,752,864,965,1056,1174,1289,1438,1560,1662 +Monaco,43.7333,7.4167,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,7,7,7,7,11,11,23,23,23,31,33,42,42,46,49,52,55,60,64,66,73,77,79,81,84,90,92,93 +Mongolia,46.8625,103.8467,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,5,6,6,6,10,10,10,10,10,11,11,12,12,12,12,14,14,14,14,14,15,15,16,16,16,16,16 +Montenegro,42.5,19.3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,14,14,21,27,47,52,69,82,84,85,91,109,123,144,174,201,214,233,241,248,252,255,263,272 +Morocco,31.7917,-7.0926,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,2,2,2,2,2,3,5,6,7,17,28,29,38,49,63,77,96,115,143,170,225,275,345,402,479,556,617,654,708,791,919,1021,1120,1184,1275,1374,1448,1545,1661 +Mozambique,-18.665695,35.529562,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,5,7,7,8,8,8,8,10,10,10,10,10,10,10,17,17,20,20,21 +Namibia,-22.9576,18.4904,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,2,2,3,3,3,3,4,7,7,8,8,8,11,11,11,14,14,14,14,16,16,16,16,16,16,16,16 +Nepal,28.1667,84.25,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,3,3,4,5,5,5,5,5,6,6,9,9,9,9,9,9,9,9,12 +Netherlands,107.04169999999999,-265.0278,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,6,10,18,24,38,82,128,188,265,321,382,503,503,806,962,1138,1416,1711,2058,2467,3003,3640,4217,4764,5580,6438,7468,8647,9819,10930,11817,12667,13696,14788,15821,16727,17953,18926,19709,20682,21903,23249,24571,25746 +New Zealand,-40.9006,174.886,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,3,3,4,5,5,5,5,5,5,5,6,8,8,12,20,28,39,52,102,102,155,205,283,368,451,514,589,647,708,797,868,950,1039,1106,1160,1210,1239,1283,1312,1330 +Nicaragua,12.8654,-85.2072,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,2,2,2,2,4,4,4,5,5,5,5,5,6,6,6,6,7,7,8,9 +Niger,17.6078,8.0817,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,3,3,7,10,10,10,18,27,27,74,98,120,144,184,253,278,342,410,438,491,529 +Nigeria,9.082,8.6753,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,3,8,8,12,22,30,40,44,51,65,70,89,111,131,135,174,184,210,214,232,238,254,276,288,305,318,323 +North Macedonia,41.6086,21.7453,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,3,3,3,3,7,7,7,14,14,14,18,26,35,48,67,85,115,136,148,177,201,219,241,259,285,329,354,384,430,483,555,570,599,617,663,711,760,828 +Norway,60.472,8.4689,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,6,15,19,25,32,56,87,108,147,176,205,400,598,702,996,1090,1221,1333,1463,1550,1746,1914,2118,2385,2621,2863,3084,3369,3755,4015,4284,4445,4641,4863,5147,5370,5550,5687,5865,6086,6086,6211,6314,6409,6525 +Oman,21.0,57.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,4,4,4,6,6,6,12,15,16,16,16,16,16,18,18,18,19,19,22,22,24,39,48,48,52,55,66,84,99,109,131,152,167,179,192,210,231,252,277,298,331,371,419,457,484,546,599 +Pakistan,30.3753,69.3451,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,4,4,4,5,5,5,6,6,6,6,16,19,20,28,31,53,136,236,299,454,501,730,776,875,972,1063,1201,1373,1495,1597,1717,1938,2118,2421,2686,2818,3157,3766,4035,4263,4489,4695,5011,5230 +Panama,8.538,-80.7821,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,8,11,27,36,43,55,69,86,109,137,200,313,345,345,443,558,674,786,901,989,1181,1181,1317,1475,1673,1801,1988,2100,2249,2528,2752,2974,3234 +Papua New Guinea,-6.315,143.9555,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2 +Paraguay,-23.4425,-58.4438,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,5,5,6,6,6,8,9,11,11,13,18,22,22,27,37,41,52,56,59,64,65,69,77,92,96,104,113,115,119,124,129,133,134 +Peru,-9.19,-75.0152,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,6,7,11,11,15,28,38,43,86,117,145,234,234,318,363,395,416,480,580,635,671,852,950,1065,1323,1414,1595,1746,2281,2561,2954,4342,5256,5897,6848,7519 +Philippines,13.0,122.0,0,0,0,0,0,0,0,0,1,1,1,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,5,6,10,20,33,49,52,64,111,140,142,187,202,217,230,307,380,462,552,636,707,803,1075,1418,1546,2084,2311,2633,3018,3094,3246,3660,3764,3870,4076,4195,4428,4648 +Poland,51.9194,19.1451,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,5,5,11,16,22,31,49,68,103,119,177,238,251,355,425,536,634,749,901,1051,1221,1389,1638,1862,2055,2311,2554,2946,3383,3627,4102,4413,4848,5205,5575,5955,6356,6674 +Portugal,39.3999,-8.2245,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,5,8,13,20,30,30,41,59,59,112,169,245,331,448,448,785,1020,1280,1600,2060,2362,2995,3544,4268,5170,5962,6408,7443,8251,9034,9886,10524,11278,11730,12442,13141,13956,15472,15987,16585 +Qatar,25.3548,51.1839,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,3,7,8,8,8,8,15,18,24,262,262,320,337,401,439,439,452,460,470,481,494,501,526,537,549,562,590,634,693,781,835,949,1075,1325,1604,1832,2057,2210,2376,2512,2728,2979 +Romania,45.9432,24.9668,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,3,3,3,3,4,6,9,9,15,15,25,45,49,89,123,131,158,184,260,277,308,367,433,576,794,906,1029,1292,1452,1815,2109,2245,2460,2738,3183,3613,3864,4057,4417,4761,5202,5467,5990,6300 +Russia,60.0,90.0,0,0,0,0,0,0,0,0,0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,3,3,3,4,13,13,17,17,20,20,28,45,59,63,90,114,147,199,253,306,367,438,495,658,840,1036,1264,1534,1836,2337,2777,3548,4149,4731,5389,6343,7497,8672,10131,11917,13584,15770 +Rwanda,-1.9403,29.8739,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,5,7,8,8,17,17,19,36,40,41,50,54,60,70,70,75,82,84,89,102,104,105,105,110,110,118,120,126 +Saint Kitts and Nevis,17.357822,-62.782998,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,2,2,7,8,8,9,9,9,10,10,11,11,11,12,12,12 +Saint Lucia,13.9094,-60.9789,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,2,2,2,2,2,2,2,3,3,3,3,3,3,9,9,13,13,13,13,14,14,14,14,14,14,15,15,15 +Saint Vincent and the Grenadines,12.9843,-61.2872,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,3,7,7,7,8,8,12,12,12,12 +San Marino,43.9424,12.4578,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,8,10,16,21,21,23,36,36,51,62,69,80,80,101,109,109,119,119,144,144,175,187,187,208,208,223,224,224,230,236,236,245,245,259,266,266,279,279,333,344,356,356 +Sao Tome and Principe,0.18636,6.613081,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,4,4,4,4,4 +Saudi Arabia,24.0,45.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,5,5,5,11,15,20,21,45,86,103,103,118,171,171,274,344,392,511,562,767,900,1012,1104,1203,1299,1453,1563,1720,1885,2039,2179,2402,2605,2795,2932,3287,3651,4033,4462 +Senegal,14.4974,-14.4524,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,4,4,4,4,4,4,4,4,4,10,10,24,24,26,31,31,38,47,67,79,86,99,105,119,130,142,162,175,190,195,207,219,222,226,237,244,250,265,278,280 +Serbia,44.0165,21.0059,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,5,12,19,35,46,48,55,65,83,103,135,171,222,249,303,384,384,457,659,741,785,900,1060,1171,1476,1624,1908,2200,2447,2666,2867,3105,3380,3630 +Seychelles,-4.6796,55.492,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,3,4,4,6,7,7,7,7,7,7,7,7,8,8,8,10,10,10,10,10,10,11,11,11,11,11,11,11 +Sierra Leone,8.460555000000001,-11.779889,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,2,2,4,6,6,6,7,7,8,8,10 +Singapore,1.2833,103.8333,0,1,3,3,4,5,7,7,10,13,16,18,18,24,28,28,30,33,40,45,47,50,58,67,72,75,77,81,84,84,85,85,89,89,91,93,93,93,102,106,108,110,110,117,130,138,150,150,160,178,178,200,212,226,243,266,313,345,385,432,455,509,558,631,683,732,802,844,879,926,1000,1049,1114,1189,1309,1375,1481,1623,1910,2108,2299,2532 +Slovakia,48.669,19.699,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,3,7,10,16,32,44,54,63,72,105,123,137,178,185,186,204,216,226,269,292,314,336,363,400,426,450,471,485,534,581,682,701,715,728,742 +Slovenia,46.1512,14.9955,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,7,7,16,16,31,57,89,141,181,219,253,275,275,286,341,383,414,442,480,528,562,632,684,730,756,802,841,897,934,977,997,1021,1059,1091,1124,1160,1188,1205 +Somalia,5.1521,46.1996,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,2,3,3,3,3,5,5,5,7,7,7,7,8,12,12,21,21,25 +South Africa,-30.5595,22.9375,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,3,3,7,13,17,24,38,51,62,62,116,150,202,240,274,402,554,709,927,1170,1187,1280,1326,1353,1380,1462,1505,1585,1655,1686,1749,1845,1934,2003,2028,2173 +South Sudan,6.8770000000000024,31.307,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,3,4,4,4 +Spain,40.0,-4.0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,6,13,15,32,45,84,120,165,222,259,400,500,673,1073,1695,2277,2277,5232,6391,7798,9942,11748,13910,17963,20410,25374,28768,35136,39885,49515,57786,65719,73235,80110,87956,95923,104118,112065,119199,126168,131646,136675,141942,148220,153222,158273,163027,166831 +Sri Lanka,7.0,81.0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,6,10,18,28,44,51,60,73,77,82,97,102,102,106,106,113,117,122,143,146,151,159,166,176,178,185,189,190,190,198,210 +Sudan,12.8628,30.2176,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,2,2,2,2,2,2,3,3,3,3,5,6,6,7,7,8,10,10,12,12,14,14,15,17,19,19 +Suriname,3.9193,-56.0278,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,4,4,5,5,7,8,8,8,8,8,8,9,10,10,10,10,10,10,10,10,10,10,10,10 +Sweden,63.0,16.0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,7,7,12,14,15,21,35,94,101,161,203,248,355,500,599,814,961,1022,1103,1190,1279,1439,1639,1763,1934,2046,2286,2526,2840,3069,3447,3700,4028,4435,4947,5568,6131,6443,6830,7206,7693,8419,9141,9685,10151,10483 +Switzerland,46.8182,8.2275,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,8,8,18,27,42,56,90,114,214,268,337,374,491,652,652,1139,1359,2200,2200,2700,3028,4075,5294,6575,7474,8795,9877,10897,11811,12928,14076,14829,15922,16605,17768,18827,19606,20505,21100,21657,22253,23280,24051,24551,25107,25415 +Syria,34.802075,38.99681500000001,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,5,5,5,5,9,10,10,10,16,16,16,19,19,19,19,19,19,25,25 +Taiwan*,23.7,121.0,1,1,3,3,4,5,8,8,9,10,10,10,10,11,11,16,16,17,18,18,18,18,18,18,18,20,22,22,23,24,26,26,28,30,31,32,32,34,39,40,41,42,42,44,45,45,45,45,47,48,49,50,53,59,67,77,100,108,135,153,169,195,215,235,252,267,283,298,306,322,329,339,348,355,363,373,376,379,380,382,385,388 +Tanzania,-6.369,34.8888,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,3,6,6,6,12,12,12,12,13,13,14,14,19,19,20,20,20,20,22,24,24,25,25,32,32,32 +Thailand,15.0,101.0,2,3,5,7,8,8,14,14,14,19,19,19,19,25,25,25,25,32,32,32,33,33,33,33,33,34,35,35,35,35,35,35,35,35,37,40,40,41,42,42,43,43,43,47,48,50,50,50,53,59,70,75,82,114,147,177,212,272,322,411,599,721,827,934,1045,1136,1245,1388,1524,1651,1771,1875,1978,2067,2169,2220,2258,2369,2423,2473,2518,2551 +Timor-Leste,-8.874217,125.727539,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2 +Togo,8.6195,0.8248,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,9,16,16,18,20,23,23,25,25,25,30,34,36,39,40,41,44,58,65,70,73,76,76,76 +Trinidad and Tobago,10.6918,-61.2225,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,4,5,7,9,9,49,50,51,57,60,65,66,74,78,82,87,90,94,98,103,104,105,107,107,109,109,112,113 +Tunisia,34.0,9.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,2,2,5,7,7,16,18,18,20,24,29,39,54,60,75,89,114,173,197,227,278,312,312,394,423,455,495,553,574,596,623,628,643,671,685,707 +Turkey,38.9637,35.2433,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,5,5,6,18,47,98,192,359,670,1236,1529,1872,2433,3629,5698,7402,9217,10827,13531,15679,18135,20921,23934,27069,30217,34109,38226,42282,47029,52167,56956 +US,37.0902,-95.7129,1,1,2,2,5,5,5,5,5,7,8,8,11,11,11,11,11,11,11,11,12,12,13,13,13,13,13,13,13,13,15,15,15,51,51,57,58,60,68,74,98,118,149,217,262,402,518,583,959,1281,1663,2179,2727,3499,4632,6421,7783,13747,19273,25600,33276,43847,53740,65778,83836,101657,121465,140909,161831,188172,213372,243762,275586,308853,337072,366667,396223,429052,461437,496535,526396,555313 +Uganda,1.0,32.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,9,9,14,14,23,30,33,33,44,44,45,48,48,52,52,52,53,53,53,53,54 +Ukraine,48.3794,31.1656,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,3,3,3,7,14,14,16,29,47,73,73,97,145,196,310,356,475,548,645,794,897,1072,1225,1308,1319,1462,1668,1892,2203,2511,2777 +United Arab Emirates,24.0,54.0,0,0,0,0,0,0,0,4,4,4,4,5,5,5,5,5,5,7,7,8,8,8,8,8,8,9,9,9,9,9,9,13,13,13,13,13,13,19,21,21,21,27,27,29,29,45,45,45,74,74,85,85,85,98,98,98,113,140,140,153,153,198,248,333,333,405,468,570,611,664,814,1024,1264,1505,1799,2076,2359,2659,2990,3360,3736,4123 +United Kingdom,270.0299,-482.92470000000003,0,0,0,0,0,0,0,0,0,2,2,2,2,2,2,2,3,3,3,8,8,9,9,9,9,9,9,9,9,9,9,9,9,13,13,13,15,20,23,36,40,51,86,116,164,207,274,322,384,459,459,802,1144,1145,1551,1960,2642,2716,4014,5067,5745,6726,8164,9640,11812,14745,17312,19780,22453,25481,29865,34173,38689,42477,48436,52279,55949,61474,65872,74605,79874,85206 +Uruguay,-32.5228,-55.7658,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,8,29,50,79,94,110,158,162,162,189,217,238,274,304,310,338,338,350,369,400,400,406,424,424,456,473,494,480 +Uzbekistan,41.3775,64.5853,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,10,15,23,33,43,43,46,50,60,75,88,104,144,149,172,181,205,227,266,342,457,520,545,582,624,767,865 +Venezuela,6.4238,-66.5897,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,10,17,33,36,42,42,70,70,77,84,91,107,107,119,119,135,135,143,146,153,155,159,165,165,167,171,171,175,181 +Vietnam,16.0,108.0,0,2,2,2,2,2,2,2,2,2,6,6,8,8,8,10,10,13,13,14,15,15,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,18,30,30,31,38,39,47,53,56,61,66,75,85,91,94,113,123,134,141,153,163,174,188,203,212,218,233,237,240,241,245,249,251,255,257,258,262 +West Bank and Gaza,31.9522,35.2332,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,7,16,16,19,26,30,30,31,35,38,38,39,41,44,47,48,52,59,59,59,84,91,98,109,116,119,134,161,194,217,237,254,261,263,263,267,268,290 +Western Sahara,24.2155,-12.8858,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,4,4,4,4,4,4,6 +Yemen,15.552726999999999,48.516388,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1 +Zambia,-15.4167,28.2833,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,2,3,3,3,12,16,22,28,29,35,35,36,39,39,39,39,39,39,39,39,40,40,43 +Zimbabwe,-20.0,30.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,3,3,3,3,3,5,7,7,7,8,8,9,9,9,9,10,11,11,11,13,14,14 diff --git a/Project/NCTU_DL/HW2/requirements.txt b/Project/NCTU_DL/HW2/requirements.txt new file mode 100644 index 0000000..cb459cc --- /dev/null +++ b/Project/NCTU_DL/HW2/requirements.txt @@ -0,0 +1,5 @@ +sklearn +numpy +matplotlib +pygal +pytorch diff --git a/Project/TFjs/README.md b/Project/TFjs/README.md new file mode 100644 index 0000000..51c1912 --- /dev/null +++ b/Project/TFjs/README.md @@ -0,0 +1,5 @@ +# My First Tensorflow.js + +https://github.com/tensorflow/tfjs-models +https://js.tensorflow.org/ +https://www.youtube.com/playlist?list=PLRqwX-V7Uu6YIeVA3dNxbR9PYj4wV31oQ diff --git a/reminder.txt b/reminder.txt new file mode 100644 index 0000000..c22016e --- /dev/null +++ b/reminder.txt @@ -0,0 +1,47 @@ +Data Smoothing: N-GramModel.md + DataSmoothing + +word representation (語意 前三個ppt) + word embedding (Lect8) + +CNN + FCNN + Layers + +NLP總整理 有些以前的筆記實在不堪入目 (有一些筆記在MLPractice那邊 如HMM、CRF 可能要link一下) +NLPAP的注意力與Transformer的PPT + +可能把所有常見task都整理一下 可用的模型 這樣 + +Machine Translation Notes + +Specify the TensorFlow version (1.x or 2.x) + +Maybe implement computational graph: +* each node type: + * compute its output + * compute its gradient wrt. its inputs +* topological sort (order) for forwardprop and reverse order for backprop + +模型壓縮 +蒸餾模型 Distilled Model +https://zhuanlan.zhihu.com/p/67871864 +https://zhuanlan.zhihu.com/p/24337627 +Paper: Distilling the Knowledge in an Neural Network +https://zhuanlan.zhihu.com/p/39945855 + +Links to each notes may changed + + + + + +Machine Reading Comprehension (MRC) vs. sequence labeling + + +Remove NLPTransferLearning and put into README + + + +Evaluation Metrics <==> Tasks + + + + +[AMiner - Exploring the Motive Force of Scientific and T](https://aminer.org/) \ No newline at end of file