Misc.txt

// Notes on all other technical stuff I learn.


@@@@@@@@@@@@@@@@@@@@@      PROGRAMMING       @@@@@@@@@@@@@@@@@@@@@@@@@@

Rob Pike's 5 Rules of Programming

Rule 1. You can't tell where a program is going to spend its time. Bottlenecks
        occur in surprising places, so don't try to second guess and put in a
        speed hack until you've proven that's where the bottleneck is.

Rule 2. Measure. Don't tune for speed until you've measured, and even then
        don't unless one part of the code overwhelms the rest.

Rule 3. Fancy algorithms are slow when n is small, and n is usually small.
        Fancy algorithms have big constants. Until you know that n is
        frequently going to be big, don't get fancy. (Even if n does get big,
        use Rule 2 first.)

Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder
        to implement. Use simple algorithms as well as simple data structures.

Rule 5. Data dominates. If you've chosen the right data structures and
        organized things well, the algorithms will almost always be
        self-evident. Data structures, not algorithms, are central to
        programming.

Pike's rules 1 and 2 restate Tony Hoare's famous maxim "Premature optimization
is the root of all evil."

Ken Thompson rephrased Pike's rules 3 and 4 as "When in doubt, use brute
force.".

Rules 3 and 4 are instances of the design philosophy KISS.

Rule 5 was previously stated by Fred Brooks in The Mythical Man-Month. Rule 5 is
often shortened to "write stupid code that uses smart objects".

Reference:
http://users.ece.utexas.edu/~adnan/pike.html

@@@@@@@@@@   RIPGREP  @@@@@@@@

rg –help | more		Make help useful on Windows
rg -l NEEDLE		List matching files only
rg -c NEEDLE		List matching files, including a count
rg -i NEEDLE		Search case-insensitively
rg --no-filename NEEDLE	Don’t print filenames, handy when you care about the match more than the file
rg -v NEEDLE		Invert matching: show lines that do not match
rg NEEDLE README.md	Search only in specified file(s)
rg -c ‐‐sort path|modified|accessed|created NEEDLE      Sort the results (-sortr to reverse)

rg -g '*.nuspec' NEEDLE     Only search in *.nuspec files (can use multiple -g)
rg -g '!*.nuspec' NEEDLE    Search in everything but *.nuspec files
rg -g 'folder/*' NEEDLE     Search in `folder`
rg -g '!folder/*' NEEDLE    Search everywhere except in `folder`

rg -p NEEDLE | less -R      Force pretty printed output even in pipes
rg -e NEEDLE1 -e NEEDLE2    Search for multiple patterns
rg -z NEEDLE		Search in gzip, bzip2, xz, LZ4, LZMA, Brotli and Zstd compressed files
rg --type-list		Displays built-in available types and their corresponding globs
rg -tcs -tconfig	Search in file types cs and config
rg -Tconfig             Don’t search in file type config


rg image utils.py		Search in a single file utils.py
rg image src/			Search in dir src/ recursively
rg image			Search image in current dir recursively
rg '^We' test.txt		Regex searching support (lines starting with We)
rg -i image			Search image and ignore case (case-insensitive search)
rg -s image			Smart case search
rg -F '(test)'			Search literally, i.e., without using regular expression
rg image -g '*.py'		File globing (search in certain files), can be used multiple times
rg image -g '!*.py'		Negative file globing (do not search in certain files)
rg image --type py or rg image -tpy     Search image in Python file

rg image -Tpy			Do not search image in Python file type
rg -l image				Only show files containing image (Do not show the lines)
rg --files-without-match image
                        Show files not containing image
rg -v image				Inverse search (search files not containing image)
rg -w image				Search complete word
rg --count				Show the number of matching lines in a file
rg --count-matches		Show the number of matchings in a file
rg neovim --stats		Show the searching stat (how many matches, how many files searched etc.)


Reference:
[1] https://www.philipdaniels.com/blog/2019/ripgrep-cheatsheet/
[2] https://jdhao.github.io/2020/02/16/ripgrep_cheat_sheet/

@@@@@@@@@@   CFFI  @@@@@@@@@@@
- C Foreign Function Interface.

https://av.tib.eu/media/21110

@@@@@@@@@@   Homebrew Commands    @@@@@@@@@@
# Install nightly
brew install --HEAD <package>


@@@@@@@@@@   XML  @@@@@@@@@@@
Various Nodes and their meaning
---
<union> - Collection of other nodes such as sequences, elements etc. Handlers
cannot be on this node.

<sequence> - 0 or more collection of nodes, such as element nodes. Handlers are
on this node.

<element> - Can be optional. Must provide a value after this. This is leaf node
in XML.

<choice>
<validator>
<attr-req>


@@@@@@@@@@   VLC Media Player Shortcut Keys  @@@@@@@@@@@

Jumps:
Very short backwards jump (3s): Shift + Left (Ctrl + Cmd + Left)
Very short forward jump (3s): Shift + Right (Ctrl + Cmd + Right)
Short backwards jump (10s): Alt + Left (or just Left)
Short forward jump (10s): Alt + Right (or just Right)
Medium backwards jump (1m): Ctrl + Left (Shift + Cmd + Left)
Medium forward jump (1m): Ctrl + Right (Shift + Cmd + Right)

Speed:
[+]/-/= Faster, slower, normal

[ or – : Decrease playback speed. [ decreases it by less, and – decreases it by
more.

] : Increase playback speed

= : Return to the default playback speed

@@@@@@@@@@@@@@@@@@@   QEMU    NOTES      @@@@@@@@@@@@@@@@@@

// Basic qemu tutorial
https://fosspost.org/tutorials/use-qemu-test-operating-systems-distributions

// On Windows, download Intel's Hardware Acceleration (HAX) technology
www.qemu.org/2017/11/22/haxm-usage-windows/

// pre-built qcow2 images
https://people.debian.org/~aurel32/qemu/amd64/

// To start an image
qemu-system-x86_64 -drive format=raw,file=debian-8.2.0-amd64-DVD-1.iso

For me, following steps worked:
1. Download qemu
2. Download HAX
3. Run above command


@@@@@@@@@@@@@@@@@@@   NODEJS  NOTES      @@@@@@@@@@@@@@@@@@

Reading/Learning NodeJS
=====
- Use of 'var' and 'const'
- global object, modules
- Module wrapper function:
    * All code is wrapped inside this wrapper
        (function (exports, require, module, __filename__, __dirname__) {
            /* your code */
        })

- Template strings (Javascript concept - ES6)

Events and Signals
==================
- Look at this code:
    const EE = require('events');   // events class.
    const em = new EE();
    * here EE is a CLASS and not an object of a class, a method or a variable
    * therefore, em is an instance of EE() class
    em.emit("signalEvent", {id: 1, url: 'http://'});

- A listener must listen to above event
    em.on('signalEvent', function(eventArg) {
        console.log('signalEvent invoked. Listener called', eventArg);
    })

- Arrow functions
    em.on('signalEvent', (eventArg) => {
        console.log('signalEvent invoked. Listener called', eventArg);
    })


Reference:
[1] https://www.youtube.com/watch?v=TlB_eWDSMt4

@@@@@@@@@@@@@@@@@@@@   GCP SDK Commands Cheat Sheet   @@@@@@@@@@@@@@@@@@@@@@@@
Deploy your app
	$ gcloud app deploy

List gcloud configurations
	$ gcloud list

You can stream logs from the command line by running
	$ gcloud app logs tail -s default

To view your application in the web browser run
	$ gcloud app browse

gcloud functions deploy  helloGCSGeneric --runtime nodejs6 --trigger-resource gcs-cf-nodejs --trigger-event google.storage.object.finalize



[1] https://gist.github.com/pydevops/cffbd3c694d599c6ca18342d3625af97
[2] https://github.com/gregsramblings/google-cloud-4-words

@@@@@@@@@@@@@@   GCP Functions Emulator Commands Cheat Sheet   @@@@@@@@@@@@@@@@@
// prefix below commands with `gcloud` and you can do everything with gcloud SDK
// instead of function emulators

$ functions --help
$ functions <command> --help

$ functions config set projectId PROJECT_ID
$ functions start

$ functions deploy helloGET --trigger-http
$ functions call helloGET
$ functions call helloGreeting --data '{"name":"Local Emulator"}'

$ functions debug <function-name>
$ functions inspect <function-name>

$ functions delete FUNCTION_NAME
$ functions reset FUNCTION_NAME

$ functions list
$ functions logs read
$ functions logs clear
$ functions status  << status of log file

$ functions stop
$ functions kill

@@@@@@@@@@@@@@@@@@@   RabbitMQ  vs  Apache Kafka  @@@@@@@@@@@@@@@@@@

Origins:
========
RMQ is a message broker and first mature open source implementation of AMQP.
It's written in Erlang.  AMQP is an "Open Wire Protocol" for messaging with
powerful routing features. While language specific message brokers existed, AMQP
was first cross-language protocol and RMQ its most mature implementation.

Kafka came from LinkedIn, written in Scala and solved the problem to connect
different internal systems. Adopted by Apache SF and useful in event-driven
architecture.


Architecture & Design
=====================
RMQ is a general purpose message broker. A broker in RMQ comprises of two
components: Exchange and Queues
- Exchange takes messages from Producer and maps it to Queues
- Consumer reads messages from Queues

This way, Producer and Consumer are decoupled. Messaging can be synchronous or
asynchronous as required. RMQ uses smart broker/dumb consumer model.


                                      +---------+
                                +---->| Queue   |---------> Consumer
                                |     +---------+
                                |
                                |C
                +-----------+   |
            A   |           |   |     +----------+
Producer -------> Exchange  |-------->| Queue    |--------> Consumer
                |    B      |   |     +----------+
                +-----------+   |
                                |
                                |     +----------+
                                +---->| Queue    |--------> Consumer
                                      +----------+

            A = Message + Routing Key
            B = Exchange Types (4 types)
            C = Binding + Binding Key. Binding is mapping Exchange with Queue


There are 4 types of exchanges: Direct, Topic, Fanout, Headers

There's also a default (nameless) exchange where routing key is compared with
queue name


Exchange uses "routing keys" to "bind" queues to message. A routing key is
generated for a message and used to determine which queue(s) it must be
forwarded to

Direct Exchange: 1:1 mapping between message and queue.
        Routing Key == Binding Key

Fanout: Every message is routed to ALL queues
Topic: Routing Key is partially matched to Binding Key. Not clear how it's done.
Header: There's no routing key. Message header is used in Queue matching

Messages are removed from the queue after Consumer reads it.

RMQ can be setup on multi-node cluster and make it distributed deployment (HA
etc).



Apache Kafka is not a message broker but more like a message "store" or message
"log". It is fundamentally different in implementation (distributed). Kafka uses
Zookeeper to implement and manage all distributed concepts (leader election,
config management, replication state management etc)

Kafka uses dumb broker/smart consumer model in the sense that consumer is
responsible for maintaining message offsets.

Topics and Partitions:
---
Kafka uses topics and partitions within the topics that acts similar to RMQ
queue. A topic can have many partitions. Ordering of messages within a partition
is FIFO.

Consumer Groups:
---
Kafka has concept of Consumer Groups, where a bunch of Consumer nodes are
grouped together (duh!). This is used to expand and reduce based on message flow
in partitions. There is 1:1 mapping of partition and consumer node. There cannot
be two consumer nodes from same consumer group reading from same partition.
Offset maintenance will be a problem. Offsets per partition are maintained per
consumer group.


Requirements and Use Cases
==========================
TBD

Security
========
TBD

Developer Experience
====================
TBD

Performance
===========
TBD

Making the Call
===============
TBD


Reference:
[1]
https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka
[2] https://www.opsclarity.com/understanding-kafka-consumer-lag/

@@@@@@@@@@@@@@@@@@@   SOAP     vs    REST         @@@@@@@@@@@@@@@@@@
SOAP = Simple Object Access Protocol
REST = Representational State Transfer

- Both are communication protocol for web services

- REST is a stateless protocol and operates through interfaces called resources
  (or APIs).
- SOAP exposes application logic as services rather than data.


Benefits of REST over SOAP:
- REST uses HTTP to communicate with APIs. Much simpler protocol than SOAP's
  messaging protocols
- REST allows variety of data formats (JSON, YAML). SOAP only uses XML
- REST offers better support for browser clients and has superior performance
- Uses less bandwidth


Benefits of SOAP over REST
-

Reference:
[1] https://stackify.com/soap-vs-rest/

@@@@@@@@@@@@@@@@@@@   ELASTIC    SEARCH   NOTES   @@@@@@@@@@@@@@@@@@
- ES is a document datastore. Documents are stored in JSON format.
- ES uses indices to track documents. We say "Document is added to index XXX"
- Documents also have a type.

Example: Adding a document to ES
----

ADD DOCUMENT TO ES:

POST localhost:9200/accounts/person/1
{
    "name" : "John",
    "lastname" : "Doe",
    "job_description" : "Systems administrator and Linux specialit"
}

index = 'accounts'
document type = 'person'
id = 1
Content of the document is the JSON

Return to above HTTP POST will confirm document creation, as follows:

{
    "_index": "accounts",
    "_type": "person",
    "_id": "1",
    "_version": 1,
    "result": "created",
    "_shards": {
        total": 2,
        "successful": 1,
        "failed": 0
    },
    "created": true
}


- Let's get the same document back



READ DOCUMENT FROM ES:

GET localhost:9200/accounts/person/1

Response:
---
{
    "_index": "accounts",
    "_type": "person",
    "_id": "1",
    "_version": 1,
    "found": true,
    "_source": {
        "name": "John",
        "lastname": "Doe",
        "job_description": "Systems administrator and Linux specialit"
    }
}

- Look at "_source" for your document

UPDATE THE DOCUMENT:

POST localhost:9200/accounts/person/1/_update
{
      "doc":{
       "job_description" : "Systems administrator and Linux specialist"
     }
}

- Notice that 'specialit' was incorrectly spelled. That is updated now.


SEARCH DOCUMENTS:

GET localhost:9200/_search?q=john




References:
[1] https://www.elastic.co/blog/a-practical-introduction-to-elasticsearch


@@@@@@@@@@@@@@@@@@@   UTF-8  INTRODUCTION     @@@@@@@@@@@@@@@@@@@@@@
- Variable sized encoding. Size range from 1 to 4 bytes

How are characters encoded in UTF-8?
- Manipulating numbers at binary level
- High order bits of first byte are important. They tell how many bytes are
  used to encode a value

- In 1 byte encoding, which range from 0 to 127, high order bit will always be
  zero

        +---------------------------+
        | 0 |                       |
        +---------------------------+

- In 2 byte encoding, first 3 high order bits of first byte is 110 and first 2
  high order bits of second byte is 10

    +---------------------------+
    | 1 | 1 | 0 |               |
    +---------------------------+
    +---------------------------+
    | 1 | 0 |                   |
    +---------------------------+

- This gives 11 (5+6) bits to encode the char

- In 3 byte encoding, we have 1110, 10 and 10 in higher order of each byte

    +---------------------------+
    | 1 | 1 | 1 | 0 |           |
    +---------------------------+
    +---------------------------+
    | 1 | 0 |                   |
    +---------------------------+
    +---------------------------+
    | 1 | 0 |                   |
    +---------------------------+

- This gives 16 bits (4+6+6) to work with

- 4 byte encoding follows same pattern as 3 byte encoding

    +---------------------------+
    | 1 | 1 | 1 | 0 |           |
    +---------------------------+
    +---------------------------+
    | 1 | 0 |                   |
    +---------------------------+
    +---------------------------+
    | 1 | 0 |                   |
    +---------------------------+
    +---------------------------+
    | 1 | 0 |                   |
    +---------------------------+

- Total of 21 chars to work with


Reference
https://www.youtube.com/watch?v=sqPTR_v4qFA


@@@@@@@@@@@@@@@   CPU    CACHES      @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- Three common types of caches
    * Data Cache (D-cache, aka D$)
    * Instruction (I-cache, aka I$)
    * Translation Lookaside Buffer (TLB)
        - Caches virtual addr to real addr

- Cache Hierarchies (per core)
    * L1 I-cache, L1 D-cache
    * L2 cache
        - Holds both instructions and data
    * L3 cache
        - Holds both instructions and data

- Assume 100MB program at runtime (code+data)
    * 8% fits in L3 cache of Intel Core i79xx
    * 0.25% fits in each L2 cache
    * 0.03% fits in each L1 cache

- Cache speeds (for Core i79xx)
    * L1 latency is 4 cpu cycles
    * L2 latency is 11 cpu cycles
    * L3 latency is 39 cpu cycles
    * Main Memory (RAM) is 107 cpu cycles

Cache Lines
---
- Caches consists of "lines", each holding multiple adjacent words
- On Core i7, cache lines hold 64 bytes (common for Intel/AMD processors)
    64 bytes = 16 32-bit values, 8 64-bit values, etc
        Eg., 16 32-bit array elements

- Main memory read/written in terms of cache lines
    * Reading a byte that's not in cache (cache miss) will read a full cache
      line from main memory
    * Writing a byte will write a full cache line to main memory (eventually)



Reference:
https://www.youtube.com/watch?v=WDIkqP4JbkE

@@@@@@@@@@@@     YAML   SYNTAX   RULES       @@@@@@@@@@@@@@@@@@@@@@@@

RULE 0: Indentation
- YAML uses fixed indentation scheme to represent relationship between data
  layers

RULE 1: Scalars
- Scalars are ordinary values: numbers, strings, booleans

Ex:
number-value: 42
floating-point-value: 3.141592
boolean-value: true
# strings can be both 'single-quoted` and "double-quoted"
string-value: 'Bonjour'



RULE 2: Colons (or Dictionaries)
- YAML uses dictionaries represented as key:value pairs, similar to Python
  dictionaries

Ex:
[1]
my_key: my_value

[2] Uncommon, but this is valid
my_key:
  my_value


- Dictionaries can be nested
Ex:
first_level_dict_key:
  second_level_dict_key: value_in_second_level_dict

- There's a space following colon and then value (this is mandatory)


RULE 3: Dashes (or Lists)
- To represent list of items, single dash followed by a space is used.
- Multiple items belong to same list if indentation level is same.

Ex:
- list_value_one
- list_value_two
- list_value_three


- Value of a dictionary can be list

Ex:

my_dictionary:
  - list_value_one
  - list_value_two
  - list_value_three


YAML Mutli Documents
---
- YAML allows multiple docs to be embedded in a single file. They only have to
  be separated using triple-dash separator (---)

Reference:
https://learnxinyminutes.com/docs/yaml/  <<<< This is very good, quick intro
https://docs.saltstack.com/en/latest/topics/yaml/
https://gettaurus.org/docs/YAMLTutorial/


@@@@@@@@@@@@@@@@@@@@@@@@@     TOR    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Tor:
- Passing traffic by bouncing them across various Tor nodes across the globe.
  Minimum, Tor traffic passes through 3 nodes: Entry/Gaurd, Relay & Exit.

- Traffic is encrypted in layers (like Onion) before it is sent out from source.
    # of layers of encryption = # of Tor nodes traffic is passing.

- As traffic passes through each node, that node knows how to decrypt it's layer
  and forward to next Tor node.


@@@@@@@@    MUNICIPAL SOLID WASTE MANAGEMENT IN DEVELOPING COUNTRIES @@@@@@@@

Facts:
- Humans generate 7-10 billion tons of waste per year worldwide.
- Waste from 2 billion people is not collected from their homes and
  neighborhoods.
- By end of century, 83% of world population will be in Asia and Africa.
- There are 3 billion people who lack access to controlled disposal facilities.
- By 2030 there will be 40 megacities (cities with 10 million or more
  population). Of that 32 will be in Asia and Africa.


Municipal Waste Generation and Characterization:
- Estimating Lower Calorific Value (LCV)
    LCV = 40(a+b+c+d) + 90e - 46W
    LCV in kcal/kg
    a = Paper
    b = Textiles
    c = Wood and leaves
    d = Food waste
    e = Plastic and rubber
    W = Water
    all of above in % of wet weight.

- Incineration without adding fuel requies LCV > 1000 kcal/kg
- Incineration with energy recovery requires LCV > 1500-1650 kcal/kg


[1]
https://www.coursera.org/learn/solid-waste-management/home/welcome

@@@@@@@@@@@@@   Open/R  @@@@@@@@@@@@@@

- Build a routing _platform_ and not a routing _protocol_
- Applications are built on top of this platform in C++. Routing is one
  application.
- Each node in this network as following 4 components:
    * Decision
    * FIB
    * KV-Store
    * Link Monitor
- Zero MQ for message passing between nodes.
- Thrift: Message encoding. Populat open source library.

- Key-Value Store:
    - Key = Strings, Value = Thrift objects

Optimization in KV Store:
- Group nodes in to clusters
- STP within clusters
- Between clusters, similar to BGP's path tracing mechanism is used.

Scaling in KV Store:
-

Neighbor Discovery:
- Runs as a separate process.
- Works on link-level




[1]
https://code.facebook.com/posts/1036362693099725/networking-scale-may-2016-recap/

[2]
https://code.facebook.com/posts/1142111519143652/introducing-open-r-a-new-modular-routing-platform/


@@@@@@@@@@@@@   Networks Illustrated: Principles w/o Calculus @@@@@@@@@@@@@@

Lesson 1:
---
- 8 principles in this course:
- Sharing (network medium) is hard
- Consensus is hard
    * Referral system.
    * Auction system with systems bidding.
    * Voting system.
- Crowds are wise
    * Amazon trying to solve Rating problem.
    * Netflix trying to solve Recommendation problem.

- Crowds are NOT wise:
    * Information cascade can happen and crowds may not think for themselves.

- Network is Expensive:
- Network of Networks:
    * internet is combination of many networks within networks.

- Layers on Layers:
- Bigger & Bigger:


Lesson 2:
---
- Mobile penetration rate:
    # of mobile subscriptions/total population

- Multiple Access
    * Morse devised Morse Code to use Telegraph Cable to transmit ./- notation.
    * Bell devised multiple transmitters and recievers on same Telegraph Cable.

- Frequency Division Multiple Access (FDMA)
    * N MHz = N cycles per second.

- 0G
    MTS (1946) - Mobile Telephone System
        * FDMA system
        * An operator was required
    IMTS (1964) - Improved MTS
        * No operator required
    DynaTAC (1973)
        * From Qualcomm

- Attenuation
    WiFi is in unlicensed frequency
    Cellular is in licensed frequency
    Attenuation is decaying of signal over distance

- Cell & 1G
    + Due to attenuation, we divided regions in to cells of hexagon shape.
    + Cell towers are at the center of hexagons and base stations at
      intersection of edges of hexagons. With this design, a base station can
      serve 3 cells.
    + Frequency ranges in one cell doesn't match with next cell to avoid
      interference, but they can match with non-adjacent cells. This is done for
      spectral efficiency.

- Frequency Reuse Factor
    + Number of frequency bands used.

- 1G
    + Cells were used first time in 1G.
    + AMPS (1986) - Advanced Mobile Phone System. Operated in 800 MHz band
      system.
    + Cellular technology transformed from 0G to 1G.
    + First operated in Chicago with 90 people and 10 cells. Demonstrated
      feasibility.

- 2G
    + Migration from Analog to Digital is the most significant change from 1G to
      2G
    + Digital screens started appearing.
    + Texting was a feature.
    + 2000 was the year when 2G came.

- TDMA (Time Division)
    + TDMA uses Frequency Division with Time Division.
    + Why didn't we do this before? Analog signalling didn't allow this.
    + Sometimes called F/TDMA.

- GSM (Global System for Mobile Communication) 1982
    + First system to employ TDMA
    + 3x more than Analog capacity.
    + GSM was quickly adopted by Europe.

- CDMA (Code Division Multiple Access) 1988
    + CTIA (Cellular Technology Industries Assoc) wanted 10x improvement.
    + Qualcomm had a different idea. They added a different dimension to time
      and frequency. A code is added with each communication.

- Codes?
    + Along with digital data signal, a "spreading code" is used to transmit
      data. data signal x spreading code is transmitted.
    + Receiver uses spreading code to decode data signal.
    + Designing spreading code is very tough.
    + It needs to have a property called "Orthogonal" that two spreading codes
      should cancel out.

- Qualcomm's CDMA improvements
    + Supposedly 40x improvement (theoretically). But in reality, much less.
    + CTIA voted Qualcomm's CDMA as a 2G standard in US.

- Near-Far Problem
    + Channel quality depends on distance from base station and obstacles in
      between.
    + How much power is needed to transmit data? It's measure in Watts (W). In
      fact, phones need power in milli Watts (mW) to transmit data.
    + Phones that are farther from BS transmit data at, say, 20 mW and by the
      time it reaches BS, it's say 2 mW. While those near to BS may be received
      at 10 mW. It is important to make these signals reach BS at constant
      power. There's easy algo for this.
    + BS tells phones at what power to transmit so it receives at constant
      power.
    + Transmit Power Control (TCP) Algorithm to solve NF Problem.

- Signal Quality
    + Power is not the problem. Quality is.
    + Increasing power by a phone causes interference.
    + Signal to Interference Ratio (SIR).

- DPC (Distributed Power Control)
    + Cells transmit data to BS. BS sends SIR to phones. Phones adjust power.
    + This process iterates and finally converges on a power.




[1] https://www.coursera.org/learn/networks-illustrated/home


@@@@@@@@@@@@@@@@@@@@@@@   C++ for C Programmers    @@@@@@@@@@@@@@@@@@@@@@@@

- static_cast <new_type> (expression), dynamic_cast <new_type> (expression)
  const_cast <new_type> (exp), reinterpret_cast <new_type> (exp)

C++ is better than C (really?)
---
    + More type safe
    + More libraries
    + Less reliance on preprocessor
    + OO vs imperative

C++ Templates
---
    template <class T>
    inline void swap (T& d, T& s)
    {   T temp = s;
        s = d;
        d = temp;
    }

C++ Generics: Sum an array
---

template <class T>  //T is a generic type
T sum (const T data[], int size, T s=0)
{
    for (int i=0; i<size; i++)
        s += data[i]; // += must work for T

    return s;
}

- Defaults for function parameter in C++ must be at the end of the param list.
- When multiple default params exists, in a function call, values are assigned
  left to right.
        void func(int x=10, int y=20, int z=30);
        func(1, 2, 3);  // x=1, y=2, z=3
        func(1, 2);     // x=1, y=2, z=30
        func(1);        // x=1, y=20, z=30
        func();         // x=10, y=20, z=30


Multiple Template Arguments
---
- Be careful, as it can get very dangerous.

template <class T1, class T2>
void copy (const T1 src[], T2 dst[], int size)
{
    for (int i=0; i<size; i++) {
        dst[i] = static_cast<T2> (src[i]);
    }
}

- static_cast<> This is safe casting.

C++ casts
---
- 4 types of casting in C++
    + static_cast <type>    // considered safe
    + dynamic_cast <type>   // used with classes
    + reinterpret_cast <type>   // highly unsafe, similar to C
    + const_cast <type>     // cast away const-ness

    reinterpret_cast and const_cast are usually discouraged by instructor.

Representation of Graph
---
- Edge list representation
- Connectivity matrix (also distances).


List representation:
---
- Representation of directed graph with n verticies using an array of n lists of
  vertices.
- List i contains vertex of j if there is an edge from vertex i to vertex j.
- A weighted graph may be represented with a list of vertex/weight pairs.
- An undirected graph may be represented by having vertex j in the list for
  vertex i and vertex i in the list for vertex j.

Matrix Representation:
---
- Well known.

- Both these don't capture weight.


Dijkstra Shortest Path
---
TODO: Watch Prof. Bob Sedgwick's video.


Enum
---
typedef enum color {
    RED,
    WHITE,
    GREEN,
} color;

- enum is of type int.
- Unary operators can be overloaded. Their precedence cannot be overriden.


Operator Overloading
---

- Example of overloading ++ operator

typedef enum days {
    SUN,
    MON,
    TUE,
    WED,
    THU,
    FRI,
    SAT
} days;

inline days operator++ (days d)
{
    return static_cast <days> ((static_cast<int>(d)+1)%7);
}

- Example of overloading << operator

ostream& operator<< (ostream& out, const days& d)
{
    switch (d) {
        case SUN: out << "SUN"; break;
        case MON: out << "MON"; break;
        ...
    }

    return out;
}

- Combine above two examples

int main()
{
    days d = MON, e;
    e = ++d;    // this is calling operator++ (days) function
    cout << d << '\t' << e << endl; // calling operator<< (out, days)
}


C++ Classes
---
- Using example of a point in 2D graphs

class Point {
    public:
        double getx () {return x;}
        void setx (double v) {x=v;}

    private:
        double x,y;

}

- public, private (default), protected are access levels.

point operator+ (point &p1, point& p2)
{
    point sum = {p1.x+p2.x, p1.y+p2.y};
    return sum;
}

ostream& operator<< (ostream& out, const point &p)
{
    out << "(" << p.x << ", " << p.y << ")";
    return out;
}

- constructors
    + default constructor has no args.
    + Same name as class name.

- 'this' keyword is self referential pointer.
- destructor

- Constructor Example:

class point {
public:
    point (double x=0.0, double y=0.0):x(x), y(y) {} // constructor
    ...
private:
    double x,y;
}

    - Note the initializer list x(x), y(y). This is equivalent to
        this->x = x, this->y = y;
    - Can be done only in constructors.

Memory Management
---
- new --> equivalent to malloc()
- delete --> equivalent to free()
- They both are allocated from heap. Unlike Java, memory from heap are not
  garbage collected automatically.

Ex:
    char *s = new char[sz];
    int *p = new int(9); // single int, initialized to 9
    delete []s; // delete an array
    delete p;   // delete single element

~ destructor
---
- Destructor always has empty arg list.
- Can't be overloaded.

Randomly Generated Graph
---
    bool** graph;
    srand(time(0)); // seed
    graph = new bool*[size];
    for (int i=0; i<size; i++)
        graph[i] = new bool[size];

//2D array representing a graph

Density = Probability an edge exists, between 0 and 1

- Suppose Density=0.19

    for (int i=0; i<size; i++)
        for (int j=0; j<size; j++) {
            if (i==j) graph[i][j] = false;
            else graph[i][j] = graph[j][i] = (prob()<density);
        }

is_connected algorithm
---
    bool is_connected (bool* graph[], int size)
    {
        int old_size=0, c_size=0;
        bool* close = new bool[size];
        bool* open = new bool[size];
        for (int i=0; i<size; i++)
            open[i] = close[i] = false;
        open[0] = true;
        ...
    }

- Each iteration, add one node to closed set.

- Copy Constructors, Deep Copy vs Shallow Copy (or Referential Copy).

[1] Coursera course by Ira Pohl (Aug 2016)

@@@@@@@@@@@@@@@@@@@@@@@   OpenSwitch Tutorials     @@@@@@@@@@@@@@@@@@@@@@@@

OVSDB Overview and Troubleshooting
---
- Pub/Sub model
- Daemons connect to OVSDB server on unix socket and register for set of
  tables/columns they are interested.
- Daemons get notifications on change to these tables/columns.
- Notifications have old and new values.
- Communication happens over JSON-rpc and follows OVSDB RFC.

- vswitch.extschema - contains info on all tables/columns. OVSDB server looks
  for this on start.
- vswitch.xml - explanation of each table/column.

Commands
--
$ ps -ef | grep ovsdb-server


@@@@@@@@@@     C++ @@@@@@@@@@@
Readup More On:
- References (vs Pointers)
- try/catch (Exception Handling)

Notes:
- Access global variables in local scope using ::

    using namespace std;
    #include <iostream>

    double a = 128;

    int main ()
    {
       double a = 256;

       cout << "Local a:  " << a   << endl;
       cout << "Global a: " << ::a << endl;

       return 0;
    }

- Use 'template' to accept any datatype in to a function

    template <class T, class U>
    T minimum (T a, U b)
    {
       T r;

       r = a;
       if ((T) b < a) r = (T) b;	// typecasting b to T

       return r;
    }

- new, delete, new[], delete[]; allocate and free memory.
	char *c = new char [15];
	delete[] c;

-

@@@@@@@@@@@    STANDARD   TEMPLATE   LIBRARY    @@@@@@@@@@@@@

- STL containts five kinds of components:
    * Containers: vectors, bit vectors, lists, deques, sets, multisets, maps,
                  multimaps, queues, stacks and priority queues.
    * Iterators
    * Algorithms
    * Function Objects
    * Allocators


Vectors
=======
#include <vector>

using namespace std;

vector<T> N;    // Empty vector
vector<T> N[10]; // An array of 10 vectors. "May be" not what we want.
vector<T> N(10); // Vector of size 10.
int count = v.size();   // size of vector. Don't use this to determine if vector
                        // is empty.
bool is_empty = v.empty();

vector<int> v;
v.push_back(100);   // adds 100 at end of vector. Don't worry about memory
                    // allocation. It's NOT done one at a time.

v.resize();     // resize

NOTE: Is you push_back() after resize(), new elements are added AFTER newly
created memory and NOT INTO it.

v.clear();  // vector now has zero elems.


Vector Initialization
-------
vector<int> v1;     // Default values are zeros
...
vector<int> v2 = v1;
vector<int> v3(v1);


vector<int> nums (20, 5);   // Default values are 5.
vector<string> names (20, "Unknowns");  // Defaults to "Unknowns"


Multidimensional Vectors
--------
vector< vector<int> > Matrix;
vector< vector<int> > Matrix(N, vector<int>(M));    // of size NxM, with zero
                                                    // default values
vector< vector<int> > Matrix(N, vector<int>(M, -1));    // of size NxM, -1
                                                        // initial value.



Pairs
=====

A simple form:

template<typename T1, typename T2> struct pair {
     T1 first;
     T2 second;
};

pair<int, int> P;   // pair of ints
pair<string, pair<int, int> > P;    // pair of string and two ints.

 pair<string, pair<int,int> > P;
 string s = P.first; // extract string
 int x = P.second.first; // extract first int
 int y = P.second.second; // extract second int



Iterators
=========

References:
[1]
https://www.topcoder.com/community/data-science/data-science-tutorials/power-up-c-with-the-standard-template-library-part-1/
[2]
http://cs.brown.edu/~jak/proglang/cpp/stltut/tut.html
[3]
http://www.tutorialspoint.com/cplusplus/cpp_stl_tutorial.htm
[4]
http://en.cppreference.com/w/cpp

@@@@@@@@@@@@@@@@    NAND 2 TETRIS COURSE    @@@@@@@@@@@@@@@@@@@@@@


Week 1: Boolean Logic
-----
Commutative Law:
---
x AND y = y AND x
x OR y = y OR x

Associative Law:
---
x AND (y AND z) = (x AND y) AND z
x OR (y OR z) = (x OR y) OR z

Distributive Law:
---
x AND (y OR z) = (x AND y) OR (x AND z)
x OR (y AND z) = (x OR y) AND (x OR z)


De Morgan Law:
---
NOT (x OR y) = NOT (x) AND NOT (y)
NOT (x AND y) = NOT (x) OR NOT (y)

Idempotent Law:
---
w AND w = w
w OR w = w



Truth Table to Boolean Expression: How to?
---
- For every row resulting in 1, write a boolean expression for that row alone
  (ignore other rows). OR all these expressions.

Theorem 1:
Any boolean function can be represented using an expression containing AND, OR
and NOT.

Theorem 2:
A boolean function can be represented using AND and NOT gates only.

Theorem 3:
A boolean function can be represented using NAND gate alone.

x NAND y = NOT (x AND y)

- How can an AND be done using NAND?
    x AND y = NOT (x NAND y)

- How can a NOT be done using NAND?
    NOT (x) = x NAND x

Logic Gates
---

Truth Table of NAND gate:
    x   y   out
    0   0   1
    0   1   1
    1   0   1
    1   1   0

Functional Spec: if (x==1 and y==1) then out = 0
                 else out = 1


Other elementary gates: AND, OR and NOT


Composite Gates:
---
- Some of logic gates to form a more complex gate.


HDL:
---
- HDL is a functional/declarative language.
- Order of HDL statements is insignificant.
- Interface of logic gates:
    Not (a=, out=), And (a=, b=, out=), Or (a=, b=, out=)

- Common HDLs: VHDL, Verilog and many more.
-


Multiplexor:
-----
Three Inputs: a, b, sel
One Output: out

if (sel ==0) out = a
else out = b


Demultiplexor: Inverse of Multiplexor
---
Input: in, sel
Output: a, b

if (sel==0) {a,b} = {in,0}
else {a,b} = {0,in}



Project 1: 3 types of logic gates will be built.

    Elementary Logic Gates: Not, And, Or, Xor, Mux, DMux
    16-bit variants: Not16, And16, Or16, Mux16
    Multi-way variants: Or8Way, Mux4Way16, Mux8Way16, DMux4Way, DMux8Way



Reference:
https://www.coursera.org/learn/build-a-computer/


@@@@@@@@@@@@@@@@    UPLOAD SITES TO AWS     @@@@@@@@@@@@@@@@@@@@@@
// Steps to upload HTML/CSS static websites to AWS

1) Hosting Static Website on Amazon Web Services (AWS)

3 main steps:
    * Host/Deploy static website using AWS S3 (storage)
    * Associate domain name with your website using Route 53 (DNS)
    * Speed up your site using CloudFront (CDN)

Host Static Website:
------
- Create Amazon S3 "buckets", to store files for your website
- Upload files to this bucket.
- Configure bucket to act as website.

// Read [1] for more details explaining behavior of each entity described
// above.

- Sign up for AWS
- Sign up for IAM (Identity and Access Mgmt). [2] gives more details



References:
[1] http://docs.aws.amazon.com/gettingstarted/latest/swh/website-hosting-intro.html
[2] http://docs.aws.amazon.com/gettingstarted/latest/swh/setting-up.html

===============================================================================
Deep Dive in to Hyper-V Networking [1]
---
Few key concepts:
                            Availability
                            Reliability
          Manageability     Predictability
                            Security
                            Extensibility


- Built concepts similar to LAG for NIC failover (called LBFO - Load Balancing
  Fail Over).


    +--------------------+
    |   Management OS    |
    |                    |
    |                    |    VM1   VM2   VM3  .....  VMn
    |                    |
    |   Live  Migration  |    +--------------------------+
    |                    |    |                          |
    |      Storage       |    |  Hyper-V virtual switch  |
    |                    |    |                          |
    |    Management      |    +--------------------------+
    |                    |                |
    +--------------------+    +---------------------------+
                              |     LBFO Team NIC         |
                              +---------------------------+
                                          |
                                       PHY NICs


- Few other concepts: Port ACLs, Private VLANs, DHCP Guard

- Single Route - IO Virtualization (SR-IOV). Single physical NIC talks
  directly to VM instead of going through Hyper-V Switch.
- For SR-IOV to work, Physical NICs must be capable of this feature.

- Dynamic VM Queue
- Many more features, that I saw in Cisco/Juniper type switches. Wonder why
  they are all built in to Windows Server.




[1] https://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-437T

===============================================================================
Envoy, an L7 Proxy from Lyft - Talk by Architect Matt Klien

- Lyft's architecture was very simple in 2013


    client <---> internet <----> Amazon ELB <----> PHP/Apache <------> DB
                                                    monolith


- By 2015 architecture changed


    client <---> internet <----> Amazon ELB <----> PHP/Apache <------> Amazon ELB
                                 (external)         monolith           (internal)
                                                       +                   ^
                                                HAProxy+Messaging          |
                                                       |                   |
                                                       |                   |
                       +------------------------------ +                   |
                       |                               |                   |
                       |                               |                   |
                       v                               v                   v
                    MongoDB                         DynamoDB <--------> Python
                                                                       Services


- State of SoA (Service Oriented Architecture) Networking in Industry
// Very detailed. Watch video

- What is Envoy?
    - It's a self contained proxy. It is NOT a library. Think like HAProxy.
      It's like a server (Nginx).
    - Other names used is 'sidecar'.
    - It's an L3/L4 filtering mechanism: It's byte proxy at it's core. Can be
      used for things other than HTTP
    - L7 filter architecture: Packets can be routed based on L7 info
    - Built with HTTP/2 first in mind. Uses gRPC
    - Does Service Discovery and Active/Passive health checking.
    - Advanced Load Balancing: Retry, circuit breaking, timeouts, rate
      limiting, outlier detection, shadowing etc
    - Best in class Observability: stats, logging, tracing
    - Edge proxy: routing and TLS (can be used as Nginx replacement)


    +------------------------+                 +-----------------------+
    |    Service Cluster     |                 |    Service Cluster    |
    |                        |                 |                       |
    |       Service          |                 |       Service         |
    |         ^              |                 |         ^             |
    |         |              |                 |         |             |
    |         |              |                 |         |             |
    |         v              |     HTTP/2      |         v             |
    |       Envoy <----------------------------------> Envoy           |
    |         ^              |   REST/gRPC     |         ^             |
    |         |              |                 |         |             |
    +---------|--------------+                 +---------|-------------+
              |                                          |
              |                                          |
              +-----------> External Services <----------+
              |              (DynamoDB etc)              |
              |                                          |
              +----------->    Discovery      <----------+

- Service in a cluster only talks to Envoy client in local cluster


- With Envoy, Lyft's architecture today

       clients                        +---> Legacy Monolith <--+--> MongoDB
         ^                            |      (+Envoy)          |
         |                            |                        |
         |                            |                        |
         v                            |                        |
    internet <----> "Front" Envoy <---+---> Go Services <------+--> DynamoDB
                    (via TCP ELB)     |      (+Envoy)          |
                                      |                        |
                                      |                        |
                                      +---> Python Services <--+--> Stats, Tracing
                                             (+Envoy)          |
                                                               |
                                                               |
                                                               +--> Discovery


Eventually Consistent Service Discovery
----
- Fully consistent service discovery problems are very popular (etcd,
  Zookeeper, consul).
- They are very hard to run at scale.
- Service discovery is an eventually consistent problem. Envoy is designed for
  it.

Advanced LB:
----
- Different Service Discovery types
- Zone aware LB
- Dynamic stats: Per zone, canary specific stats etc
- Circuit breaking: Max connections, requests, retries
- Rate limiting:
- Request Shadowing: Fork traffic to test cluster (similar to port mirroring)
- Retries:
- Timeouts:
- Outlier detection: Consecutive 5xx
- Deploy control


Envoy Deployment @Lyft
----
- > 100 services
- > 10000 hosts
- > 2M RPS (Requests Per Sec)
- All service to service traffic (REST and gRPC)
- Many more


[1] Lyft's Envoy L7 Proxy
https://www.youtube.com/watch?v=RVZX4CwKhGE&list=WL&index=20&t=25s

===============================================================================
Introduction to Virtual Memory

- Simple systems have single address space:
    * no protection
    * processes can stomp on each other's memory

- x86 Physical Memory Map

    +---------------------------------+
    |   Memory Mapped I/O (PCI)       |
    +---------------------------------+
    |   Extended Memory               |
    |                +----------------+
    |                | High Mem Area  |
    +---------------------------------+
    |  BIOS                           |
    +---------------------------------+
    |  VGA Memory                     |
    +---------------------------------+
    |  DOS Area                       |
    +---------------------------------+

- Hard to develop for Physical Memory.

Advantages of VM
- Each process can have it's own view of memory
- Kernel memory is invisible to user space
- Process Memory can be moved and/or swapped to disk, all this is done by Kernel
  and user space program doesn't know.
- Physical RAM can be mapped to multiple processes at once (shared mem)
- Memory regions can have access permissions
    * read/write/exec

Memory Management Unit (MMU)
- Hardware responsible for implementing virtual memory
- Sits between CPU core and RAM. On most CPUs, it's part of the core.
- Separate from RAM controller
- Responsible for mapping user space programs to RAM and/or peripheral devices
- Handles Load/Store instructions
- Generates exceptions (Page Faults) on invalid access
    * Unmapped address or insufficient permission

Translation Lookaside Buffer (TLB)
- TLB is a list of mappings from virtual memory to physical address in H/W.
    * Also holds permission bits
- TLB is part of MMU.
- There are fixed # of entries in TLB, varies by CPU.


+-----------------------------------+
| SoC                               |
|                   +-------+       |
|                   |  ALU  |       |       +-------+
|                   +-------+       |       |  RAM  |
|                       |           |       |       |
|                       |           |       |       |
|   +-------+       +-------+       |       |       |
|   |  TLB  +-------+  MMU  |       |       |       |
|   +-------+       +-------+       |       |       |
|   +-------+           |           |       |       |
|   +-------+           |           |       |       |
|   +-------+       +-------+       |       |       |
|                   |Memory |       |       |       |
|                   |Contro-|-------|-------|       |
|                   |ller   |       |       +-------+
|                   +-------+       |
|                                   |
+-----------------------------------+

TLB Task
- Maps virtual address to physical address
- If not present or insufficient permission, generate a page fault

Page Faults
- It's a CPU exception, generated when software attempts to use invalid virtual
  address
    * virtual address is invalid or not mapped to physical addr
    * process has insufficient permissions
    * virtual address is valid, but swapped out

Kernel Virtual Memory
- In Linux, kernel uses virtual memory, just like user space processes. Not all
  OSes do this.
- Virtual address space is split
    * upper part is used by kernel
    * lower part is used by user space
- On 32-bit, split is at: 0xC0000000 (3GB)

    +-------------------+ 0xFFFFFFFF
    | Kernel Address    |
    +-------------------+ CONFIG_PAGE_OFFSET
    |                   | (default 0xC0000000)
    |                   |
    |                   |
    +-------------------+ 0x00000000

- Virtual address space is 4GB on 32-bit system, regardless of how much RAM we
  have.
- Kernel uses top 1GB and user space uses bottom 3GB

- 3 kinds of Virtual Addresses
    * Kernel:
        + Logical Address (kmalloc)
        + Virtual Address (vmalloc)
    * User space:
        + Virtual Address

- Kernel Logical Address:
    * Direct mapping to physical memory. If physical memory is small (say 8MB),
      then logical address maps entire 8MB directly. It helps in DMA and such.
    * In large memory systems, not all physical memory can be mapped in to
      kernel's address space. So, it's top 1GB of VM is mapped to BOTTOM half of
      physical RAM.
    * ~104MB at top of kernel VM is reserved for non-contiguous allocations
        + Used by vmalloc()
        + This is on 32-bit only. On 64-bit, there's lot more address space.



            Virtual Address Space
         +------------------------+ 0xFFFFFFFF
         |                        |
         | Kernel Virtual Address |
         |                        |
         +-------------------------------------+
         |                        |            |
         |Kernel Logical Address  |            |
         |                        |0xC0000000  |
         +-----------------------------+       |
         |                        |    |       |
         |                        |    |       |
         |                        |    |       |
         |                        |    |       |           Physical RAM
         |                        |    |       |      +------------------------+
         |                        |    |       |      | RAM without direct     |
         |   User Space Address   |    |       |      | mapping in to kernel   |
         |                        |    |       |      | space (aka High Mem)   |
         |                        |    |       +----> +------------------------+
         |                        |    |              | RAM with direct        |
         |                        |    |              | mapping in to kernel   |
         |                        |    |              | space (aka Low Mem)    |
         +------------------------+    +------------> +------------------------+


- Kernel Virtual Address:
    * Area above Kernel Logical Address space. Also called vmalloc area.
    * Non-contiguous memory mapping
    * Often used for large buffers, using vmalloc()
    * Used for memory-mapped I/O
        + Map peripheral devices in to kernel (PCIe, SoC IP blocks)
        + request_mem_region(), kmap()

User Space Virtual Addresses
----
- Only used portion of RAM are mapped, memory is non-contiguous, may be swapped
  out and can be moved.
- User space VA makes full use of MMU.
- Each process has it's own memory map:
    struct mm, pointers in task_struct
- At context switch time, the memory map is changed



Memory Management Unit (MMU)
---
- H/W component that maps VM to Physical RAM
- Page, basic unit of memory size:
    ARM - 4K
    ARM64 - 4K to 6K
    MIPS - widely configurable during kernel build time
    x86 - 4K

Terminology:
---
- Page: Unit of memory sized and aligned at page size.
- Page Frame (Frame): refers to page-sized and page-aligned physical memory
  block.
- Page is abstract, Frame is concrete
- pfn: Page Frame Number refers to physical page frames


MMU
---
- TLB holds VA + PA + Permissions

Page Faults
---
- Process accesses region of memory that's not mapped, MMU will generate a PF
- Fact of life in MMU, not a bad thing but want to minimize
- Page faults at context switch time
- Lazy allocation: Kernel will wait until pages are actually used to allocate it
  in Phys RAM

Shared Memory
---
- mmap() sys call allows user space process to request specific virtual address
  to map the shared mem region.
    * kernel may not be able to grant this request and mmap() can fail

Lazy Allocation
---
- User space program is never aware of page faults
- For processes that are time-sensitive, pages can be pre-faulted, or simply
  touched, at start of execution (something like setting to zero using memset)
    * If required, mlock() and mlockall() locks these pages in Physical memory
      and does not swap them out to disk.

Page Tables
---
- Entries in TLBs are limited resource.
- Far more mappings exists than in TLB at one time. Kernel must keep track of
  all of them.
- Kernel stores this in Page Tables
    * See struct_mm and vm_area_struct

User Space Memory Allocation
---
- There are *alloc() functions
- mmap() can be directly used to allocate and map pages
    MAP_ANONYMOUS - normal mem alloc
    MAP_SHARED - shared mem with other process
- brk()/sbrk() can be used to resize heap
- malloc/calloc uses
    brk() - for small mem alloc
    mmap() - for large mem alloc
- Stack expansion:
    * If process access mem beyond stack, page fault happens
    * Kernel allocates new page to extend the stack
    * See __do_page_fault() in arch/arm/mm/fault.c




[1] https://www.youtube.com/watch?v=EWwfMM2AW9g&index=16&list=WL&t=2s

===============================================================================
VBA Scripting

Dim <varname>
MsgBox

If answer = vbYes Then
Else
End If
===============================================================================
Performance Analysis of QUIC protocol

QUIC - Quick UDP Internet Connections

Goals of QUIC:
- Security protection comparable to TLS
- Reduce connection latency

Setup Time: QUIC vs TLS
- TLS runs over TCP, so TLS key establishment follows TCP handshake
- QUIC combines these two in to single step, kind of reducing latency by half:
    * connection establishment and key agreement

Data Exchange: QUIC vs TLS
- TLS first establishes session key and then data exchange
- QUIC sends data along with session key establishment. So, 1 RTT saved

                    QUIC Protocol
    Client                              Server
cid = {0,1}^64
                c_i_hello:cid
                ------------------->    generate stk based on client IP



cid: Connection ID picked by cliend (64 bit)
stk: source-address token used to prevent spoofing
scfg: server config contains servers public Diffie-Hellman (DH) values




[1] https://www.youtube.com/watch?v=vXgbPZ-1-us&index=38&list=WL

===============================================================================
REST API's with Flask and Python

Python Concepts
====
- Notes are in Python.txt file

HTTP Verbs
====

[verb] [path] [protocol]
Ex: GET / HTTP/1.1

Other Verbs: POST, DELETE, PUT, OPTIONS, HEAD and much more

REST Principles
====
What is REST API?
----
- It's a way of thinking on how a webserver (or service) responds to requests.
- REST responds with "resources" and not with "data"

- REST is Stateless



Reference:
[1] https://learnpythonthehardway.org/python3/
[2] http://book.pythontips.com/en/latest/
[3] Safari Books Online: REST APIs with Flask and Python (video)


=======================================================================
- How Dockers Work 1: chroot

Python example:

---
import os

uid = os.getuid()
gid = os.getgid()
os.chdir("/tmp")
os.chroot("/tmp")
os.setgroups([])
os.setgid(gid)
os.setuid(uid)
os.umask(077)
os.listdir("/")
---

- How Dockers Work 2: cgroups
- How Dockers Work 3: namespaces

- Namespaces is implemented using clone() syscall

- Dockers Design
    * A server process runs as root and controls all containers (this is
      Dockers Achiles heel)
    * This process is controlled by CLI tool 'docker'
    * Dockerfiles control images
    * The process manages containers, which are images in execution

- Registries
    * https://hub.docker.com
    * cloud.google.com/tools/container-registry <-- Google's own registry


Kubernetes Structure:
1) 'kubectl' is entry to Kubernetes deployment
2) Controller Manager controls cluster-level functions
3) Scheduler talks to nodes to allocate resources
4) nodes then host pods
5) kube-proxy runs on each node and controls the network on nodes
6) pods then have groups of docker containers in them
7) docker containers share the same logical volume

Definitions:
POD: A group of coordinating containers. It's the smallest unit the Scheduler
understands. The containers within the pod share some context and interact,
directly or indirectly, with each other.

NODES: Closely associated with machines no network

SERVICES:

KUBE-PROXY: Manages network access on a node. Controls access to services on
the node. It also does load balancing.

SCHEDULER: Which Nodes have available resources and runs Pods on them.

CONTROLLER MANAGER: controls all other cluster level functions

KUBECTL: Talks to REST endpoint (Scheduler/Controller Mgr) to make it do
things





Resources:
[1] O'Reilly Kubernetes Bootcamp: Get up to speed with Docker and Kubernetes -
Zed Shaw


===============  PROTOBUFS   ====================
- Protocol Buffers, replacements for JSON, XML and such
- Creation of Google in 2008 for internal server to server communication.
- Much faster than JSON/XML etc

- How do they work?
    * Binary serialization
    * Uses pre-determined schema for encoding/decoding
    * Compiled in to many languages

- Schema, how does it look?

    [required|optional]  [type] [var_name] = [seq no] [default=]

- Schema fields are indicated and aliased with a number and a tag
- Required, optional and repeated
- Schemas allow messages to be extensible

Advantages of Protobufs
---
- Lightweight: less space and faster transmission
- Easy to modify
- Extensible


[1] https://www.youtube.com/watch?v=72mPlAfHIjs&index=51&t=119s&list=WL
[2] https://www.youtube.com/watch?v=EAFK-tN_yaw&index=52&list=WL


===============  KAIROS DB   ====================
- Storage in Kairos can be viewed as following Java collection

HashMap<partition_key, TreeMap<cluster_key, value>>

- An example schema in Kairos looks like:

public static final String DATA_POINTS = "" +
    "CREATE TABLE IF NOT EXISTS data_points (" +
    " metric    text, " +
    " row_time  timestamp," +
    " data_type text," +
    " tags frozen<map<text, text>>," +
    " offset int, " +
    " value blob," +
    " PRIMARY KEY ((metric, row_time, data_type, tags), offset)" +
    ") WITH COMPACT STORAGE ";


[1] https://www.youtube.com/watch?v=F2ukOa1gGlo

===============  JSON SYNTAX    ====================

- JSON syntax is derived from JavaScript object notation syntax:
	* Data is in name/value pairs
	* Data is separated by commas
	* Curly braces hold objects
	* Square brackets hold arrays

- More JSON syntax:
	* Numbers in JSON can be int or float and don't need quotes
		{ "age":30 }
	* JSON values can be an array
		{ "employees":[ "John", "Anna", "Peter" ] }
	* JSON values can be JSON objects
		{ "employee":{ "name":"John", "age":30, "city":"New York" } }
	* JSON Booleans
		{ "sale":true }
	* JSON null
		{ "middlename":null }

- JSON Object Syntax
	* JSON objects are surrounded by curly braces {}.
	* JSON objects are written in key/value pairs.
	* Keys must be strings, and values must be a valid JSON data type (string,
	  number, object, array, boolean or null).
	* Keys and values are separated by a colon.
	* Each key/value pair is separated by a comma.


Differences				JSON					Javascript

Keys				Strings in double	|	Can be strings, numbers or
					quotes				|	identifier names
										|
Values				string, number, JSON|	all of JSON values + function,
					object, array, bool |   a date, undefined??
					null				|
										|
Syntax		        All strings are in  |   Strings can be in single or double
					double quotes       |   quotes
										|


				JSON			Python
				----------------------
				object			dict
				array 			list
				string 			str
                number (int) 	int
                number (real) 	float
                true 			True
                false 			False
                null 			None

Python
------
json.dump(j, f):    Write JSON object 'j' to file or file-like object 'f'. Here
                    'j' is a python dictionary.
				- Serialize object 'j' as JSON formatted stream to 'f'

json.dumps(j):      Output JSON object 'j' as string.
				- Serialize object 'j' in to string

json.load(f):   Create a python dict from a file or file-like object containing
                data in JSON format.
				- Deserialize content in 'f' to a python object (dict)

json.loads(s):  Do same thing as above, but input is a string in JSON format.
         ^
         The 's' here indicates string
				- Deserialize string in JSON format, to a python object (dict)


Example
-------
Assume following in file.txt:
{
    "title" : "Gattaca",
    "release_year" : 1997,
    "is_awesome" : true,
    "won_oscar" : false,
    "actors" : ["Ethan Hawke", "Uma Thurman", "Alan Arkin", "Loren Dean"],
    "budget" : null,
    "credits" : {
            "director" : "Andrew Niccol",
            "writer" : "Andrew Niccol",
            "composer" : "Michael Nyman",
            "cinematographer" : "Sławomir Idziak"
        }
}


>>> json_file = open("/home/user/file.txt", "r", encoding="utf-8")
>>> movie = json.load(json_file)
>>> type(movie)
<class 'dict'>
>>>
>>> movie
{'title': 'Gattaca', 'release_year': 1997, 'is_awesome': True, 'won_oscar': False, 'actors': ['Ethan Hawke', 'Uma Thurman', 'Alan Arkin', 'Loren Dean'], 'budget': None, 'credits': {'director': 'Andrew Niccol', 'writer':'Andrew Niccol', 'composer': 'Michael Nyman', 'cinematographer': 'Sławomir Idziak'}}


- Note that 'null' was converted to None, true to True and false to False in python dict.



json.loads(s)
---
>>> s = """
... {
...     "title" : "Gattaca",
...     "release_year" : 1997,
...     "is_awesome" : true,
...     "won_oscar" : false,
...     "actors" : ["Ethan Hawke", "Uma Thurman", "Alan Arkin", "Loren Dean"],
...     "budget" : null,
...     "credits" : {
...             "director" : "Andrew Niccol",
...             "writer" : "Andrew Niccol",
...             "composer" : "Michael Nyman",
...             "cinematographer" : "Sławomir Idziak"
...         }
... }
... """
>>>
>>>
>>> movie = json.loads(s)
>>> type(movie)
<type 'dict'>
>>> movie
{u'won_oscar': False, u'release_year': 1997, u'title': u'Gattaca', u'budget': None, u'credits': {u'director': u'Andrew Niccol', u'writer': u'Andrew Niccol', u'cinematographer': u'S\u0142awomir Idziak', u'composer': u'Michael Nyman'}, u'actors': [u'Ethan Hawke', u'Uma Thurman', u'Alan Arkin', u'Loren Dean'], u'is_awesome': True}

- Note that non-ASCII char is converted to unicode point (\u0142)
- If you don't want that, use the following

>>> movie = json.


===============  CELERY CONECPTS   ====================
- Used for async task scheduling.
- If a web app doesn't have to wait for a result (not-synchronous) they can
  use Celery
- It needs a message broker and a backend. Recommended: RMQ and Redis
- When using Celery, don't send task objects to Celery whose state can change.
- Celery has built in log handler and it's very useful.
- Flower - Web interface for Celery
- Celery Alternatives
    Huey
    RQ
    But they are not as configurable as Celery

[1] https://www.youtube.com/watch?v=7ZkZr7apcJs
[2] https://samgclarke.github.io/presentations/#/
[3]

===============  SCALA NOTES ===============
EXPRESSIONS
scala> 1 + 1
res0: Int = 2

VALUES
scala> val two = 1 + 1
two: Int = 2

VARIABLES
scala> var name = "steve"
name: java.lang.String = steve

scala> name = "marius"
name: java.lang.String = marius


FUNCTIONS ('def' keyword)
scala> def addOne(m: Int): Int = m + 1
addOne: (m: Int)Int

- You can leave off parens on functions with no args

Example:
scala> def three() = 1 + 2
three: ()Int

scala> three()
res2: Int = 3

scala> three
res3: Int = 3


ANONYMOUS FUNCTIONS
scala> (x: Int) => x + 1
res2: (Int) => Int = <function1>

scala> res2(1)
res3: Int = 2




[1] https://twitter.github.io/scala_school/basics.html

===============  LINUX PERFORMANCE TRAINING NOTES ===============

Key useful tools:
    free -m
    top
    iostat 1
    sar -B 1
    sar -n DEV
    vmstat 1
    pidstat


Contents Covered:
====
- The USE method
- CPU sampling with perf
- Recording CPU stacks with 'perf record -ag -F'
- Flame graphs and missing symbols

- Resolving stack symbols for JIT- or interpreted runtimes using perf maps
- Java perf-map-agent, Node.js --perf_basic_prof
- Use jps and perf-map-agent to record stack traces of a JVM application

Kernel tracepoints, kprobes, and perf-tools
----
- Kernel tracepoints and perf support for recording them (perf -e)
- Interesting kernel tracepoints: scheduler, disk, network device
- Demo: recording fork events with perf -e (poor man’s execsnoop)
- Brendan Gregg’s perf-tools repository -- an ftrace + perf_events frontend
- Demo: characterizing load generated by a process using syscount -c -p PID
- Dynamic instrumentation with kprobes and uprobes (tracepoints aren’t available)
- Discovering functions to probe with /proc/kallsyms and objdump -T
- Demo: probe-based ad-hoc investigations (failed malloc(), long pthread_mutex_lock(), MySQL queries)
- Diagnose an application that fails to start because it keeps looking for a
configuration file that doesn’t exist

- First, identify heavy disk writes performed by the application, and their latency
- Then, collect call stacks with perf -g -p PID -e … for the block_rq_insert
tracepoint and optionally create a flame graph to pinpoint where the writes are
coming from


eBPF
====
- The challenges with perf and similar tools that we’re trying to address, mostly
that of overhead (transferring all events to user space for analysis via a
file)
- BPF history -- from Berkeley Socket Filters to tomorrow’s tracing infrastructure
- BPF support in various kernel versions
- Structure and architecture of a BPF tracing program
- Example: contrast between perf-based slow file I/O investigation and BPF-based (only aggregate interesting events and report summaries)


perf notes
=======
- Find out where a program is taking so long to run
	$ perf record -g -F 997 -- ./primes

	/*
        -g  record call stack
        -F  inspection frequency (per second)
        -a  includes events from other processes
        -t  trace single thread within a process
	*/

- Inspecting perf's report
    $ perf report --stdio
    $ perf annotate

- Using FlameGraphs
    $ perf script > primes.stacks
	$ FlameGraph/stackcollapse-perf.pl primes.stacks > primes.collapsed
	$ FlameGraph/flamegraph.pl primes.collapsed > primes.svg

Profiling Java Code
====
- To help perf find Java symbols for your code, use perf-map-agent
- More scripts in perf-map-agent repo to record and report Java app behavior
	perf-java-record-stack
	perf-java-report-stack


Lab: Broken File I/O
---
- Lookup where process is spending most of it's time
    $ pidstat -u -p $(pidof <name>) 1   // show every second
    $ syscount -c -p $(pidof <name>)    // syscount is from perf-tools
    $ opensnoop -p $(pidof <name>)      // snoops all open() syscalls


Lab: Slow File I/O
---
    $ ./iolatency       // get an idea of disk I/O
    $ ./bitesize        //

$ perf record -p $(pidof logger) -e block:block_rq_insert -g -- sleep 10


Day 2
====
- Look at tpoint, kprobe, uprobe in perf-tools/ folder

- tpoint - tracepoints. Statically available in kernel
- kprobe - Kernel dynamic tracing
- uprobe - Userspace dynamic tracing

- perf-tools are much more useful/streamlined than just perf itself


eBPF
====
- Disadvantages of perf?
    * perf_event handler intercepts workflow in kernel
    * perf relies on pushing lot of data to userspace through files for
      analysis
    * perf designed to push small data from kernel to userspace for analysis
    * perf cannot do analysis of packets efficiently

- BPF - Very efficient way of filtering packets in kernel

- seccomp and XDP uses BPF to provide ...

- BCC BPF Front-End Tools
    * BPF Compiler Collection (BCC)

- USDT probes
- uprobes



// some of these pages may be removed (from class notes)
[1] https://gist.github.com/goldshtn/c5656c1fe91b7970d7182add128a21c8
[2] https://github.com/goldshtn/linux-tracing-workshop/blob/master/perf.md
[3]

[] https://github.com/BrendanGregg/FlameGraph
[] https://github.com/jrudolph/perf-map-agent
[] https://www.kernel.org/doc/Documentation/kprobes.txt
[] https://github.com/brendangregg/perf-tools
[] https://github.com/iovisor/bcc


===============  LEARNING REST - STARING WITH REST API  ==============

Contents
--------
Theory                          Applied
REST Defined                    A nodeJS Dev environment
REST & HTTP                     nodeJS and REST
REST & CRUD ops                 REST routes
Designing for REST              REST controllers
                                Testing our REST APIs

- REST: REpresentational State Transfer
    An architectural style for designing networked applications

HTTP Request Types
-----
GET: Retrieves content from a specified resource
POST: Submits data for processing to a URI
PUT: Sends info to update a server resource
DELETE: Sends request to delete a resource

Others: HEAD, OPTIONS, PARTIAL

API Structure Example
-----
GET: https://mydomain/api/items
GET: https://mydomain/api/items/details/22
POST: https://mydomain/api/items/details/22

- There can be multiple request types to same end point


REST + CRUD ops
---------------


API Type -->    Create      Read        Update      Delete
                new item    an item     an item     an item
------------------------------------------------------------
HTTP Request    POST        GET         UPDATE      DELETE
                                        (or PUT)

Database        Create      Read        Update      Delete


Example APIs
------------
https://api.chucknorris.io
https://api.chucknorris.io/jokes/random
https://api.chucknorris.io/jokes/random?category={category}
https://api.chucknorris.io/jokes/categories
https://api.chucknorris.io/jokes/search?query={query

< --- stopped the video after a while. It went too much in to Node and stuff,
which I was not interested at this time ---->

[1] Safari Books Online Video


===============   LEARNING DB DESIGN WITH MYSQL   =====================
Contents
---
- Junction Tables
- Relations & Foriegn Keys
- Aliases & Joins
- Math, IN clauses, GROUP BY, ORDER BY
- Subqueries


// there are not_null, primary_key and such important attributes to add for
// each field
CREATE TABLE <tablename> (field1 TYPE ...)

INSERT INTO <tablename> (field1, field2, ...) VALUES ("value1", "value2", ....)

// ALTER is used to change datatype of a field
ALTER TABLE <tablename> MODIFY <field> type

SELECT field1, field2 FROM database.table

- search names (or part of fields) using % char (similar to * for grep
  searches)
- && is logical AND. Can use either
- || is logical OR

- ORDER BY <field> [DESC]; orders the fields in ascending or descending order
- LIMIT <start>, <end>; that many records are displayed

- UPDATE <tablename> SET name="New name" WHERE id=<id>

// foriegn key constraints needs to be addressed first.
// CASCADE - if primary entry is deleted, all foriegn key entries are deleted
// RESTRICT - prevent deletion of entry if foriegn key exists
// no restriction - deleting an entry leaves foriegn keys in respective tables
- DELETE FROM <tablename> WHERE id=<id>

Aliases & Joins
// Column aliasing
- SELECT first_name AS "first name",
         last_name AS "last name"
  FROM customers


// Table aliasing
- SELECT p.name, pc.name, p.price
  FROM product_categories AS pc, products AS p
  WHERE p.category=pc.id


// JOINS
- INNER, LEFT, RIGHT

// INNER JOIN
SELECT <fields>
FROM table_a A
INNER JOIN table_b B
ON A.key = B.key

// LEFT JOIN
SELECT *
FROM table_a A
LEFT JOIN table_b B
ON A.key = B.key

// RIGHT JOIN
SELECT *
FROM table_a A
RIGHT JOIN table_b B
ON A.key = B.key

// INNER JOIN 3 tables
SELECT <fields>
FROM table_a A
INNER JOIN table_b B ON A.key = B.key
INNER JOIN table_c C ON C.key = B.key
ORDER BY C.first_name   // optional


// Concat, Count, Union
- Union of two tables only if both have same # of columns and datatypes of
  those columns. Column names can be different

// Gives one instance of each city in both tables
SELECT city FROM customers
UNION
SELECT city FROM vendors
ORDER BY city

- UNION ALL will gives all instances of eacy city in tables
SELECT city, zipcode FROM customers
WHERE zipcode='11111'
UNION ALL
SELECT city, zipcode FROM vendors
WHERE zipcode='11111'
ORDER BY city

// Concat
SELECT id, CONCAT(first_name, '-', email) AS "Name & Email" FROM customers

// Count
SELECT COUNT(*) FROM products

// Distinct option
SELECT COUNT(DISTINCT category) FROM products


// IN clause, similar to OR statements
// NOT IN clause is opposite of IN clause
SELECT city, state FROM customers WHERE state IN ("MA", "FL", "MD")

// Output with operations as column names and result as values
SELECT (1+1), (2*3), (4/2)

// Max, Min, Avg
SELECT MAX(price) FROM products
SELECT MIN(price) FROM products
SELECT AVG(price) FROM products

// Subqueries
SELECT name, MIN(price) FROM products
WHERE id IN (SELECT id FROM products WHERE description LIKE "%treadmill%")


// Version of SQL
SELECT VERSION()

// Date & Time
SELECT NOW()


// GROUP BY function
SELECT customer, COUNT(*) AS "Number of Orders" FROM orders WHERE customer=1

SELECT customer, COUNT(*) AS "Number of Orders" FROM orders GROUP BY customer

TODO: Whatever fields are SELECT'ed, must also be present in GROUP BY.
Research more on this.


Cheat Sheet
----
- Pretty print. Couple of methods:
mysql> pager less -SFX
mysql> SELECT * FROM sometable\G; << pretty print only this command

- Details about current database
mysql> SELECT database();
mysql> \s
mysql> status;
mysql> show variables;  << tons of info
mysql> show tables;     << list all tables in db, including views
mysql> show full tables where Table_Type = 'BASE TABLE' << list all table, and no views
mysql> describe <table-name>;   << gives lot of details about the table


- How to list all views?
SHOW FULL TABLES IN database_name WHERE TABLE_TYPE LIKE 'VIEW';
SHOW FULL TABLES WHERE TABLE_TYPE LIKE 'VIEW'; << if you are already in a database


mysql> show triggers;   << list all triggers in DB
mysql> show tiggers like '%tablename%';     << list all triggers on table

mysql> show create procedure <proc_name>;   << show procedure source code

// List all procedures
mysql> show procedure status [like 'pattern' | where search_condition];

// List all indexes on a table
mysql> show index from <yourtable>;

// How to measure query time
mysql> SET profiling = 1;
mysql> <run your query>
mysql> SHOW PROFILES;
mysql> SHOW PROFILE FOR QUERY 2; (your query number, to get details)


Show tables by sorted order
---
SELECT table_name, engine FROM information_schema.tables WHERE table_type='BASE TABLE' AND table_schema='fwaas_1' ORDER BY table_name ASC;
SELECT table_name, engine FROM information_schema.tables WHERE table_type='BASE TABLE' AND table_schema='fwaas_1' AND (table_name LIKE '%master%' OR table_name like '%mapping%') ORDER BY table_name ASC;


Show warnings
---
SHOW WARNINGS;
SELECT @warning_count;


// Find all tables with specific column name in them
SELECT DISTINCT TABLE_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_NAME IN('allow_listed') AND TABLE_SCHEMA = 'fwaas_1';


mysql> select CURRENT_TIMESTAMP;
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2023-10-12 19:19:04 |
+---------------------+
1 row in set (0.00 sec)


// Upto 6 decimals precision
mysql> select CURRENT_TIMESTAMP(6);
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2023-10-12 19:19:04 |
+---------------------+
1 row in set (0.00 sec)


mysql> select now();
+---------------------+
| now()               |
+---------------------+
| 2023-10-12 19:22:38 |
+---------------------+
1 row in set (0.00 sec)


Resources:
[1] Udemy course on Database Design -
[2] Many Stackoverflow links

===============   PERFORCE  USEFUL  COMMANDS     =====================
Investigate these commands/flags?
---
p4 change -d 995890
p4 revert -a -c 995890
p4 revert -a //depot/saas/main/src/apps/orchestrator/lib/orchestrator.py
p4 sync -f //depot/path/file1.py#84 //depot/path/file2.py  <<< version?
p4 change -i
p4 edit -c 995890 //depot/path/file1.py //depot/path/file2.py



Setup Commands
---
P4CLIENT=bhaskar; export P4CLIENT
- All p4 commands will now use workspace: bhaskar

p4 client
    - Describes client workspace to Perforce server
    - Contains number of fields: Root and View are most imp
    Root: Topmost directory where all files are saved.
    View: Files in depot that are mapped to client workspace view

p4 sync     - syncs all files in the view from depot to your workspace
p4 sync ... - syncs only files in current directory from depot to workspace


Few commands to look in to
---
p4 workspace
p4 add, edit, delete, revert
p4 submit


// If you haven't checked out files that you edited, do following to get changes
// tracked by Perforce
p4 reconcile ...

//
p4 revert


Reporting Commands
---
p4 help commands
p4 help <command>
p4 help usage
p4 help views
p4 help
p4 info

p4 have
p4 sync -n



Typical Perforce Workflow [1]
---
1. Run `p4 edit` on each file you want to edit. You need to tell Perforce which files
you're editing. If you're adding new files, use `p4 add`. If you're deleting
files, use `p4 delete`.

2. Make your code changes.

3. Run `p4 change` to create a changeset. Here you can create a description of your
change and optionally add or remove files from your changeset too. You can run
`p4 change CHANGE_NUMBER` to edit the description later if necessary.

4. You can make additional code changes if you need to. If you need to
add/edit/delete other files, you can use `p4 {add,edit,delete} -c CHANGE_NUMBER
FILE`.

5. Run `p4 sync` to pull in the latest changes from the server.

6. Run `p4 resolve` to resolve any conflicts from syncing.

7. When you're ready to submit your change, run `p4 submit -c CHANGE_NUMBER`.


FAQ
===
Users Specific
---
1. How to see all changes submitted by a user
    p4 changes -s submitted -u bupadhyayula //depot/...    entire depot
    p4 changes -s submitted -u bupadhyayula ...            current repo
    p4 changes -s submitted -u bupadhyayula <filename>     just this file


Changelist Specific
---
1. Edit description of a changelist
    p4 change <changelist-number>


2. Show code changes in a changelist
    p4 describe <changelist-number>
    p4 describe -s <changelist-number>  Only description and changed files

3. Backing out submitted changelist
    Read this article --> https://community.perforce.com/s/article/3474

4. Changelists between certain dates
    p4 changes -s submitted //...@2011/05/16:00:01:00,2011/05/16:07:00:00

5. Add a file to Changelist
    p4 reopen -c <changelist-number> file    // can be used to move files between CL

6. Show code diff in changelist that's not submitted yet
    p4 describe $CHANGELIST | sed -ne 's:^\.\.\. \(.*\)#[0-9][0-9]* [a-z][a-z]*$:\1:p' | xargs p4 diff -du
    https://stackoverflow.com/a/28323794/630866


Files Specific
---
1. Show list of modified (but not committed) files
    p4 diff -f -sa
    p4 diff -f -sa ...

2. All the files that's modified but not committed
    p4 opened

3. All commits to a file
    p4 filelog <file-name>
        -l  long form description

4. How to get diff of all files in pending changelist
    p4 opened -c 999 | awk 'BEGIN { FS = "#" } // { print "p4 diff " $1 }' | csh

5. Add a file to Changelist
    p4 reopen -c <changelist-number> file    // can be used to move files between CL

6. Changes to a file between two versions
    p4 diff file#7 file#4


Merges
---
- Merging algorithm and notes:
  https://www.perforce.com/perforce/r14.2/manuals/p4guide/chapter.resolve.html


Others
---
1. Merge changes from one branch to another
   p4 integ -d -i ~/pan/$1/...@$3,@$3 ~/pan/$2/...
    $1 = source dir
    $3 = changelist number
    $2 = dest dir


Reference:
[1] https://stackoverflow.com/a/17306729/630866


===============   GIT  vs   PERFORCE             =====================

$ git checkout:
    The equivalent is 'p4 sync' in CLI and 'Get latest revision' from GUI.

$ git commit:
    p4 submit. But this operation does more. It combines 'git commit and push'

$ git push or pull:
    There's no equivalent in p4

Reference:
[1] https://stackoverflow.com/a/17331274/630866

===============   Jeff Bezos Talk with David Rubenstein   ============

On How He Operates:
--------
- He works on "Regret minimization framework". Projects himself to 80 years old
  and works backwards. Do things so you wont regret at 80 years.

- Executives make small but important decisions. Perhaps 3 decisions/day. Very
  imp they rest and are in best shape to make those decisions.

- High level executives must not be working on "current quarter". They must be
  working on things 2-3 years from now.

On Media:
--------
- On President Trump criticising media: Media is integral to success of
  democracy. It's freedom is in Constitution. Must not be vilified.

- Elected officials must accept that criticism is part of the job and is a good
  thing.

- Regulatory checks on big companies is good for the company.

https://www.youtube.com/watch?v=f3NBQcAqyu4


========================================================================
"Architectural Patterns of Resilient Distributed Systems" - Ines Sombra
https://www.youtube.com/watch?v=ohvPnJYUW1E

- Couldn't see full talk. My bad, I couldn't distill the knowledge from talk in
  to practical/usable knowledge for current needs.

===============     Python   Profiling    ==============================
Amjith Ramanujam
https://www.youtube.com/watch?v=QJwVYlDzAXs

- cProfile - included with standard library
    $ python -m cProfile lcm.py

- You can output results from cProfile in to a file and analyze using pstats
  (from standard library)
    $ python -m cProfile -o file.prof script.py

>>> import pstats
>>> p = pstats.Stats('file.prof')
>>> p.sort_stats('calls')
>>> p.print_stats(5)

- Can run primitive SQL like queries

- RunSnakeRun is GUI version of pstats

- Targeted Profiling, Hybrid Profiling, Instrumented

- Statistical Profilers
    * StatProf
        - CLI based, no GUI
    * PLOP (from Dropbox. Opensource)
        - Has GUI (D3 based)
    * NewRelic

- mem_profiler gives memory profiler. Above ones are CPU profilers. But very
  slow.

===============     Neocomplete Vim Plugin Notes      ==================
- What are the letters in []?
Link:
https://github.com/Shougo/neocomplete.vim/blob/master/doc/neocomplete.txt#L974

    file              -> [F] {filename}
    file/include      -> [FI] {filename}
    dictionary        -> [D] {words}
    member            -> [M] member
    buffer            -> [B] {buffer}
    syntax            -> [S] {syntax-keyword}
    include           -> [I]
    neosnippet        -> [neosnip]
    UltiSnips         -> [US]
    vim               -> [vim] type
    omni              -> [O]
    tag               -> [T]
    other sources     -> [plugin-name-prefix]


================     Hugo   CLI    Commands    ===========================
- Creating new posts
  hugo new posts/post-title.md

- Start server with drafts
  hugo server -D

- Publish your website
  hugo
        * Publishes your website to public/ folder


================     Public Speaking Training  ===========================

- Important elements in Public Speaking
    1 Preparation
    2 Presentation
    3 Style
    4 Content Delivery
    5 Audience Engagement

1 Preparation
    a Know Your Audience
        - What do they NEED, WANT and EXPECT?
    b Organize Your Content
        - Have structure, but don't chunk at each word.
        - There's difference between 'writing for the stage' and 'writing for
          the page'.
        - Look to influence, instead of informing.
            + 3 I's: Influence, Inform, Inspire.
    c Practice
        - More practice makes it appear natural and easy.


2 Presentation
    a Start presentation with a BANG, as we have only 7 seconds to make first
      impression. Don't start with needless intro.
    b Have "Hooks" in presentation called: "Attention Grabber". Ex:
        * Ask questions
        * Have analogy
        * Personal story
        * Can have humor/jokes, but make sure they are tasteful and relavant to
          topic.
        * Have quote, or some _shocking_ information and make sure to give
          citation.

    c Present ONE big idea.
    d Keep the momentum.
        - Every bit of info should build on previous and eventually lead to the
          one whole idea.
        - Use humor, anecdotes, story, insights, case study. Audience should
          recall it like a story.

3 Style
    a Be Your Authentic Self

    b Voice, Tonality, Pitch
        - Voice is your most important tool.
        - 5 kinds of voice: Pitch, Pace, Tone, Melody, Volume
            * High Pitch is considered weak personality. Low Pitch is taken
              seriously.
            * Monotone is fastest way to loose audience.
            * Create dynamics with change in volumes.

    c Articulation
        - Open your mouth while speaking.
        - Articulate every single sylable.

    d Body Language
        - 1960s Albert Mahebian studied how much of meaning comes from:
            * Visual, Vocal and Actual Words
            * 55% of meaning of message comes from what people see.
            * 38% from volume of message.
            * 7% from actual words.

        - Hands: Use your hands naturally.
        - Stance:
            * Stay in your stance, it's preferred.
            * If you have to move, move it with a purpose. That is, go over to a
              chart <<< I'm not so sure about this.
            * NEVER DO THIS: Rocking stance.

    e Energy
        - Abstract concept. But it's the most important thing.
        - Energy is contagious. If it is at right level, your audience will be
          with you.


4 Content Delivery
    a Be a Story Teller
        - Facts Tell, but Stories Sell.
    b Make It Emotional
    c Power of Pause
        - Use pauses to let audience absorb
        - Use sentence completion techniques where you stop halfway in a quote:
            * When going gets tough, the tough gets ...
            * Audience leans in to this.


5 Audience Engagement
    a Connect With Your Audience
    b Audience Participation
    c Incorporate Activities


6 Closing
    a Actionable Close. What action I want my audience to take?
    b Stick To Your Time.
    c End Speech on High Note.



Reference:
[1] https://www.youtube.com/watch?v=vB2pl1QbY3I


    ----------------------------------------------------------

Tip #1: Visualization
    - Visualize the room, giving the talk.

Tip #2: Dare to be Dull
    - Comes from Improv world.

Tip #3: Cognitive reframing technique
    - Instead of saying "I'm nervous, scared etc". Reframe it to "I'm excited,
      this is fun, ..." because on physical level, the chemecial reaction is
      same in both situation.

Tip #4: Have a Mantra that you repeat to yourself like: "I have value to add to this audience"
    - For one person it was "Last time this went well".
    - "It's not about me. It's about the content".

Tip #5: Instead of talking to the audience, facilitate the conversation.
    - A Google senior executive would always open the talk like:
        - "Good morning. Let's watch this 30s video".
        - Then they would ask what audience thought about it and go from there.

Tip #6: Normalizing public speaking anxiety.
    - An example of someone feeling better knowing everyone also feels anxious.
    - Not sure how this helps :)

- For most people, they are most anxious 1 min before and 1 min after start of talking.


Tip #7: Inhale and Long exhalation. Straw breath.
    - What your body feels, your mind thinks same thing. Example of a research
      about participants holding cold coffee and hot coffee and judging people.


Tip #8: Warm up before the talk
    - Athletes warm up before competition. Do same thing.
    - Matt does tongue twisters before a talk.



[1] How to speak confidently and persuasively. Matt Abrahams.
https://youtu.be/LpbBzmXrzEY


================     Bitcoin: Talk by Sarva    ===========================
- 3 problems posed by Cryptocurrencies
    * Currency creation rules
    * Counterfeiting
    * Double Spending Problem

- Centralization solved all 3 problems above.

Double Spending Problem
----
- Familiar problem in academic.
- A public database maintains ALL bitcoin transactions. Every new transaction is
  validated against this databse.

- The DB in bitcoin is called Blockchain. It's a linked list of blocks.

	┌─────────────────┐       ┌─────────────────┐                        ┌─────────────────┐
	│                 │       │                 │                        │                 │
	│  Block Header   │       │   Block Header  │                        │   Block Header  │
	│                 │       │                 │                        │                 │
	├─────────────────┤       ├─────────────────┤       .........        ├─────────────────┤
	│                 │       │                 │                        │                 │
	│   List of       │       │   List of       │                        │   List of       │
	│   Transactions  │       │   Transactions  │                        │   Transactions  │
	│                 │       │                 │                        │                 │
	│                 │       │                 │                        │                 │
	└─────────────────┘       └─────────────────┘                        └─────────────────┘
		 Block 0                   Block 1                                     Block N
	 (Genesis Block)



					 Block Header (80 bytes)
			┌──────────────────────────┐
			│       nVersion           │       4 bytes
			├──────────────────────────┤
			│     hashPrevBlock        │       32 bytes
			├──────────────────────────┤
			│     hashMerkleRoot       │       32 bytes
			├──────────────────────────┤
			│         nTime            │       4 bytes
			├──────────────────────────┤
			│         nBits            │       4 bytes
			├──────────────────────────┤
			│        nNonce            │       4 bytes
			└──────────────────────────┘


[1] Dr. Saravana Vijayakumar
https://www.youtube.com/watch?v=tySLqNScRIc



================     Udemy Video Player Keyboard Shortcuts   =============
Play/Pause      Space
Speed Slower    Shift + <--
Speed Faster    Shift + -->
Fullscreen      F
Exit Fullscreen ESC
Add Bookmark    B
Go back 5s      <--
Go forward 5s   -->
Volume up       up arrow
Volume down     down arrow


===============    AWS   Cognito       ======================

OAuth2.0
---
- OAuth2.0 is an open standard for authorization.
    * Users authorize on one website and access their info on another website
      but without giving their passwords/credentials.
- OAuth is not an API or service but open standard.
- OAuth authorizes devices, APIs, Servers, Applications with "access tokens
  rather than credentials".


OpenID Connect (OIDC)
---
- OIDC vs OAuth2
    * OAuth2 = broad spec for authorization
    * OIDC = spec for identity on top of OAuth2
- Tokens
    * JSON Web Tokens (JWT)
    * ID (Identity claims)
    * Access Tokens (bearer token used to access resources)
    * Refresh Tokens (to obtain a new access token). Valid for 30 days.

Dissecting JWT Token (valid for 1 hour after issuing).
---
    Header
        kid:
        alg:
    Payload
        bunch of attributes
    Signature
        signature containing payload. Used for verification that token worked.


AWS Cognito
---
- Cognito gives 3 types of tokens (Access, Identity and Refresh).
- Cognito gives 3 types of authorization
    a) User Pools --> User Pool Authorizers
        - When you want simple Yes/No for user authorization
        - Valid or Invalid user

    b) Federated Identities --> AWS IAM authorization
        - IAM based authorization for fine grained access.
        - IAM makes use of Federated Identities to provide IAM policy for the
          user.

    c) Custom Identity Providers --> Custom Authorizers


- Most secure of the 3 is IAM based authorization as _every_ request is signed.
- With other forms of authorization, if header can be read and token is taken,
  impersonation can be done for duration of that tokens validity.



References:
https://www.youtube.com/watch?v=VZqG7HjT2AQ

===============    AWS   API Gateway   ======================
- HTTP API vs REST API
    * HTTP API is inexpensive than REST API
    * 70% less cost
    * 60% faster than p99. Not clear when compared to REST API.
    * <10ms per API request. Must faster than REST API.


- API Gateway Management
    * AWS Management Console
    * AWS CLI
    * AWS Serverless Application Model (SAM) <<< built on top of CloudFormation
    * AWS CloudFormation
    * Swagger/OpenAPI   <<< industry standard
    * AWS Cloud Development Kit (CDK)   <<< programatic way of doing

- CORS meaning (at 16 min). Cross-Origin Resource Sharing
    * If you don't know this, you can open up security and make mistakes.


-

Reference:
[1] I didn't know API GW can do that
    * https://www.youtube.com/watch?v=yfJZc3sJZ8E


===============    AWS CDK Concepts    ======================
Constructs:

Apps:

Stacks:

Environments:
- Environment Variables
    - Using hard coded values, passed as dictionaries
    - Using environment variables and access from CDK using os.environ[]
    - Using shell scripts

Parameters: Custom values supplied at deployment time and incorporated into CF
template.
    - Generally not recommended ....
    - Use following in CLI
        $ cdk deploy MyStack --parameters MyStack:ParameterName=ParameterValue

Context:
    - 6 ways of doing


===============    AWS CloudTrail    ======================
- What can you answer using a CloudTrail event?
    - WHO made the API call?
        - Detailed info for identity types
            - Root user, IAM user, Federated User, Role
        - Info includes
            - Friendly username
            - AWS AccessKeyId
            - 12 digit AWS account number
            - ARN
            - Session context and issuer info
            - invokedBy identifies AWS Service making request on behalf of user

    - WHEN was the API call made?
        - "eventTime": "2013-10-23T23:30:42Z"
        - ISO 8601 format

    - WHAT was the API call?
        - API call name and service the API call belongs to
        - "eventName": "RunInstances"
          "eventSource": "ec2.amazonaws.com"

    - WHICH resources were acted up on in the API call?
    - WHERE was the API call made from and made to?

- S3 logs can be pushed to Cloud Athena and run SQL queries on them.


[1] Turn on CloudTrail: Log API Activity in Your AWS Account
    * https://www.youtube.com/watch?v=BJprWgompq0
[2] How can I use CloudTrail to API calls and actions
    * https://www.youtube.com/watch?v=4ztTv5rIRv8


===============    CMU Databases Systems     ================
Lec 1:

Data Manipulation languages (DML)
---
- Procedural: Query specifies the (high-level) strategy the DBMS should use to
  find the desired result. Relational Algebra
- Non-Procedural: The query specifies only what data is wanted not how to find
  it. Relational Calculus

Relational Algebra
---
R ∪ S = (SELECT * FROM R) UNION ALL (SELECT * FROM S)
R ∩ S = (SELECT * FROM R) INTERSECT (SELECT * FROM S)
R - S = (SELECT * FROM R) EXCEPT (SELECT * FROM S)
R x S = SELECT * FROM R CROSS JOIN S;
        SELECT * FROM R, S;
R S = SELECT * FROM R NATURAL JOIN S;

Rename, Assignment, Duplicate Elimination, Aggregation, Sorting, Division
operators

==================================================================
Lec 2:

Using following 3 tables for explanation
    * STUDENT(sid, name, login)
    * COURSE(cid, name)
    * ENROLLED(sid, cid, grade)

Aggregates:
    * Functions that return a single value from a bag of tuples.
    * Aggregate functions can only be used in the SELECT output list.
---
AVG(col) - Return the average col value.
MIN(col) - Return the minimum col value.
MAX(col) - Return the maximum col value.
SUM(col) - Return the sum of all values in col.
COUNT(col) - Return the # of values for col.


- Get # of students with a "@cs" login:
    SELECT COUNT(login) AS cnt FROM student WHERE login LIKE '%@cs%';
    SELECT COUNT(*) AS cnt FROM student WHERE login LIKE '%@cs%';
    SELECT COUNT(1) AS cnt FROM student WHERE login LIKE '%@cs%';

- Can have multiple aggregates in single statement:
    SELECT AVG(gpa), COUNT(id) FROM student WHERE login LIKE '%@cs%';

- DISTINCT aggregates are possible: COUNT, SUM, AVG support DISTINCT
    SELECT COUNT(DISTINCT login) FROM student WHERE login LIKE '%@cs%';


- Output of other columns along with aggregate is undefined.
    SELECT AGV(s.gpa), e.cid
      FROM enrolled AS e, student AS s
    WHERE e.sid = s.sid

    # Most DBs return an error.
    # SET SESSION sql_mode='traditional' <<< returns random value for e.cid
    # SET SESSION sql_mode='ansi'  <<<< returns error


GROUP BY:
--------
- Project tuples in to subsets and calculate aggregates against each subset
    SELECT AVG(s.gpa), e.cid
      FROM enrolled AS e, student AS s
    WHERE e.sid = s.sid
    GROUP BY e.cid;
    # This will group tuples by e.cid. Then computes average on those groups (or
      subsets)

- Non-aggregated values in SELECT clause output must appear in GROUP BY clause.
    SELECT AVG(s.gpa), e.cid, s.name
      FROM enrolled AS e, student AS s
    WHERE e.sid = s.sid
    GROUP BY e.cid, s.name;
    # This won't work


HAVING:
-------
- Following won't work because WHERE clause does filtering. Aggregation follows
  that. Therefore, "avg_gpa > 3.9" cannot be in WHERE clause.
    SELECT AVG(s.gpa) AS avg_gpa, e.cid
      FROM enrolled AS e, student AS s
    WHERE e.sid = s.sid
      AND avg_gpa > 3.9
    GROUP BY e.cid;

- This will work. HAVING clause is similar to WHERE clause. It filters on output
  from previous SELECT
    SELECT AVG(s.gpa) AS avg_gpa, e.cid
      FROM enrolled AS e, student AS s
    WHERE e.sid = s.sid
    GROUP BY e.cid
    HAVING avg_gpa > 3.9;

- In declarative languages like SQL, query optimizer knows there is a HAVING
  clause. So, it optimizes output of SELECT and only has tuples with avg > 3.9


STRING OPERATIONS
----
                              STRING OPERATIONS
          ───────────────────────────────────────────────────────────
              DB             String Case            String Quotes
          ───────────────────────────────────────────────────────────
            SQL-92           Sensitive              Single Only
            Postgres         Sensitive              Single Only
            MySQL            Insensitive            Single/Double
            SQLite           Sensitive              Single/Double
            DB2              Sensitive              Single Only
            Oracle           Sensitive              Single Only


- LIKE clause.
    * '%' matches any substring, including empty strings.
    * '_' matches any one char.

- SUBSTRING, UPPER, LOWER. Can be used  anywhere: SELECT, WHERE, HAVING etc.

- Concatenation operation:
    SQL-92:
    SELECT name FROM student
    WHERE login = LOWER(name) || '@cs';

    MySQL:
    SELECT name FROM student
    WHERE login = LOWER(name) + '@cs';

    Postgres:
    SELECT name FROM student
    WHERE login = CONCAT(LOWER(name), '@cs');


DATE/TIME OPERATIONS
----
- Details get weird between DBs


OUTPUT REDIRECTION
----
- Store query results in another table:
    * Table must not already be defined.
    * Table will have same # of columns with same types as the input.

    SELECT DISTINCT cid INTO CourseIds FROM enrolled;
    is same as following:
    CREATE TABLE CourseIds (SELECT DISTINCT cid FROM enrolled);
    or
    INSERT INTO CourseIds (SELECT DISTINCT cid FROM enrolled);


OUTPUT CONTROL
----
- ORDER BY <column*> [ASC|DESC]

    SELECT sid FROM enrolled
      WHERE cid = '15-721'
    ORDER BY grade DESC, sid ASC; << note that grade is not part of output.


- LIMIT <count> [offset]
    * Can set an offset to return a "range".

    SELECT sid, name FROM student
      WHERE login LIKE '%@cs%'
    LIMIT 20 OFFSET 10



        ADVANCED  SQL
        =============

NESTED QUERIES
----
- Usually JOINs are better option.
	* SELECT name FROM student WHERE sid IN (SELECT sid FROM enrolled)

- Get names of students enrolled in '15-445':
    a) SELECT name FROM student WHERE sid = ANY(
            SELECT sid FROM enrolled WHERE cid = '15-445'
        );

    b) SELECT (SELECT S.name FROM student AS S WHERE S.sid = E.sid) AS sname
        FROM enrolled AS E
       WHERE cid = '15-445';

    c) SELECT name FROM student
         WHERE sid IN (
           SELECT sid FROM enrolled
             WHERE cid = '15-445'
         )

- Some other operators to use in nested queries:
    ALL -> Must satisfy expression for all rows in sub-query
    ANY -> Must satisfy expression for at least one row in sub-query
    EXISTS -> At least one row is returned
    IN -> Equivalent to '=ANY()'


- Some complex nested queries:
- Find student record with the highest id that is enrolled in at least one
  course.

    // This is wrong
    SELECT MAX(e.sid), s.name FROM enrolled as e, student AS s
    WHERE e.sid = s.sid

    // Some logic like this
    SELECT sid, name FROM student WHERE sid "is greater than every"
       (SELECT sid FROM enrolled)

    // Solution 1
    SELECT sid, name FROM student WHERE sid => ALL (SELECT sid FROM
    enrolled)

    // Solution 2
    SELECT sid, name FROM student WHERE sid IN (SELECT MAX(sid) FROM
    enrolled)

    // Solution 3
    SELECT sid, name FROM student WHERE sid IN (SELECT sid FROM enrolled
    ORDER BY sid DESC LIMIT 1)


- Find all courses that has no students enrolled in it.
    //My solution
    SELECT * FROM course WHERE cid NOT IN (SELECT DISTINCT cid FROM enrolled);

    // Professor solution
    SELECT * FROM course WHERE NOT EXISTS (SELECT * FROM enrolled WHERE
    course.cid = enrolled.cid)


WINDOW FUNCTIONS
----
- Performs a 'sliding' calculation across a set of tuples that are related.
- Like aggregation but tuples are not grouped into a single output tuple.

- Syntax looks like this
    SELECT ... FUNC-NAME(...) OVER(...) FROM table-name
                  ^            ^
                  |            |
                  |            +------ How to 'slice' up data. Can also sort.
                  |
                  +------- Aggregation Functions or Special Functions.

- Special window functions:
    * ROW_NUMBER() -> # of current row.
    * RANK() -> Order position of the current row in a sorted output. If output
      is not sorted on some column, every row has rank 1.

Ex:
    SELECT *, ROW_NUMBER() OVER() AS row_num FROM enrolled
    SELECT cid, sid, ROW_NUMBER OVER (PARTITION BY cid)
      FROM enrolled ORDER BY cid

- ORDER BY can also be used for grouping
    SELECT *,
        ROW_NUMBER() OVER (ORDER BY cid)
    FROM enrolled
    ORDER BY cid


- Find student with highest grade for each course.
    SELECT * FROM (
        SELECT *,
                RANK() OVER (PARTITION BY cid       << group tuples by cid
                             ORDER BY grade ASC)    << then sort by grade
                AS rank
           FROM enrolled) AS ranking
    WHERE ranking.rank = 1  <<< pick rank value 1


- TODO: What's the difference between ORDER BY vs PARTITION?



COMMON TABLE EXPRESSIONS (CTE)
----
- Provides a way to write auxiliary statements for use in larger query.
    * Think of it like temp table just for one query.

- Alternative to nested queries and views.

    WITH cteName AS (
        SELECT 1
    )
    SELECT * FROM cteName

- You can bind output columns to names before the AS keyword.

    WITH cteName (col1, col2) AS (
        SELECT 1, 2
    )
    SELECT col1 + col2 FROM cteName

- Find student record with the highest id that is enrolled in at least one
  course.

    WITH cteSource (maxId) AS (
        SELECT MAX(sid) FROM enrolled
    )
    SELECT name FROM student, cteSource
      WHERE student.sid = cteSource.maxId;


- How is CTE different than Nested Queries?
    * You can do recursion.

    * Print sequence of numbers from 1 to 10
    WITH RECURSIVE cteSource (counter) AS (
        (SELECT 1)
        UNION ALL
        (SELECT counter + 1 FROM cteSource
            WHERE counter < 10)
    )
    SELECT * FROM cteSource


==================================================================
Lec 3: Storage
https://15445.courses.cs.cmu.edu/fall2019/slides/03-storage1.pdf

Access Times
---
        0.5 ns  L1 Cache Ref        0.5 s
          7 ns  L2 Cache Ref        7 s
        100 ns  DRAM                100 s
    150,000 ns  SSD                 1.7 days
 10,000,000 ns  HDD                 16.5 weeks
~30,000,000 ns  Network Storage     11.4 months


Buffer Pools: In memory process(?) that takes DB files on disk and bring them
back and forth.
- DB's do not rely on underlying OS for file management.
- This is because OS may not have knowledge about application symantics. It
  treats all applications same. DB may stall on page faults in virtual mem.

- There are some solutions
    * madvise: Tell OS how you expect to read certain pages.
    * mlock: Tell OS that memory ranges cannot be paged out.
    * msync: Tell OS flush memory ranges out to disk.


Storage Manager: Responsible for maintaining DB files (same as Buffer Pools?) as
"Pages".

DB Pages
---
- A "page" is fixed-size block of data
    * It can contain tuples, meta-data, indexes, log records ...
- Each page has unique id
    * DBs uses indirection layer to map page ids to physical location.
- 3 Notions of Page
    * Hardware Page (4K)
    * OS Page (4K)
    * Database Page (512B - 16KB)

- In PostgreSQL
    * SELECT r.ctid, r.* FROM r;
    * ctid gives (page number. offset) result
    * VACUUM FULL;  This cleans up and compacts tuples (like defragmentation).

- Slotted Page arrangement is one form of file organization on disk.

Lecture 4
====
- Log Structured file systems
    * Each SQL statement is logged and written to disk.
    * Sequential writes are faster than sequential reads.
- Reading is drawback.

- IEEE-754 says how to store numbers in memory.

- FIXED PRECISION NUMBERS
    * Used when rounding errors are unacceptable
    * These are numeric data types with arbitrary precision and scale.

- How is very large values stored?
    * Most DBs don't allow value to exceed single page.
    * DBs use overflow storage pages.
        - Postgres. TOAST (>2KB)
        - MySQL. Overflow (>1/2 size of page)
        - SQL Server. Overflow (> size of page)

- External Value Storage
    * Some DBs let you store large value in external file.
    * BLOB type. Binary large object
    * DB cannot manipulate contents of external file.
        - No durability or transaction protections.
    * No rule on how big value must be. DB implementation specific.
        - 256KB is rough guidance.
        - Less -> overflow pages
        - More -> external storage.

- System Catalogs
    * Meta data, index, views, stats etc about DBs.
    * INFORMATION_SCHEMA is an ANSI standard all DBs implement.

- SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE table_catalog = <db_name>;
    * Following are short cuts to same command
    \d;             PG
    SHOW TABLES;    MySQL
    .tables;        SQLite

- DESCRIBE INFORMATION_SCHEMA.TABLES

- OLTP - Online Transaction Processing
    * Simple queries reading/writing small data.
    * Ex: Amazon store front.

- OLAP - Online Analytical Processing
    * Complex queries reading large portions of DB spanning multiple entities.
    * Execute queries on data collected from OLTP apps.

- HTAP - Hyper Transaction Analytical Processing
    * In between OLAP and OLTP

- N-ary Storage Model (NSM)
    * DBMS stores all attributes for single tuple contiguously in a page.



==============================




@@@@@@@@@@@@@@@       AWS  DYNAMODB          @@@@@@@@@@@@@@@@@
What is it?
- Fast and flexibe NoSQL DB
- Fully managed. Good for document or Key-Value storage models
- Good for gaming, web, mobile, advertising tech etc

Benefits
- Stored on SSD. So, high disk throughput.
- Saved in 3 geo-locations. High availability.
- Suited for read heavy apps.

Read Consistency. 2 Options.
    - Eventually Consistent reads (default)
    - Strongly consistent reads

- DDB is charged based on 'Provisioned Throughput Capacity'
    - Write throughput cost: There is cost for
        10 units of write capacity = 36000 writes per hour
    - Read throughput cost: There is cost for
        50 units of read capacity = 180000 "strongly consistent" reads per hour
                                       OR
                                    360000 "eventually consistent" reads per hour
- There's also storage costs per GB



	             SQL                        NoSQL
	┌────────────────────────────┬───────────────────────────────┐
	│  Optimized for Storage     │     Optimized for Compute     │
	│                            │                               │
	│  Normalized/Relational     │    Denormalized/Hierarchical  │
	│                            │                               │
	│  Adhoc queries             │    Instantiated Views         │
	│                            │                               │
	│  Scale vertically          │    Scale horizontally         │
	│                            │                               │
	│  Good for OLAP             │    Good for OLTP at scale     │
	└────────────────────────────┴───────────────────────────────┘

Partition Key and Sort Key:
- It partitions data. Mandatory key.
- It's a hash index. Points where data is stored.
- There can be infinite items in this partition.
- Sort key sorts these items within this partition.


Local Secondary Index (LSI):





Reference:
[1] SimpliLearn 10 mins intro:
https://www.youtube.com/watch?v=2mVR_Qgx_RU
[2] re:Invent. DynDB Deep Dive. Rick Houlihan






@@@@@@@@@@@@@@@       AWS  NETWORKING  FUNDAMENTALS      @@@@@@@@@@@@@@@@@
Regions:
    - Connected by dedicated multiple 100 Gigabit parallel lines.
    - AWS Region is a geographical area, physically isolated from other AWS
      Regions, but connected via the backbone.
    - AWS China is exception.

Availability zones (AZ)
    - Close enough within a region to do synchronous replication with databases
    - Inside each AZ, there's one or more datacenters

Virtual Private Cloud(VPC)
    - VPC exists within a region and covers all AZ
    - Each VPC can have multiple subnets per AZ
    - Two types of subnets: Public and Private
    - EC2 instances must be placed within a subnet
    - Few services to get in and out of VPC.
        - VPC Peering connection
        - Internet Gateway
        - VPN Gateway
        - VPC Endpoints
        - Customer Gateway
        - NAT Gateway
        - AWS Transit Gateway

    - Assign Private CIDR ranges to VPC (IPv4)
        - 192.168.0.0/16
        - 172.16.0.0/12
        - 10.0.0.0/8
        - Largest CIDR range is /16 and smallest is /28
    - From this range, assign IP address ranges to subnets


IPv6 on AWS
    - You do not get to choose the range. AWS does it for you.
    - /56 VPC. AWS assigns from their range
    - /64 subnets. You can choose this
    - Your VPC will now run Dualstack (v4 and v6)
    - Link Local address and Global Unicast Address is required

"5 Things" required for Internet Traffic
    1) Public IP Address
    2) Internet Gateway attached to a VPC
    3) Route to an Internet Gateway inside EC2 instance Route Table
    4) NACL allow rule
    5) Security Group allow rule

1) Public IP Addresses for your EC2 instances
    - Auto-Assign public IP addresses (both IPv4 or v6)
    - Elastic IP Addresses (EIP) Pool

    - IPv6: AWS can auto-assign public IP address for IPv6 from the VPC CIDR range
    - IPv4: AWS can *find one* public IPv4 from it's own public IPv4 pool and
      assign to instance
        - AWS has a public IPv4 address range
        - OR we can bring our own public IPv4 address range (BYOIP Pool)

2) Internet Gateway(IGW)
    - IGW provides way for VPC to connect to Internet
    - Also acts as NAT between private IP and public IP within the VPC
    - On AWS console, after creating IGW, we must attach VPC to it

3) Add entries in route table.
    - In AWS, "Route Table" is associated with a Subnet.
    - "Route Table" defines behavior of packets originating in that Subnet.
    - Ex:
        Destination             Target
        172.31.0.0/16           local   <<< local means implicit router/LAN
        2600:1f16:14d::/56      local
        0.0.0.0/0               igw-09ef...
        ::/0                    igw-09ef...

    - Egress IGW: When you want traffic orignating in your VPC to go out, but
      not traffic originating outside to come in.
    - Public Subnet: Route table has entry to IGW
    - Private Subnet: All traffic stays within VPC

	- If instance in Private Subnet needs to download something from Internet, what
	  to do?
		- Have a NAT in public subnet that talks to Internet Gateway
		- Have a route in route table in instance in Private Subnet
		- This route points to NAT in Public Subnet

Network Security
	a) Network ACLs
	b) Security Groups
	c) VPC Flow Logs
	d) Amazon VPC Traffic Mirroring

4) NACLs:
	- Defines what traffic flows are allowed/denied between subnets in a VPC
	- Stateless. ALLOW and DENY statements.
	- Statements are evaluated in order we define.
	- By default, NACLs in AWS allows all traffic in and out of Subnets.

5) Security Groups
    - Operates with ALLOW statements
    - Provides a STATEFUL FIREWALL (what? a Firewall?)
    - Wraps around Network Interfaces of EC2 instances (ENIs).
	- They can span across Subnets and AZs.
    - But I'm not sure diff between SG and (NACLs + Routes in Route Table)


VPC Flow Logs
    - Provides Netflow, IPFix like data through Cloudwatch Logs
    - These logs do not impact bandwith, throughput or latency
    - Apply these logs to VPC, Subnet or ENI

Packet Capture
    - Some options available

Elastic Load Balancing (ELB)
    - Application LB (HTTP and HTTPS traffic)
        - L7, HTTP, HTTPS
        - Host and Path based routing
        - Supported Targets: EC2, Containers, AWS Lambda, Private IP
    - Network LB (TCP, UDP and offloading TLS traffic)
        - IPv4, L4
        - Supported Targets: EC2, Containers, Private IP
    - Classic LB (Deprecated)

ELB Components:
	1) Any LB has a listener: Port and Protocol on which LB is listening
	2) Listener Rules: on how to route traffic
		- Ex: Path based rule: if path is "/img/*", forward this IP address
		- They identify Target Groups containing Targets


Connecting VPCs
- One way to connect VPC is put IGW in each VPC and they talk over public
  internet. But use ... VPC peering.
- VPC Peering can work in same region, across regions and across accounts too.

- VPC Peering within same region is easy
    - Setup peering in Console. VPC Peering starts as pcx-....
    - Update Route table. Destination IP range must go through VPC Peering
    - Peer must accept traffic from other VPC
    - Each VPC can have multiple Peering
    - You cannot peer to 3rd VPC via a 2nd VPC. Peering has to be direct to VPC
      your VPC wants to talk to. Not transitive VPC Peering.

- Same AWS Console can be used for cross region and cross account VPC Peering.

CAN and CANNOT of VPC Peering:
- CAN reference Security Groups from peer VPC.
- CAN enable DNS hostname resolution to return private IP address.
- CAN peer for both v4 and v6 traffic.

- CANNOT have overlapping IP addresses.
- CANNOT have multiple peers between same pair of VPC.
- CANNOT use Jumbo frames across inter-region VPC peering.


Connecting to On-Premises Network
--------
1) Site-to-Site VPN - VGW
    - A Virtual Private Gateway (VGW) must be configured on AWS side.
    - Must give Autonomous System Number (ASN). It identifies the network on AWS
      side.
    - On Customer side, a Customer GW must be configured in AWS Console.
        - Indicate Static or Dynamic routing. Static means fixed IP address.
          Dynamic means BGP based.
        - Can also give Certificate based VPN
        - Not all things are clear for me here.
    - AWS creates 2 endpoints in 2 AZs, connecting to CGW. Each endpoint is
      1.25Gbps. AWS uses only 1 connection and other as failover.

2) AWS Direcct Connect - Physical Connection
    AWS Network <----> AWS Router <----> Customer Router <----> On-Prem Network
    - Above is representation of physical connections
    - 3 types of Interfaces on top of above physical connections
        - Private VIF: to connect to VPCs using private IPs, directly or via
                       Direct Connect GW
        - Transit VIF: to connect to AWS Transit GWs via Direct Connect GW
        - Public VIF: to access all AWS public services using public IP
                      addresses

    - All VIFs are 802.1q VLANs with BGP Peering
        - A VLAN is build from Customer Router to Direct Connect GW
        - VGW on VPCs connect to Direct Connect GW
        - See picture at 41:21s to understand


AWS Transit Gateway
- Transit GW sits in middle of VPCs and attaches to them
- Can connect 1000s of VPCs
- AWS Transit GW can be used in On-Prem use case. It'll have 2 tunnels to
  Customer GW. Route tables in VPCs and TGW must be updated with correct
  entries. Time 44:20s
- With Direct Connect GW (DX GW), a Transit VIF is created in AWS Router in
  below diagram
    AWS Network <----> AWS Router <----> Customer Router <----> On-Prem Network

    - Transit VIF attaches to DX GW, which attaches to TGW. Time: 44:50


Name Resolution (DNS): Amazon Route 53 Resolver


Reference:
[1] AWS Networking Fundamentals re:Invent talk


@@@@@@@@@@@@@@@       AWS  X-Ray      @@@@@@@@@@@@@@@@@

Concepts:
--------
Trace - end-to-end data related to single request across services
Segments - Portion of trace corresponding to single service.
Sub-Segments - Remote call or local compute sections within a service
Annotations - Business data that can be used to filter traces
Metadata - Business data that can be added to the trace but not used for
           filtering traces
Errors - Normalized error message and stack trace
Sampling - Percentage of requests to be captured as traces


Sampling Configuration:
    - Used to configure which services to trace and lot more (TBD)

- A subset of requests must be traced
- Best practice: Trace all errors

X-Ray can be used Off-Cloud, any cloud, not just on AWS

Service Map
Traces

X-Ray SDK

X-Ray Daemon
    Runs locally, collects traces over UDP from your application and sends to
    X-Ray service every second or when buffer fills up.

X-Ray API
    PutTraceSegments
    BatchGetTraces
    GetServiceGraph
    GetTraceSummaries

Code
----
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
from aws_xray_sdk.core import patch   <<<< Look into this
from aws_xray_sdk.core.context import Context

xray_recorder.configure()
xray_recorder.configure(
    sampling=False,
    sampling_rules=SAMPLING_RULE_JSON,
    context_missing='LOG_ERROR',
    service='Name of service',
    context=Context(),
    streaming_threshold=0)
)

context_missing – Set to LOG_ERROR to avoid throwing exceptions when your
                  instrumented code attempts to record data when no segment is open.

daemon_address – Set the host and port of the X-Ray daemon listener.

service – Set a service name that the SDK uses for segments.

plugins – Record information about your application's AWS resources.

sampling – Set to False to disable sampling. This disables sampling and global
           recorder is enabled. ALL incoming requests are recorded

sampling_rules – Set the path of the JSON file containing your sampling rules.
sampling_rules=<rules>
    <rules> is either dictionary of rules or absolute path to JSON file
    containing sampling rules

xray_recorder.begin_segment()
xray_recorder.end_segment()
xray_recorder.begin_subsegment()
xray_recorder.end_subsegment()

xray_recorder.current_subsegment()
xray_recorder.record_subsegment(    # Look at examples on https://grep.app
    ...
)

xray_recorder.put_annotation(key, value)
xray_recorder.put_metadata(key, dict, value)

Using decorator
@xray_recorder.capture('some name')
def function():
  ...


Sample Code:
https://github.com/aws/aws-xray-sdk-python/blob/master/tests/test_recorder.py


Reference:


@@@@@@@@@@@@@@@       Sourcegraph  Search   @@@@@@@@@@@@@@@@@@
Three types of search
    - Literal
    - Regular Expression
    - Structural
They can be changed in search bar, by clicking one of the 3 small icons




@@@@@@@@@@@@@@@       AWS  CloudFormation   @@@@@@@@@@@@@@@@@@

- Flow of CFT

    Template --------> S3 Bucket ---------> AWS CloudFormation ------> Stack -----> AWS Resources

- Stack names are unique per region

YAML Crash Course
- Key, Value pairs
- Nested Objects
- Supports Arrays
- Multi line strings (not possible in JSON), using a Pipe bar
- Comments (not possible in JSON)


CFT Resources
=============
Following 3 are required
    - Name
    - Type
    - Properties
        - AWS docs on properties have Update property
        - 3 types of Update: 'No Interruption', 'Some Interruption', 'Replacement'



CFT Stack Creation Options
==========================
- Rollback on failure
- Timeout
- Termination protection


CFT Designer
============
- Helps build CFT visually

CFT Building Blocks
===================
Template Components
    AWSTemplateFormationVersion
    Description
    Transform: One or more macros used to process Template
    Metadata
    Resources (MANDATORY)
    Parameters: Dynamic inputs for your Template
    Mappings: Static inputs for your Template
    Outputs: References to what has been created
    Conditionals: List of conditions to perform resource creation
    Rules: validate a parameter(s) during stack create/update

Template Helpers
    References
    Functions: Transform *some inputs* to templates

Cost Estimates
==============
- In CFT section, for a Template, there's a 'Estimate Cost' link. This takes
  you to Simple Monthly Calculator


Parameters: A way to provide input to CFT
    - Powerful, controlled and can prevent errors, thanks to 'Types'
    - Can be cross validated using Rules

Parameter Settings
    Type: String, Number, CommaDelimitedList, List<Number>, AWS-Specific Param.
          List<AWS-Specific Param>, SSM (get Param from SSM Parameter store)
    Description
    ConstrainedDescription: Min/Max length, Min/Max value, Default value,
                            Allowed Values, NoEcho

How to reference parameters?
    The Fn:Ref to reference parameters

# Look at CFT notes for various types
Parameters
    SomeParameterName:
        Description:
        Type: String
        MinValue: 1150
        MaxValue: 65535
        Default: t2.small
        AllowedValues:
          - t1.micro
          - t2.nano
          - t2.micro
          - t2.small
        ConstraintDescription: must be a valid EC2 instance type.
        NoEcho: true
        AllowedPattern: (\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})


SSM Parameter Type
- Reference parameters in Systems Manager Parameter Store
- Example: Fetch latest AMI IDs

Resources
- Core of CFT (mandatory field)
- Represent different AWS Components/Services
- Resources are declared and can be referenced
- AWS figures out creation, update, deletion
- There are > 700 types of resource
    AWS::aws-product-name::data-type-name



@@@@@@@@@@@@@@@       Pacal vs Camel vs Snake Case         @@@@@@@@@@@@@
Pascal Case = PascalCase
    Used for Class names

Camel Case = camelCase
    Used for variable names in classes, method names,

Snake Case = snake_case
    Used in scripting languages, for file names

Kebab Case = kebab-case

@@@@@@@@@@@@@@@   Beancount Double-Entry Accounting    @@@@@@@@@@@@@


- Beancount, double-entry accounting.
    - Account must start with one of the following
        - Assets
        - Liabilities
        - Income
        - Expense
        - Equity

    - Assets = What you own
    - Liabilities = What you owe
    - Equity = What you own *before* you started
    - Income = What you earn
    - Expenses = What you spend
    ---> Asset = Liabilities + Equity + Income - Expenses

    - Rules of the game
        - "Debit" increases an asset account.
        - "Credit" decreases an asset account.
        - "Liability" accounts are reveresed.
        - "Transaction" is Credit or Debit with a timestamp.
        - All transactions will sum to zero (they will "balance").
        - Double-Entry Accounting = Credit from one account, Debit into another account.
        - Income and Expense are accounts in their own right, with balances.
        - Special account type "Equity" is used when getting started.
        - Each account handles only *one* type of currency.



HOW TO BOOKKEEPING?
    - Step 1: Take a complete inventory of *everything* you own or owe.
    - Step 2: Everytime you move money from one place to another, write it
      down. Every transaction must balance to zero.
    - Step 3: Keep doing Step 2 forever.



@@@@@@@@@@@@@@@          KQL (KUSTO QUERY LANGUAGE)      @@@@@@@@@@@@@
- Best Practice:
    - Always `limit`
    - Always use `timestamp` or range

- This tutorial is primarily to use KQL to query DBs in Azure.

- Azure Data Explorer Cluster
    - It's a place to connect various DBs in Azure and query them using KQL
- https://web.kusto.windows.net/clusters
    - A place to play around with KQL
    - There's sample DBs there.
- Azure Data Studio is another tool to run KQL queries.

- Commands:
    .show databases
    .show database
    .show cluster databases
    .show database schema
    .alter database prettyname
    .execute database script

- ABANDONED [1] after 3 videos. Not useful to me at this point.

    SecurityEvent
    | take 10


    TimeGenerated is automatically added by Azure on the logs.
    SecurityEvent
    | where TimeGenerated >= ago(1d)

    SecurityEvent
    | where TimeGenerated >= now(-15m)

    SecurityEvent
    | where TimeGenerated >= now(-15m)
    | project TimeGenerated, Account, Computer, EventID

    SecurityEvent
    | where TimeGenerated >= now(-15m)
    | where EventID == 4625
    | project TimeGenerated, Account, Computer, EventID

    SecurityEvent
    | where TimeGenerated >= now(-15m) and EventID == 4625
    | project TimeGenerated, Account, Computer, EventID

    SecurityEvent
    | where TimeGenerated >= now(-15m) and EventID == 4625
    | project TimeGenerated, Account, Computer, EventID
    | summarize FailedLogons = count() by Computer

    SecurityEvent
    | where TimeGenerated >= now(-15m) and EventID == 4625
    | project TimeGenerated, Account, Computer, EventID
    | summarize FailedLogons = count() by Computer
    | order by FailedLogons

    #  following chart can be pinned to a dashboard.
    SecurityEvent
    | where TimeGenerated >= now(-15m) and EventID == 4625
    | project TimeGenerated, Account, Computer, EventID
    | summarize FailedLogons = count() by Computer
    | order by FailedLogons
    | render piechart

- Filter by `where` clause as soon as possible.
- Filter by timestamp (`TimeGenerated`) quickly.

    Perf
    | count

    Perf
    | where ObjectName == "System"
    | where CounterName == "System Up Time" or CounterName == "Uptime"
    | extend UpTime = CounterValue * 1s
    | project  TimeGenerated, Computer, UpTime, InstanceName
    | summarize arg_max(TimeGenerated, *) by Computer
    | order by UpTime desc

- There's a hint, to read help page of a keyword. Click the '>' in the suggestion box.

    SecurityEvent
    | where Channel == 'Security'
    | distinct Activity


- `summarize` is group by a certain field.
- `getschema` tells fields and their datatypes.

- JOINS
    - innerunique (default)
        - Get all records, but ignore multiple records on the key/value residing on left table.
        - T1 | join (T2) on CommonColumn

        Table 1                 Table 2                 Merged Table
        =======                 =======                 ============
        Name    Alignment       Name        MLevel      Name    Alignment   MLevel
        Luke    Light           Luke        25000       Luke    Light       25000
        Anakin  Dark            Anakin      27000       Anakin  Dark        27000

    - inner
        - Get all records where the common column matches
        - T1 | join kind=inner (T2) on CommonColumn

        Table 1                 Table 2                 Merged Table
        =======                 =======                 ============
        Name    Alignment       Name        MLevel      Name    Alignment   MLevel
        Luke    Light           Luke        25000       Luke    Light       25000
        Anakin  Light           Anakin      27000       Anakin  Light       27000
        Anakin  Dark                                    Anakin  Dark        27000

    - Outer Join
        - The result of a outer join will ALWAYS contain all records of the
          defined kind (left, right, full) even if the join condition doesn't
          find any matching records.

        - leftouter
            T1 | join kind=leftouter (T2) on key

            Table 1                 Table 2                 Merged Table
            =======                 =======                 ============
            Name    Alignment       Name        MLevel      Name    Alignment   MLevel
            Luke    Light           Luke        25000       Luke    Light       25000
            Anakin  Light           Anakin      27000       Anakin  Light       27000
            Jing    Light                                   Jing    Light

        - rightouter
            T1 | join kind=rightouter (T2) on key
        - fullouter

    - Anti Join
        - Return all records that do not match any record from other side.
        - T1 | join kind=leftanti (T2) on key

            Table 1                 Table 2                 Merged Table
            =======                 =======                 ============
            Name    Alignment       Name        MLevel      Name    Alignment   MLevel
            Luke    Light           Luke        25000       Jing    Light
            Anakin  Light           Anakin      27000
            Jing    Light

        - leftanti
        - rightanti

    - Semi Join
        - Return columns only from one side.
        - T1 | join kind=leftsemi (T2) on key
            Table 1                 Table 2                 Merged Table
            =======                 =======                 ============
            Name    Alignment       Name        MLevel      Name    Alignment   MLevel
            Luke    Light           Luke        25000       Luke    Light
            Anakin  Light           Anakin      27000       Anakin  Light
            Jing    Light

        - leftsemi
        - rightsemi


- UNION
    union *
        // union of all tables

    union *
    | where * has 'some string'

    union *Logs
        // union of all tables endings with `Logs` name

    union T1, T2
        // Union of two tables (T1 and T2)

    union isfuzzy=true T1, T2, T3
        // Enable fuzzy union. It enables union even if something does not exist.

    union withsource=SourceTable kind=inner *
        // This query will search all tables. "kind=inner" ensures only similar
        // columns are outputted in the result. `withsource` adds another
        // column that specifies what table that row came from.



Reference:
[1] https://youtube.com/playlist?list=PLWf6TEjiiuIDDXxYJQryn0KxZ5OjdG83n  -- Not so relevant
[2] KQL: https://youtu.be/DuWBLsgqhaI
[3] KQL Overview: https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/
[4] https://www.youtube.com/playlist?list=PLM3TOIlrnaI4hwmXTxrYGE665q-9fyTfB


@@@@@@@@@@@@@@@@@@@@@@@          SQLite (PYTHON)        @@@@@@@@@@@@@@@@@@@@@@




@@@@@@@@@@@@@@@@@@@@@@@     Single Sign On (SSO)        @@@@@@@@@@@@@@@@@@@@@@

                               7
                ╔════════════════════════════════════╗     ─────────────▶   Authentication flow
                ║        Validate SAML               ║
                ║         response and               ║     ═════════════▶   Token flow
                ║            token                   ║
                ║                                    ║
  Return        ║                                    ║
secure page  8  ║                                    ║            Generates SAML auth
  to user       ║                               ┌────╩───┐        request (XML format)
                ║                               │ Web App│          and redirects to
                ║                               │        │            Windows AD.
                ║      ┌───────────────────────▶│ Server │ ───────────────────────────────┐
                ║      │                        │        │                 3              │
                ║      │                        │        │                                │      User
                ║      │                        └──▲─────┘                              4 │ authenticated
                ║      │  2                        ║                                      │
                ║      │                           ║                                      ▼
                ║      │ Request                   ║                                  ┌────────┐
                ║      │access to                  ║                                  │ Active │
                ║      │ source                  6 ║  Token                           │        │
                ║      │                           ║ Redirect                         │Directory  (IdP)
                ║      │                           ║                                  │        │
                ║      │                           ║                                  │  Server│
                ║      │        ╔══════════════════╝                                  └───╦────┘
                ║      │        ║                                                         ║
                ║      │        ║                                                         ║
           ┌────▼──────┴────────╩───┐                           5                         ║
 (browser) │ http://www.example.com │◀════════════════════════════════════════════════════╝
           └────────────────────────┘                      SAML Token
                       ▲                                  (XML format)
                       │
                       │
                    1  │ User opens a
                       │   browser
                    ┌──┴──┐
                    │User │
                    └─────┘
                  (principal)

References:
[1] https://www.youtube.com/watch?v=cD6pkL020q4
[2] https://developer.okta.com/docs/concepts/saml/



@@@@@@@@@@@@@@@@@@@@@@@@   HELIX editor    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Cheat Sheet
    $ brew install helix
    $ hx -V
    $ hx --tutor
    $ hx --health   # gives LSP integration


@@@@@@@@@@@@@@@@@@@@@@@@   AZURE BICEP     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Chapter 1
---------
- Bicep is DSL for provisioning Azure resources declaratively.
- Wrapper around ARM Templates.
- Bicep is converted (transpiled) into ARM Templates and then submitted to ARM engine.

    // Sample hello world
    param yourName string
    var hello = 'Hello World'
    output helloWorld string = '${hello} $[yourName}'


Bicep Playground:
    https://aka.ms/bicepdemo

Bicep Playground Features:
- It has 130+ bicep templates.
- It allows you to search for templates that fits your needs.
- Generate unique link to Bicep template you create and share with team mates.
- It can transpile Bicep code to ARM template and vice versa (ARM to Bicep code).


Create Resource Group
---------------------
    targetScope = 'subscription'

    resource az_bicep_resource_group 'Microsoft.Resource/resourceGroup@2021-04-01' = {
        name: 'az_bicep_resource_group'
        location: 'eastus'
    }


Chapter 2
---------
App Service Plan
- It defines what's required to run the VMs in RG, such as:
    - OS to run in VMs
    - Pricing Tiers
        Dev and Test: Free, Basic, Shared plans
        Production: Standard, Premium, Isolated


- To run bicep scripts
    az deployment group -g <rg-name> -f <filename.bicep>

- To preview changes before deploying bicep scripts
    az deployment group -g <rg-name> -f <filename.bicep> --confirm-with-what-if

    or use shortcut

    az deployment group -g <rg-name> -f <filename.bicep> -c

    - Options given during preview
        - Create:
        - Modify:
        - NoChange:
        - Ignore:


Chapter 3
---------
App Service
- Create App services inside App Service Plan
    - Must indicate dependency on App Service Plan, otherwise both are deployed
      in paralled (App Service Plan and App Service). The App Service could
      error out if it gets deployed before App Service Plan. To avoid must use
      `dependsOn` attribute.


Chapter 4
---------
Modules
- Each bicep file is a module. It deploys some resources to Azure.
- You can deploy just single module. Or you can have a main module (like an
  Orchestrator module) that deploys other modules.


Chapter 5
---------
Parameters
Communication between Modules using Outputs



@@@@@@@@@@@@@@@@@@@@@@@@   S.O.L.I.D Design Priciples  @@@@@@@@@@@@@@@@@@@
    SRP     Single Responsibility Principle
    OCP     Open Close Principle
    LSP     Liskov Substitution Principle
    ISP     Interface Segregation Principle
    DIP     Dependency Inversion Principle





@@@@@@@@@@@@@@@@@@@@@@@@   Intro to Negotiation     @@@@@@@@@@@@@@@@@@@@@@

Week 1
------
- Pie Framework
- Negotiation should be done using Principled Arguments

- Think of Negotiation as Math problem instead of Tug of War
    - This allows you to be dispassionate.

- Goal 1: Understand what is the Pie?
    - Pie is how much MORE two parties, A and B, can acheive by working
      together than they can get if they don't reach an agreement.
    - Pie = Net benefit from working together - (A's net benefit on its own +
      B's net benefit on its own), where net benefit is the benefit leftover
      after costs.




Coursera
[1] https://www.coursera.org/learn/negotiation/

@@@@@@@@@@@@@@@@@@@@@@@@   IPVS Cheat Sheet         @@@@@@@@@@@@@@@@@@@@@@

- Start an IPVS virtual service. The -t flag indicates TCP connection. VIP
  indicates the Virtual IP (or real IP) address on which Load Balancer receives
  Clients traffic.
        $ ipvsadm -A -t <VIP>
        # ipvsadm -A -t 10.2.111.231    # weighted least connection
        # ipvsadm -A -t 10.2.111.231 -s rr  # round robin

- Add a 'Real Server' to forward incoming request.
        $ ipvsadm -a -t <VIP> -r <real-server-ip>

- Delete a 'Real Server' from Virtual Server table
        $ ipvsadm -d -t <VIP> -r <real-server-ip>

- List the Virtual Server Table
        $ ipvsadm -ln


@@@@@@@@@@@@@@@@@@@@@@@@   Python Hypothesis Library     @@@@@@@@@@@@@@@@@
- @given()
    - Takes strategies as input.
    - Passes them as arguments to test functions.
- @example()
    - Run specific test case.
- Code sample
    import hypothesis.strategies as st
        - st.text()
        - st.integers()
        - st.integers().filter(lambda x: x%2 == 0)

- hypothesis.note()
    - Report this value for minimal failing example

- hypothesis.event()
    - Records events that occurred during this test. Stats will be reported for
      each event as as a summary.
- hypothesis.assume()
    - Calling assume is like an assert that marks the example as bad, rather
      than failing the test. This allows you to specify properties that you
      assume will be true, and let Hypothesis try to avoid similar examples in
      future.

- Test statistics. Run with this flag.
    --hypothesis-show-statistics




@@@@@@@@@@@@@@@@@@@@@@@@   Python Hypothesis Library     @@@@@@@@@@@@@@@@@
"Mitchell Hashimoto" from Hashicorp about on contributing to complex softwares
    - https://mitchellh.com/writing/contributing-to-complex-projects

    - Here are the tip:
        1) Become a User first. Play with *using* the software before anything.
        2) Build the project before reading the code.
        3) Learn how to run test suite.
        4) Learn "Hot-Path Internals" - take the "Trace Down, Learn Up"
          approach.
            - Trace Down:
                - Take a small/tiny feature and run it. Trace down the code
                  path, files, line numbers etc. Do not try to understand
                  yet.
                - Learn feature by feature.
            - Learn Up:
                - Now, learn the system from inside out.
        5) Experiment and break things
            - Add logs.
            - Add tiny functional changes.
        6) Read and Reimplement recent commits
            - Check out the code to previous commit.
            - Repro the bug.
            - Put a solution yourself.
        7) Make bite-sized changes.



@@@@@@@@@@@@@@@@@@@@@@@@   API Design Best Practices     @@@@@@@@@@@@@@@@@


API Best Practices:
    1) Accept and respond with JSON
    2) Use Nouns instead of Verbs in endpoint paths
        - HTTP method already has Verb. So, have Noun in endpoint name.
        - GET /articles/
        - POST /articles/
        - PUT /articles/:id
        - DELETE /articles/:id

        - NOTE: In REST APIs colon (:) is used as placeholder for dynamic
          values (variables). These are parameters that will be replaced with
          real values.

    3) Use logical nesting on endpoints
        - Group APIs containing associated info.
        - Avoid mirroring API design based on database design.
        - Ex:
            - GET /articles/    Get all articles
            - GET /articles/:id/    Get an article
            - GET /articles/:id/comments    Get all comments of an article

        - However, nesting can go deep. After about the second or third
          level, nested endpoints can get unwieldy. Consider, instead,
          returning the URL to those resources instead, especially if that data
          is not necessarily contained within the top level object.

        - For example, suppose you wanted to return the author of particular
          comments. You could use
            /articles/:articleId/comments/:commentId/author.

        - But that's getting out of hand. Instead, return the URI for that
          particular user within the JSON response instead:
                "author": "/users/:userId"

    4) Handle errors gracefully. Return standard error codes.

            Common error HTTP status codes include:

            400 Bad Request - This means that client-side input fails validation.
            401 Unauthorized - This means the user isn't not authorized to access a resource. It usually returns when the user isn't authenticated.
            403 Forbidden - This means the user is authenticated, but it's not allowed to access a resource.
            404 Not Found - This indicates that a resource is not found.
            500 Internal server error - This is a generic server error. It probably shouldn't be thrown explicitly.
            502 Bad Gateway - This indicates an invalid response from an upstream server.
            503 Service Unavailable - This indicates that something unexpected happened on server side (It can be anything like server overload, some parts of the system failed, etc.).

    5) Allow filtering, sorting, and pagination

    - Name collections with plural nouns
    - Maintain Good Security Practices
    - Cache data to improve performance
    - Versioning our APIs


- Lately HATEOAS is used in REST API designs.
- HATEOAS stands for "Hypermedia as the Engine of Application State."
    - It's a principle in RESTful API design that emphasizes the use of
      hypermedia links within the API responses to enable clients to
      dynamically navigate and interact with the application.

    - HATEOAS promotes a more flexible and discoverable API design by providing
      navigation cues within API responses, fostering better interaction
      between clients and servers in RESTful architectures.

- Some think HATEOAS is very useful. UI will know what available actions it can
  perform next, based on the response. Some think HATEOAS bloats API response.
  UI does not need to know all that info.

Reference:
[1] ChatGPT
[2] https://stackoverflow.blog/2020/03/02/best-practices-for-rest-api-design/
[3] https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design
[4] https://swagger.io/resources/articles/best-practices-in-api-design/


@@@@@@@@@@@@@@@@@@@@@@@@   neo4j Cypher Query Language   @@@@@@@@@@@@@@@@@

Graph Patters is represented as
    (node)-[:RELATIONSHIP]->(node)


CALL db.schema()

// Find all "Breweries" Businesses that have IN_CATEGORY relationship with
// Category
MATCH (c:Category)<-[:IN_CATEGORY]-(brew:Business)
WHERE c.name CONTAINS "Breweries" AND brew.city = "Phoenix"
RETURN brew

// Breweries near me that my friends also like
// TODO: How to find my friends?
//       How to define 'like'?
MATCH (brew:Business)<-[:REVIEWS]-(r:Review)<-[:WROTE]-(friend:User)-[:FRIENDS]-(me:User {user_id: "abcd123"})
WHERE r.star > 4
RETURN brew.name, brew.address, avg(r.stars) AS average_stars ORDER BY average_stars DESC


Notes from ChatGPT
==================
    ### Node Creation and Matching:

    - **Create Node:**
      ```
      CREATE (node:Label {property: value})
      ```
    - **Match Node:**
      ```
      MATCH (node:Label {property: value})
      ```

    ### Relationship Creation and Matching:

    - **Create Relationship:**
      ```
      CREATE (source)-[:RELATIONSHIP_TYPE {property: value}]->(target)
      ```
    - **Match Relationship:**
      ```
      MATCH (source)-[:RELATIONSHIP_TYPE]->(target)
      ```

    ### Node and Relationship Deletion:

    - **Delete Node:**
      ```
      MATCH (node:Label {property: value}) DELETE node
      ```
    - **Delete Relationship:**
      ```
      MATCH (source)-[r:RELATIONSHIP_TYPE]->(target) DELETE r
      ```

    ### Filtering and Conditions:

    - **Where Clause:**
      ```
      WHERE condition
      ```
    - **Multiple Conditions:**
      ```
      WHERE condition1 AND/OR condition2
      ```

    ### Aggregation and Grouping:

    - **Count Nodes:**
      ```
      MATCH (node) RETURN count(node)
      ```
    - **Grouping:**
      ```
      MATCH (node)-[:RELATIONSHIP_TYPE]->() RETURN node, count(*) AS count
      ```

    ### Ordering and Limiting Results:

    - **Order By:**
      ```
      ORDER BY property ASC/DESC
      ```
    - **Limit Results:**
      ```
      LIMIT n
      ```

    ### Pattern Matching:

    - **Variable-Length Relationships:**
      ```
      MATCH (node)-[:RELATIONSHIP_TYPE*..n]->(target)
      ```
    - **Optional Match:**
      ```
      OPTIONAL MATCH (node)-[:RELATIONSHIP_TYPE]->(target)
      ```

    ### Functions and Expressions:

    - **String Functions:**
      ```
      TOUPPER(), TOLOWER(), SUBSTRING(), REPLACE()
      ```
    - **Math Functions:**
      ```
      ROUND(), CEIL(), FLOOR(), ABS()
      ```

    ### Conditional Expressions:

    - **CASE Statement:**
      ```
      CASE WHEN condition THEN result ELSE default END
      ```

    ### Indexing and Constraints:

    - **Create Index:**
      ```
      CREATE INDEX ON :Label(property)
      ```
    - **Create Unique Constraint:**
      ```
      CREATE CONSTRAINT ON (node:Label) ASSERT node.property IS UNIQUE
      ```

    ### Union and Combining Results:

    - **Union:**
      ```
      MATCH (a) RETURN a UNION MATCH (b) RETURN b
      ```
    - **Union All:**
      ```
      MATCH (a) RETURN a UNION ALL MATCH (b) RETURN b
      ```

This cheat sheet covers many common Cypher query operations. Cypher is a rich
and expressive language, so there are many more features and functions
available. For detailed information, refer to the official Neo4j Cypher
documentation.



@@@@@@@@@@@@@@@@@@@@@@@@   delve Go debugger @@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Specify target and start debugging with the default terminal interface:
    dlv debug [package]
    dlv test [package]
    dlv exec <exe>
    dlv attach <pid>
    dlv core <exe> <core>
    dlv replay <rr trace>

Trace target program execution
    dlv trace [package] <regexp>

Start a headless backend server only and connect with an external frontend client:
    dlv --headless <command> <target> <args>
        • starts a server, enters a debug session for the specified target and
          waits to accept a client connection over JSON-RPC or DAP

        • <command> can be any of debug, test, exec, attach, core or replay
        • if `--headless` flag is not specified the default terminal client will be automatically started instead
        • compatible with `dlv connect`, VS Code Go, GoLand

    dlv dap
        • starts a DAP-only server and waits for a DAP client connection to specify the target and arguments
        • compatible with VS Code Go
        • NOT compatible with dlv connect, GoLand

    dlv connect <addr>
        • starts a terminal interface client and connects it to a running headless server over JSON-RPC

Help information
    dlv help [command]
    dlv log
    dlv backend
    dlv redirect
    dlv version


Debugging Go Unit Tests
----
    // Will generate a 'package.test' file
    go test -c /path/to/package

    // Use delve to debug
    dlv exec package.test
        (dlv) break package.TestCaseOfInterest
        (dlv) next/step/continue etc



@@@@@@@@@@@@@@@@@@@@@@@@   Markdown Cheat Sheet  @@@@@@@@@@@@@@@@@@@@@@@@@

BASIC SYNTAX
------------
Headers:
    # Header 1
    ## Header 2
    ### Header 3


Emphasis:
    *Italic*
    **Bold**

Unordered Lists:
    - Item 1
    - Item 2

Ordered Lists:
    1. Item 1
    2. Item 2

Links:
    [Link Text](URL)

Images:
    ![Alt Text](Image URL)

Blockquotes:
    > Quote

Code:
   `Inline Code`

   ```Code Block```

Horizontal Rule:
    ---

Tables:
    | Header 1 | Header 2 |
    | -------- | -------- |
    | Cell 1   | Cell 2   |

Escape Characters:
    \*Escaped\*



EXTENDED SYNTAX
---------------
Fenced Code Block:
    ```
    {
      "firstName": "John",
      "lastName": "Smith",
      "age": 25
    }
    ```

Footnote:
    Here's a sentence with a footnote. [^1]

    [^1]: This is the footnote.

Heading ID:
    ### My Great Heading {#custom-id}

Definition List:
    term
    : definition

Strikethrough:
    ~~The world is flat.~~

Task List:
    - [x] Write the press release
    - [ ] Update the website
    - [ ] Contact the media

Emoji:
    That is so funny! :joy:

Highlight:
    I need to highlight these ==very important words==.

Subscript:
    H~2~O

Superscript:
    X^2^

Reference:
[1] https://www.markdownguide.org/cheat-sheet/


@@@@@@@@@@@@@@@@@@@@@@@@   jq Cheat Sheet  @@@@@@@@@@@@@@@@@@@@@@@@@

Basic Syntax
    jq 'filter':
        filter: The expression to process the JSON data.
        .: Represents the entire JSON document.

Assume we have JSON file named `data.json`

    {
      "name": "John",
      "age": 30,
      "city": "New York",
      "skills": ["Python", "JavaScript", "Go"],
      "address": {
        "street": "123 Main St",
        "zipcode": "10001"
      }
    }

Print entire JSON
    jq '.' data.json

Access a specific key
    jq '.name' data.json
    # Output: "John"

Access nested keys
    jq '.address.street' data.json
    # Output: "123 Main St"

Access array elements
    jq '.skills[0]' data.json
    # Output: "Python"

Iterate over array elements
    jq '.skills[]' data.json
    # Output: "Python" "JavaScript" "Go"

Filter elements from an array

Assume data.json contains an array of objects:
    [
      {"name": "John", "age": 30},
      {"name": "Jane", "age": 25},
      {"name": "Doe", "age": 22}
    ]

To filter elements where age is greater than 23:
    jq '.[] | select(.age > 23)' data.json
    # Output:
    # {
    #   "name": "John",
    #   "age": 30
    # }
    # {
    #   "name": "Jane",
    #   "age": 25
    # }

Modify a value
    jq '.age = 31' data.json
    # Output:
    # {
    #   "name": "John",
    #   "age": 31,
    #   "city": "New York",
    #   "skills": ["Python", "JavaScript", "Go"],
    #   "address": {
    #     "street": "123 Main St",
    #     "zipcode": "10001"
    #   }
    # }

Add a new key-value pair
    jq '. + { "country": "USA" }' data.json
    # Output:
    # {
    #   "name": "John",
    #   "age": 30,
    #   "city": "New York",
    #   "skills": ["Python", "JavaScript", "Go"],
    #   "address": {
    #     "street": "123 Main St",
    #     "zipcode": "10001"
    #   },
    #   "country": "USA"
    # }

Remove a key
    jq 'del(.city)' data.json
    # Output:
    # {
    #   "name": "John",
    #   "age": 30,
    #   "skills": ["Python", "JavaScript", "Go"],
    #   "address": {
    #     "street": "123 Main St",
    #     "zipcode": "10001"
    #   }
    # }

Combine multiple filters
    jq '.name, .age' data.json
    # Output: 
    # "John"
    # 30

Pretty print JSON
    jq '.' data.json

Compact (minify) JSON output
    jq -c '.' data.json


Advanced Examples

Group by a key. Assume data.json is an array of objects like before. To group by age:
    jq 'group_by(.age)[]' data.json
    # Output:
    # [
    #   {
    #     "name": "Doe",
    #     "age": 22
    #   }
    # ]
    # [
    #   {
    #     "name": "Jane",
    #     "age": 25
    #   }
    # ]
    # [
    #   {
    #     "name": "John",
    #     "age": 30
    #   }
    # ]

Aggregating values

Summing up values:
    jq '[.[] | .age] | add' data.json
    # Output: 77

Getting keys of an object
    jq 'keys' data.json
    # Output: ["address", "age", "city", "name", "skills"]

Useful Options
-r or --raw-output: Output raw strings, not JSON-encoded strings.
    jq -r '.name' data.json
    # Output: John


-c or --compact-output: Produce compact output.
    jq -c '.' data.json

-f or --from-file: Read filter from a file.
    jq -f filter.jq data.json


@@@@@@@@@@@@@@@@@@@@@@@@   Concurrency vs Parallelism   @@@@@@@@@@@@@@@@@@@@@@@@@

Concurrency:
    - Multiple tasks making progress simultaneously
    - Tasks can start, run, and complete in overlapping time periods
    - Does NOT require multiple CPU cores
    - About dealing with multiple things at once
    - Example: Single chef juggling multiple dishes
      - While pasta is boiling, chop vegetables
      - While sauce is simmering, prepare garnish
      - Tasks are interleaved but not truly simultaneous


Parallelism:
    - Multiple tasks executing at exactly the same moment
    - Requires multiple CPU cores/processors
    - About doing multiple things at the same time
    - Example: Multiple chefs working simultaneously
      - Chef 1 prepares main dish
      - Chef 2 prepares side dish
      - Chef 3 prepares dessert
      - All working truly simultaneously


Key Differences:

Hardware Requirements:
    Concurrency: Can work with single core
    Parallelism: Requires multiple cores

Execution:
    Concurrency: Tasks progress by switching context
    Parallelism: Tasks progress simultaneously

Design Focus:
    Concurrency: Structure (dealing with multiple tasks)
    Parallelism: Execution (doing multiple tasks)



Common Use Cases:

Concurrency:
    - I/O operations
    - User interface handling
    - Network requests
    - Database operations

Parallelism:
    - Image/video processing
    - Scientific calculations
    - Big data processing
    - Neural network training



Programming Models:

Concurrency:
    - Async/await
    - Callbacks
    - Event loops
    - Coroutines

Parallelism:
    - Threading
    - Multiprocessing
    - Distributed computing
    - GPU computing


Remember:

- You can have concurrency without parallelism
- You can have parallelism without concurrency
- You can have both together
- Choice depends on your specific use case

Reference: Claude


@@@@@@@@@@@@@@@@@@@@@@@@   JSON Web Token (JWT)     @@@@@@@@@@@@@@@@@@@@@@@@@
- JWT has 3 main sections separated by dots

1. HEADER: contains metadata about the token:
	alg (algorithm): The signing algorithm used. eg. HS256, RS256
	typ (type): Usually set to "JWT"
	kid (key ID): Optional identifier for the signing key
		{
			"alg": "HS256",
			"typ": "JWT"
		}

2. PAYLOAD (Claims): contains the actual data/claims about the user or entity. Common fields include:

Standard Claims (RFC 7519)
	iss (issuer): Who issued the token
	sub (subject): Subject identifier (usually user ID)
	aud (audience): Intended audience for the token
	exp (expiration): Expiration timestamp
	nbf (not before): Token not valid before this time
	iat (issued at): When the token was issued
	jti (JWT ID): Unique identifier for the token

Custom Claims:
	name: User's full name
	email: User's email address
	roles: User permission/roles
	scope: API access scope

	{
	  "sub": "1234567890",
	  "name": "John Doe",
	  "email": "john@example.com",
	  "roles": ["user", "admin"],
	  "iat": 1516239022,
	  "exp": 1516242622
	}

3. SIGNATURE: Created by encoding the header and payload, then signing with a secret key:

	HMACSHA256(
	  base64UrlEncode(header) + "." + base64UrlEncode(payload),
	  secret
	)

Complete JWT Structure:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

- JWT structure ensures the token is stateless, verifiable and contains all necessary information for authentication and authorization.