Study Guide 2023+

engineering

Warning: These notes are partial, ongoing, incomplete, and may contain typos/inaccuracies. (They are kept factually accurate, time permitting.)

They are being united from many disparate notes created in the past and the layout/organization will gradually improve with time!

Please view them on a computer as they are not optimized for mobile (although you can still view them on Mobile along with the Flashcards at your own risk)!

Topics and code examples are lazy-loaded and may require two-clicks from the TOC to correctly calculate the updated x,y coordinates (after rendering). Thanks!

Engineering: HTTP and SSL

  1. REST - Representational State Transfer (read below).
  2. HTTP - Hypertext Transfer Protocol - often equated with REST. The basis for the World Wide Web typified by Request/Response Objects, Headers, Parameters, HTTP Methods, and Sessions.
  3. SSL - TLS - HTTPS - Hypertext Transfer Protocol with Security.

Certificates

  1. Certificate Authority - validates a Certificate (say, X.509).
  2. Public and Private Keys - a kind of Symmetric Encryption that bifurcates Credentials into two pieces.

REST

REST (Representational State Transfer) - a client-server architectural paradigm that encourages the following core precepts:

  1. Client-Server - the familiar Client to Server paradigm. A Client connects to a Server.
  2. Statelessness - user Session data isn't stored or persisted between Sessions.
  3. Layered System - read Service Layer Architecture article.
  4. Uniform Interface - common data types, HTTP Methods, etc.

Methods:

  1. OPTIONS - indicates that a Client is seeking a Server and further HTTP Request information (required HTTP Headers, etc.).
  2. GET - indicates that a Client will READ a persisted resource.
  3. POST - indicates that a Client will CREATE a persisted resource.
  4. PUT - indicates that a Client will UPDATE a persisted resource.
  5. PATCH - indicates that a Client partially UPDATE a persisted resource.
  6. DELETE - indicates that a Client intends to DELETE a persisted resource.

Request / Response Lifecycle

The HTTP Request/Response Lifecycle:

  1. Request Object -
    1. Headers - specifies the Content-Type, CORS information, Credentials or Tokens, etc. (sent back from the Web Client).
    2. HTTP Method - OPTIONS, GET, POST, PUT, PATCH, DELETE .
    3. URL (Uniform Resource Locater) - the IP Address with optional Port number, DNS address, etc.
    4. Parameters (optional) - specify query filters to narrow down some query result: id, page, etc.
    5. Body (optional) - submitted or requested information can be regimented as JSON, text, a form, etc., and encapsulated by the Request and Response Objects.
  2. Response Object -
    1. Headers - specifies the Content-Type, CORS information, Credentials or Tokens, etc. of the Response Object (sent back from the Web Server).
    2. Body (optional) - submitted or requested information can be regimented as JSON, text, a form, etc., and encapsulated by the Request and Response Objects.

OData

OData defines many of the familiar items above (and more). Some key topics:

  1. Unique identifiers for Objects/Items to be accessed by.
  2. Uniform Resource Locators
  3. ISO/IEC 20802-2:2016 specifies JSON Format (which got rolled into OData).
  4. Common HTTP Headers
  5. Recommending different URL for READ and WRITE -type operations (REST Uniform Interface).

OData is nearly synonymous with REST (which is itself nearly synonymous with HTTP) in practice.

Engineering: TCP, UDP, IP, DNS

  1. IP - Internet Protocol Address
    • IPv6
      • Hexadecimal
      • 128-Bit IP Address
      • Eight groups of four hexadecimal digits
      • Example: 2031:0ca7:75×3:13f3:0103:8b1e:0310:7532
    • IPv4
      • Purely numeric (decimal)
      • 32-Bit IP Address
      • Four octets in the inclusive range 0-255
      • Example: 127.0.0.1
    • CIDR - Classless Inter-Domain Routing
      • Solution for allocating IP Addresses and IP Routing.
      • The convention <MY_IP_ADDRESS>/<CIDR_LENGTH> (e.g. - 10.0.0.0/16) specifies the CIDR Length for the IP Address (Prefix).
      • The CIDR Length defines the number of bits reserved for the Netmask.
        • The Netmask divides an IP Address into sections akin to dividing a Zip Code into Street Addresses.
        • Used to specify which part of the Host IP Address space is reserved for the Host number and which is reserved the Subnet number.
        • The lower the CIDR Length, the more IP Addresses are available within the network.
      • Classes and their respective Netmasks:
        • Class A - 255.0.0.0
        • Class B - 255.255.0.0
        • Class C - 255.255.255.0
    • CIDR Notation Host Formula Available Hosts
      /8 232-8 - 2 16,777,214
      /9 232-9 - 2 8,388,606
      /10 232-10 - 2 4,194,302
      /11 232-11 - 2 2,097,150
      /12 232-12 - 2 1,048,574
      /13 232-13 - 2 524,286
      /14 232-14 - 2 262,142
      /15 232-15 - 2 131,070
      /16 232-16 - 2 65,534
      /17 232-17 - 2 32,766
      /18 232-18 - 2 16,382
      /19 232-19 - 2 8,190
      /20 232-20 - 2 4,094
      /21 232-21 - 2 2,046
      /22 232-22 - 2 1,022
      /23 232-23 - 2 510
      /24 232-24 - 2 254
      /25 232-25 - 2 126
      /26 232-26 - 2 62
      /27 232-27 - 2 30
      /28 232-28 - 2 14
      /29 232-29 - 2 6
      /30 232-30 - 2 2
      (From: https://erikberg.com/notes/networks.html)
  2. DNS - Domain Name Service - associates a (purely) numeric IP Address (127.0.0.1) with a human readable/friendly Domain Name (localhost).
  3. TCP - Transmission Control Protocol - a Transport Layer protocol that complements IP (TCP/IP). Is used to negotiate and determine connections themselves prior to data being transmitted.
  4. UDP - User Datagram Protocol - responsible for actually sending packets and messages over TCP/IP. One-directional, requires some other system to actually manage the connections themselves. (Although it can be used without establishing or verifying connections.)
  5. HTTP/2 - introduces Huffman Encoding to compress packet sizes when initially negotiating an HTTP Request.
  6. QUIC - HTTP/3 - built on top of UDP and removes TCP as the Transport Layer and optimizes some of the initial handshaking resulting in reduced packet sizes, I/O, etc. Multiplexed, persistent connection.
  7. A Proxy is used as an intermediary between an IP Address and a destination resource.
    • Note: Proxying typically refers to Forward Proxying.
    • Forward Proxying (a Forward Proxy Server) is connected to by a Client in order to mask the Client IP Address. The Forward Proxy then connects to a desired resource. (The Proxy masks the incoming IP Address and uses the desired outgoing IP Address.)
    • A Reverse Proxy is a hidden Gateway or intermediary between the Client and desired resource. (AWS API Gateway is paradigmatic - a Client makes an HTTP Request to a specific endpoint; the endpoint is associated with a configured Lambda Function which is called on behalf of the inbound Request.)
    • Forward Proxying is deliberately used or known to the Client. By contrast, in Reverse Proxying scenarios the Proxy acts unbeknownst to the Client.
    • Note that both kinds of Proxying involve the same sequential pattern: a Client C connects to an intermediary Proxy P to access a desired target resource R.
  8. An SSH Bastion serves as an intermediary between an incoming SSH connection and a desired resource.
    • The SSH Bastion serves to "jump" the incoming SSH connection to the desired resource after the inbound connection is authenticated and authorized.
    • In this way, destination resources are protected from being exposed publically and all inbound SSH requests can be verified/controlled.
  1. https://cybermeteoroid.com/ipv4-vs-ipv6-differences-you-need-to-know/
  2. https://www.geeksforgeeks.org/what-is-ipv6/
  3. https://quicwg.org/
  4. https://www.rfc-editor.org/rfc/rfc4632.html
  5. https://stackoverflow.com/questions/46616072/cidr-blocks-aws-explanation
  6. https://www.rfc-editor.org/info/rfc1918
  7. https://erikberg.com/notes/networks.html
  8. https://docs.oracle.com/cd/E19455-01/806-0916/ipconfig-32/index.html
  9. https://stackoverflow.com/questions/224664/whats-the-difference-between-a-proxy-server-and-a-reverse-proxy-server
  10. https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#forwardreverse

Engineering: Open Systems Interconnection

Open Systems Interconnection (OSI) summary.

OSI Layers

Each OSI Layer

  1. Physical Layer
    • Physical Devices (Network Switches, Ethernet Hub, Controllers).
    • Physical transmission and serialization of raw data.
  2. Data Link Layer
    • Transference of Data over the Physical Layer.
    • Example protocols: 802.3 Ethernet and 802.11 Wi-Fi.
  3. Network Layer
    • Includes: TCP/IP (Transmission Control Protocol, Internet Protocol).
    • Packet Forwarding.
    • Internet Addressing.
  4. Transport Layer
    • Data Packets and transferrence of Data from sender to destination.
    • TCP and User Datagram Protocol (UDP).
    • Also, QUIC which runs on UDP (without TCP).
  5. Session Layer
    • The actual Connections (HTTP or otherwise) between Devices, Computers.
    • Any User- or Client Connection- specific Setting, Configuration.
    • User Authorization and Authentication info.
  6. Presentation Layer
    • Data Formatting (JSON, XML, Unicode, ASCII).
    • Data Compression, Encryption/Decryption.
  7. Application Layer
    • REST, HTTP, SMTP, FTP, etc.
    • Standards or Specifications that pertain to Applications and their intercommunication.
  1. https://en.wikipedia.org/wiki/OSI_model
  2. https://www.imperva.com/learn/application-security/osi-model/

Engineering: Status Codes

  1. 200 - Status OK: OK.
  2. 201 - Created successfully typically via POST: Created.
  3. 202 - Queued, batched successfully but not complete: Accepted.
  4. 301 - URL permanently moved
  5. 400 - Bad request - malformed: Bad Request
  6. 401 - Unauthorized, lacking credentials: Unauthorized.
  7. 403 - Forbidden, no authorization to access even with credentials: Forbidden.
  8. 404 - Page or resource can't be found at specified URL: Page Not Found.
  9. 405 - Wrong HTTP method supplied.
  10. 500 - Internal Server Error.
  11. 502 - When a server acting as a gateway (or proxy) receives an invalid response from an upstream server or resource: Bad Gateway.
  12. 503 - When a server is not ready (or available due to being overloaded or maintenance) to handle a request: Service Unavailable.
  13. 504 - When a server acting as a gateway (or proxy) doesn't get a response in time from an upstream server or resource: Gateway Timeout.

Engineering: Common Port Numbers

  1. 443 - HTTPS/TLS default port number
  2. 8080 - HTTP/URL default port number
  3. 22 - SSH default port number
  4. 25 - SMTP default port number
  5. 1433 - MSSQL default port number

Note: 80/443 are usually the default port numbers used by gRPC since it uses HTTP/S - for example as in this article I wrote: https://goteleport.com/docs/api/getting-started/

Engineering: Logging Levels

Logging levels.

Common and Log4j

Some subset of the following is likely to be encountered in most logging frameworks (Log4j) or tools (Terraform).

In descending order of visibility:

  1. TRACE - Most granular, used to gain trace information throughout the entirety of an Application (including third-party dependencies).
  2. DEBUG - Used primarily when debugging or in Development to detail the innerworkings of written code.
  3. INFO - Most granular visibility used in Production.
  4. WARN - Something unusual or unexpected but less than an outright Exception or Application terminating event.
  5. ERROR - An Exception, planned for or unexpected.
  6. FATAL - An Application terminating event.

Of the above: INFO, WARN, and ERROR are the three primary logging levels that most teams will default to.

Java Util

java.util.logging.Level differs from the above by using (in descending order):

  1. SEVERE
  2. WARNING
  3. INFO
  4. CONFIG
  5. FINE
  6. FINER
  7. FINEST

https://docs.oracle.com/javase/8/docs/api/java/util/logging/Level.html

  1. https://sematext.com/blog/logging-levels/
  2. https://logging.apache.org/log4j/2.x/manual/customloglevels.html
  3. https://docs.oracle.com/javase/8/docs/api/java/util/logging/Level.html

Engineering: Naming Conventions

Some common naming conventions:

  1. Dry - database naming convention - Person table with a Person Class domain Object Relational Mapping (ORM)
  2. Snake Case - some_person
  3. Camel Case - somePerson
  4. Pascal - Person

Engineering: Basic Terminal Commands

Shebang

Specifies where to find and which bash executable to use as the execution environment.

On modern Linux systems it's recommended to use:

#!/usr/bin/env bash

Consult: https://tldp.org/LDP/abs/html/sha-bang.html and https://stackoverflow.com/questions/10376206/what-is-the-preferred-bash-shebang

Nano and Vim

Edit a file in Bash:

nano filename

Edit (enable Insert or Interactive Mode) from within the editor:

i
# toggle Insert or Interactive Mode - click i to enable

esc
# toggle Insert or Interactive Mode - exit

Save and exit editor:

:wq!

Exit editor without saving:

:q!

:qa

https://tecadmin.net/save-and-quit-vim/ and https://vim.rtorr.com/

Save Output of Command to File

# stdout and file
MY_COMMAND | tee output.txt

# append to file but don't print to stdout
MY_COMMAND >> output.txt

# overwrite file but don't print to stdout
MY_COMMAND > output.txt

Kill Task

Taskkill /IM node.exe /F

killall node

Open SSL

openssl genrsa -out key.pem 2048
openssl req -new -sha256 -key key.pem -out csr.csr
openssl req -x509 -sha256 -days 365 -key key.pem -in csr.csr -out certificate.pem

Grant Permission

Grant (all) Read, Write, and modify permissions:

sudo chmod +rwx file

Check permissions:

ls -l

Check CPU and Hyper-V Settings

Windows cmd:

systeminfo

SSH Keys

Generate a new SSH Public and Private Key pair using ssh-keygen:

ssh-keygen -t ed25519 -C "your_email@example.com"

Open the Public Key (with suffix .pub) and copy the Public Key into the necessary cloud resource account (GitHub, etc.).

Copy the Private Key into ~/.ssh (or equivalent) and associate with ssh-agent:

ssh-add ~/.ssh/id_ed25519

SSH Key mappings will be visible in /.ssh/known_hosts.

An excellent resource: https://docs.github.com/en/authentication/connecting-to-github-with-ssh

Mac: View Hidden Files

Hold-down: Command + Shift + Period (⌘⇧.) simultaneously to see hidden (system) files.

Mac: Multiple DB Browser for SQLite Clients

To open multiple instances of DB Browser for SQLite at the same time:

"/Applications/DB Browser for SQLite.app/Contents/MacOS/DB Browser for SQLite" &

Visual Studio Code

  1. Hold crtl++ and crtl+- simultaneously to zoom in or out.

IP Address

Useful Linux DNS, IP Address, Port, Connection, Name Server, Host, Record, Route Table tools:

dig
nslookup
host
ping
netstat
whois
ipinfo

nslookup, dig, and host are very similar and perform the same functionalities.

https://github.com/Thoughtscript/bash_ip

  1. https://tldp.org/LDP/abs/html/sha-bang.html
  2. https://stackoverflow.com/questions/10376206/what-is-the-preferred-bash-shebang
  3. https://tecadmin.net/save-and-quit-vim/
  4. https://vim.rtorr.com/
  5. https://docs.github.com/en/authentication/connecting-to-github-with-ssh
  6. https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
  7. https://sqlitebrowser.org/

Code samples:

  1. https://github.com/Thoughtscript/bash_ip

Engineering: Mac Install Notes

Personal notes to get a development environment setup and configured on macOS 10.15.3.

Note: configuring .bash_profile is required for macOS-specific directory paths.

Note: configuring .bashrc is required for many Bash-specific paths.

Note: configuring .zshrc is required for many Zsh-specific paths (macOS Catalina+).

The notes below are written for Bash. In most cases, the same commands and export statements can be used interchangeably between Bash and Zsh (provided you use the correct configuration file).

I'm also not a huge fan of Homebrew (think Catalina system directory structure incompatibility, multiple lingering installs, version conflicts that typically or eventually arise locally, etc.) so the installation steps below tend to prefer the native installers wherever possible and appropriate.

Xcode

For developing Apple mobile, desktop, tablet, and watch, applications.

Swift installation:

  1. Use this link: https://apps.apple.com/us/app/xcode/id497799835 to get Swift + Xcode
  2. Test the installation using $ swift --version
  3. Test the Xcode installation using $ xcodebuild -version

CocoaPods:

  1. Requires Ruby gem manager
  2. Execute the following: $ sudo gem install cocoapods
  3. Use $ pod install to download the dependencies into your project
  4. NOTE: You must open the <PROJECT_NAME>.xcworkspace file rather than the <PROJECT_NAME>.xcodeproj file to build and compile your project (with Pods) correctly

Python 2.7

For Python, PIP, and Django apps.

Python 2.7 installation:

  1. Python 2.7 is pre-installed on macOS 10.x.
  2. Test the installation using $ python --version

PIP installation:

  1. Execute $ sudo easy_install pip
  2. Test the installation using $ pip --version

Django installation:

  1. Execute $ sudo pip install Django==3.0.3
  2. Alternatively, execute $ sudo pip install -r requirements.txt

Python 3

For Python 3.x.

Python 3.x installation:

  1. Download from: https://www.python.org/downloads/mac-osx/
  2. Test the installation using $ python3 --version

PIP:

  1. PIP is automatically installed as part of the Python 3 installation
  2. Upgrade PIP: $ python3 -m pip install --upgrade pip
  3. Install dependencies: $ python3 -m pip install -r requirements.txt
  4. List all installed libraries: $ pip freeze
  5. Clear out PIP cache: $ pip uninstall -y -r <(pip freeze)

Venv:

  1. Venv is automatically installed as part of the Python 3 installation
  2. Create a Venv environment: $ python3 -m venv VENV_ENV
  3. ... and activate it: $ source VENV_ENV/Scripts/activate

Ruby

For Ruby on Rails apps.

Ruby installation:

  1. Ruby 2.6.3p62 is pre-installed on macOS 10.x.
  2. Test the installation using $ ruby --version
  3. Test Ruby gem manager using $ gem -v

Rails installation:

  1. $ gem install rails

C++

  1. C++ GCC 4.2.1 is distributed with Xcode 11.3 (whose installation instructions are provided above).
  2. Test the installation using $ gcc --version

CMake:

  1. Download the CMake Unix/Linux Source (has \n line feeds) from: https://cmake.org/download/
  2. Extract the contents and execute $ bash bootstrap within the root directory
  3. Then execute $ cd Bootstrap.cmk and $ bash make
  4. Copy the following into .bash_profile using $ sudo nano ~/.bash_profile (and modify as needed):
export PATH=$PATH:/Users/USER_NAME/Desktop/cmake-3.16.6/bin
  1. Test the installation: $ cmake --version

Java

For Java Spring, Gradle, Maven, and Tomcat stacks.

Java installation:

  1. Use this link: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  2. Test the installation using $ javac -version

Apache Tomcat 8.5.x installation:

  1. Download the .zip from http://tomcat.apache.org
  2. Copy the extracted directory to your desktop (or some other appropriate location)
  3. Navigate to the ROOT/bin directory and execute the following Bash command sudo chmod +x *.sh
  4. Execute $ sudo bash startup.sh to launch Tomcat on the default port localhost:8080

Refer to the very helpful: https://wolfpaulus.com/tomcat/ for more comprehensive configurations.

Gradle installation:

  1. Download from: https://gradle.org/install/
  2. Copy the following into .bash_profile using $ sudo nano ~/.bash_profile (and modify as needed):
export PATH=$PATH:/Users/USER_NAME/Desktop/gradle-6.2/bin/
  1. Test the installation using $ gradle --version

Node

For NodeJS server and client apps.

NVM installation:

  1. $ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.1/install.sh | bash
  2. $ sudo touch ~/.bash_profile
  3. $ sudo touch ~/.bashrc
  4. $ sudo nano ~/.bash_profile - copy the contents below into this file
  5. $ sudo nano ~/.bashrc - copy the contents below into this file
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

(Copy the above into configuration files.)

  1. Test the installation using $ nvm ls
  2. Download the desired version of Node using $ nvm install 10.0.0 && nvm use 10.0.0

Typescript installation:

  1. Execute npm install -g typescript

Golang

  1. Go to: https://golang.org/dl/ and download the newest version
  2. Install Go using the downloaded installer
  3. Test the installation using $ go version

Rust

For Rust apps.

Rust installation:

  1. Execute $ sudo curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
  2. Test the installation using $ rustc --version

Rust uninstallation:

  1. $ rustup self uninstall

Refer to the Rust documentation.

Disable DS_Store Files

Open Terminal App:

  1. Located in Applications > Utilities

  2. Enter the following command:

    defaults write com.apple.desktopservices DSDontWriteNetworkStores true

Consult: https://www.techrepublic.com/article/how-to-disable-the-creation-of-dsstore-files-for-mac-users-folders/

Brew

As of macOS 14.4.1 (23E224) you may need to install Homebrew for certain commonly used dependencies:

  1. Download Homebrew or by running: 

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

  2. Run the commands:

    (echo; echo 'eval "$(/opt/homebrew/bin/brew shellenv)"') >> /Users/YOUR_USERNAME/.zprofile

eval "$(/opt/homebrew/bin/brew shellenv)"

    to add Homebrew to PATH.

    Note that .zprofile is the default shell config from the official documentation.

  3. Run source .zprofile (or source .bash_profile) to reload the relevant terminal configs

  4. To install pkg-config on Mac: brew install pkg-config (required for Python3)

zsh

Duplicate any .bash_profile entries into .zprofile (or .zshrc):

  1. .zprofile is for login shells
  2. .zshrc is for interactive shells

Consult: https://unix.stackexchange.com/questions/71253/what-should-shouldnt-go-in-zshenv-zshrc-zlogin-zprofile-zlogout

WinRAR on Mac

  1. Add: PATH="/Users/username/Desktop/rar:${PATH}" to ~.zprofile.
  2. Given password @ZeGV
# Will delete files and encrypt
rar a -df -p@ZeGV -r _data.rar ./_data
# No leading ./ in _data
## Special chars like ! don't appear to be allowed here
rar x -p@ZeGV _data.rar _data

The Official Command Line Syntax documentation.

  1. https://apps.apple.com/us/app/xcode/id497799835
  2. https://doc.rust-lang.org/stable/
  3. https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  4. https://www.techrepublic.com/article/how-to-disable-the-creation-of-dsstore-files-for-mac-users-folders
  5. https://unix.stackexchange.com/questions/71253/what-should-shouldnt-go-in-zshenv-zshrc-zlogin-zprofile-zlogout
  6. https://documentation.help/WinRAR/HELPCommandLineSyntax.htm

Engineering: Windows Install Notes

It's been a while since I used a Windows machine for development. (Windows has traditionally been my preferred and most commonly encountered development environment of choice.) Have to say, I'm immensely impressed by Windows 11!

There are many great changes overall (and, I disagree with some critics - the new taskbar is much improved - centering the icons is a much better experience than having to peer over in the left corner on large screens/UHD 4K TVs).

Node

This will install Node, Git, and Python 2.7.

As of 5/21/2022 - Windows 11 Pro 21H2 22000.675.

For Node:

  1. Make sure to open your terminal of choice with the Run As Administrator option (equivalent in some ways to sudo).

  2. Download Git SCM

  3. Download NVM

  4. Download Python 2.7

  5. Rename the python executable to to python2 (node-gyp requires this nomenclature).

  6. Search locally for sysdm.cpl to open System Properties -> Advanced -> Environmental Variables in Windows 11.

  7. Add the install path to your User and System Variables.

  8. nvm install X.Y.Z && nvm use X.Y.Z (in Bash or ZSH) for the specific version of Node you want.

  9. I've had the best luck using Visual Studio 2017 (Community) rather than a newer version. Download after signing into Microsoft here.

  10. Make sure to tick off:

    • C# and Visual Basic Rosyln compilers
    • MSBuild
    • Static analysis tools
    • Visual Studio C++ core features
    • Windows 10 SDK (10.0.17763.0)

  11. Run npm config set msvs_version 2017

  12. Run npm i or whatever npm or node commands you desire.

Java

This will install Java 1.18, Maven , Tomcat 10 and Gradle.

Updated 8/25/2022 - Windows 11 Pro 21H2 22000.856.

  1. Download Java 1.18
  2. Download Maven 3.8.6
  3. Download Tomcat 10
  4. Download Gradle 7.5.1
  5. Add the relevant System variables under Advanced system settings > Environment Variables.
  6. GRADLE_HOME should point at your unzipped Gradle root directory.
  7. JAVA_HOME should point to your Java root directory.
  8. MAVEN_HOME should point to your unzipped Maven root directory.
  9. Then add the following to your Path.
  10. %JAVA_HOME%\bin,
  11. %GRADLE_HOME%\bin
  12. %MAVEN_HOME%\bin

Run the folloiwng commands to verify your installs.

  1. javac -version
  2. mvn -version
  3. gradle --version

Navigate to: http://localhost:8080/ after running the new(ish) Apache Tomcat10 executable.

Engineering: Azure VM Ubuntu 18.04

Brief setup notes for Azure VM Ubuntu 18.04 on Mac OSx.

  1. Create Public and Private Keys

Create a private key. A public key will be created automatically for you.

ssh-keygen -t rsa

Make note of your private key password - this is used to authenticate below.

After creating your Azure Ubuntu 18.04 VM. Take note of your Public IP.

Ensure that the default SSH port is left open.

  1. Connect from Local Machine
sudo ssh -i path/private_key user@public_ip

Use your private key password after connecting.

Ubuntu 18.04 VM

Slightly different setup than Ubuntu 14.04.

sudo apt-get update
sudo apt-get upgrade

Git

sudo apt-get update
sudo apt install git

Node

sudo apt install nodejs
sudo apt install npm

Java

Note: It's recommended to download Java 11+ directly from Oracle.

It's further recommended to use the OpenJDK 11.0.2 (and avoid other options).

Refer to: https://jdk.java.net/archive/

sudo apt-get update
wget "https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_linux-x64_bin.tar.gz"
sudo tar -xzvf openjdk-11.0.2_linux-x64_bin.tar.gz
sudo mv jdk-11.0.2 /usr/lib/jvm/

# Config
sudo nano /etc/environment

# Add the line below
# JAVA_HOME="/usr/lib/jvm/jdk-11.0.2/"

# Config
sudo nano ~/.bashrc
# Add the lines below
# JAVA_HOME=/usr/lib/jvm/jdk-11.0.2/
# PATH=$JAVA_HOME/bin:$PATH
source ~/.bashrc

# Verify
echo $JAVA_HOME
javac --version

Tomcat:

groupadd tomcat
useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat
cd /opt/ 
sudo wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.2/bin/apache-tomcat-9.0.2.tar.gz
tar -xzvf apache-tomcat-9.0.2.tar.gz
sudo mv apache-tomcat-9.0.2 tomcat

Then:

# Permissions
sudo chown -hR tomcat:tomcat tomcat
sudo chmod +xr tomcat/bin/

# Config
sudo nano ~/.bashrc

# Add the line below
# CATALINA_HOME=/opt/tomcat

# Config
source ~/.bashrc

# Verify
echo $CATALINA_HOME

You can now access /opt/tomcat/bin/ to execute: sudo bash startup.sh.

Your Tomcat server will be available by default on http://YOUR_IP:8080 (note the lack of https here).

Engineering: Misc

Commonly encountered and useful topics.

HTML Escape Characters

Can use the following in many places where typical spaces/whitespace isn't allowed (will often force a Whitespace to appear):

&nbsp;

https://mateam.net/html-escape-characters/

  1. https://mateam.net/html-escape-characters/

Engineering: OWASP

Summary of common top 10 OWASP identified security vulnerabilities:

  1. Broken Access Control
    • Violation of The Principle of Least Privilege
    • Privilege elevation
    • CORS misconfiguration
    • Insecure direct object references
    • Metadata manipulation
  2. Cryptographic Failures
    • Weak, old, or deprecated cryptographic algorithms
    • Lack of encryption: TLS, etc.
    • Low randomness used: pseudorandom, improper seed
  3. Injection
    • SQL Injection
    • Query Injection
    • Lacking character escaping, validation, filtering, or sanitization
    • Ability for users to execute queries from a client input
  4. Insecure Design
    • Ability of malicious actors to exploit weaknesses in system design, architecture, or business logic
    • Example: not restricting the number of tickets a person can buy, not having rate limiting, etc.
  5. Security Misconfiguration
    • Leaving Ports open
    • Insufficient security hardening
    • A system whose dependencies aren't updated
    • Disabled security features
    • Incompatible dependency versions
  6. Vulnerable and Outdated Components
    • Deprecated, out-of-date, or old software and/or dependencies
    • Lacking a critical security patch
  7. Identification and Authentication Failures
    • Lacks multi-factor authentication
    • Allows for weak passwords
    • Transmission of passwords in plaintext
    • Susceptibility to automated or brute-forcing attacks
  8. Software and Data Integrity Failures
    • Downloading unsigned dependencies from a remote repository
    • Downloading dependencies from an untrusted remote repository
    • A compromised update
    • Insecure deserialization
  9. Security Logging and Monitoring Failures
    • Exposing critical or sensitive data within accessible logs
    • Unclear or unhelpful logging messages
    • Lacking proper logging for critical events
    • Lacking the proper visibility into critical events, services, or systems
    • Lacking appropriate mechanisms to remediate problems: escalation, on-call rotations, etc.
  10. Server-Side Request Forgery
    • Improperly allowing a web application and/or its resources to be accessed or controlled remotely
    • Allowing an unauthenticated, unvalidated, or unauthorized agent to access a resource remotely
  1. https://owasp.org/www-project-top-ten/

Engineering: Application Security

A concise, quick, overview.

SOC2

  1. Every completed task must be documented.
  2. Every customer-facing or production task must have a corresponding ticket.

OWASP

  1. Available here: https://owasp.org/www-project-top-ten/
  2. Common Security Vulnerabilities include XSS, SQL Injection, String Timing attackes, etc.

Encrypt at Rest

  1. Encrypt database information within the database.
  2. Never store sensitive PII information in plaintext.

Ports

  1. Verify that all unneeded Ports are closed.

Secure Access Workstation

  1. Increasingly popular.
  2. An air-gapped physical machine allowing access to a single user at a single precise time.
  3. The air-gapped physical machine then connects to an SSH Bastion.

Engineering: OAuth 2.0

Grant Types

The four main Authorization Grant Types are as follows (note that the PKCE extension is now recommended for supplemental use in all other Grant Types):

Authorization Code

The flow:

  1. A resource R1 requests access to a protected resource R2.
  2. A user (who typically owns or manages both resources) is directed to an intermediary authentication server to authorize R1’s getting R2’s data (for some duration of time).
  3. After authorizing access, the user is redirected to a specified URL (often a view in R1): the Redirect URL.
  4. The most common and secure Grant Type. Use this by default.
  5. The paradigm or model OAuth flow that inspired the others.

Client Credentials

  1. Used in server-to-server access where one resource is able to access a second (common in micro-services).

Device Code

  1. Where a device with limited human input capabilities is granted access to a resource through another human input medium.
  2. Used to give IoT devices access to some endpoint. These devices may lack a keyboard.

Refresh

  1. Used to refresh a previously acquired valid token.

Deprecated

Note that the Password and Implicit Grant Types are now deprecated:

Password

  1. Deprecated.
  2. Essentially, your standard user-submitted password to get a token authentication scheme.

Implicit

  1. Deprecated.
  2. Often used in minimal security conditions (like a simple JavaScript app) to acquire a token from an auth server through a (registered) Client ID.
  3. Tokens are not refreshed but simply reacquired.
  1. https://oauth.net/2/

Engineering: Service Layer Architecture

  1. Presentation Layer - what the user sees and interacts with. A web Client, mobile device, terminal, etc. Corresponds to the View in MVC pattern.
  2. Service Layer - contains Business Logic, Controllers, Handlers, and Services that process Data being sent from a Client, and that return processed Data to the Client.
  3. Data Layer - persisted Data that exists at rest or used in-memory.

Engineering: Model View Controller

  1. Model - programmatic representations of Data defined in a Domain. Run Time, in-memory, and programmatic representations of persisted data that may or may not be managed 1:1 through an Object Relational Mapping framework like Hibernate.
  2. View - what's rendered and presented to the user.
  3. Controller - HTTP REST Controllers, Handlers that reside at sepecifc Endpoints (e.g. - URL, URL Path, IP Address, Domain Name, Port Number) etc. to handle specific HTTP Requests / HTTP Methods.

Engineering: Design Patterns

Some common design patterns.

Singleton

Refer to the other Singleton article.

Factory

The Factory design pattern creates copies of a specific type. The Factory itself is usually instantiated once, often a Singleton (single instance object).

// Java
public class Dood {
    //...
}

public Dood doodFactory() {
    //...
    getDooder() {
        return new Dood();
    }
}

Abstract Factory

Suppose that K is a subclass of T: make a Factory of kind T that returns a Factory or makes Objects of kind K (through abstract implementation.

The Abstract Factory design pattern is essentially the Factory design pattern but involves Abstract Factories in correspondence with and supporting an abstract (here, meaning higher-order) interface implementation hierarchy or abstract class hierarchy:

First, a (higher-order) interface:

// Java
public interface A {
    //...
}

Two implementations of that interface:

// Java
public class B implements A {
    //...
}

public class C implements A {
    //...
}

Now, we create a factory hierarchy to parallel the implementation hierarchy:

// Java
public interface AbstractFactory {
    public A createA();
}

We then create two further factories concrete implementations of AbstractFactory*`:

// Java
public class BFactory implements AbstractFactory {
    public A createA() {
        B b = new B();
        return b;
    }
}

public class CFactory implements AbstractFactory {
    public A createA() {
        C c = new C();
        return c;
    }
}

Model View Controller

Refer to the other Model View Controller article.

Decorator

Uses annotations like @Component to configure some item or add functionality.

Examples:

  1. Angular
  2. Java Spring
  3. Java EE

Adapter

The Adapter pattern takes two incompatible implementations, Interfaces, or Classes, and provides a way to bridge them. (Hence, "Adapter" - to adapt them to each other.)

// Java
public interface GermanPlugConnector {
    public void giveElectricity();
}

public class GermanElectricalSocket {
    public void plugIn(GermanPlugConnector plug) {
        plug.giveElectricity();
    }
}

// Java
public interface UKPlugConnector {
    public void provideElectricity();
}

public class UKElectricalSocket {
    public void plugIn(UKPlugConnector plug) {
        plug.provideElectricity();
    }
}

These are thus far incompatible and require an Adapter to bring them into harmony:

// Java
public class GermanConnectorToUKSocketAdapter implements UKPlugConnector {
    private GermanPlugConnector plug;

    public GermanConnectorToUKSocketAdapter(GermanPlugConnector plug) {
        this.plug = plug;
    }

    @Override
    public void provideElectricity() {
        plug.giveElectricity();
    }
}

Above we implemented the compatible Plug connector (UKPlugConnector) but overrode provideElectricity() so that it now invokes giveElectricity() on the incompatible Plug connector (GermanPlugConnector).

We have thus performed a little "switch-a-roo" on the main point of incompatibility by using a third interface to bring the two incompatible types into harmony thereby.

Now we explicitly invoke our adapter to representationally allow the GermanPlugConnector to plug into the UKElectricalSocket:

// Java
GermanPlugConnector plugConnector = //.. create a GermanPlugConnector

//Create a UKElectricalSocket
UKElectricalSocket electricalSocket = new UKElectricalSocket();

//We adapt the two
UKPlugConnector ukAdapter = new GermanConnectorToUKSocket(plugConnector);

//And now receive the electricity
electricalSocket.plugIn(ukAdapter);

Example corrected, updated, and modified from: http://www.vogella.com/tutorials/DesignPatternAdapter/article.html

Retry, Backoff, and Circuit Breaker

Some or all of the following can and usually should be combined to correctly handle Errors at the Enterprise Web Application / Service level:

Strategy Description Advanced
Retry Attempts again. Dead Letter Queue, Maximum Number of Attempts
Exponential Backoff Time interval between attempts increases each try. Maximum Number of Attempts
Circuit Breaker Functionalitites are disabled. Status Flow, Blast Radius, Service/Functionalities Flows
  1. Retry
    • An essential Design Pattern that involves retrying an operation when an Exception, failure, or fault is encountered.
    • Typically combined with Exponential Backoff strategies that increase the interval of time between each retry attempt. (Otherwise an Exception can result in the "Stampeding Herd" (cascading failure) phenomenon and a single failure can become many.)
    • Often combined with a predetermined maximum number of attempts (after which subsequent events are handled by and delegated to a Dead Letter Queue).
  2. Circuit Breaker
    • An Exception, failure, fault, or specified condition "trips" or triggers disables or blocks additional calls or functionalities from occuring.
    • Events move through predefined status conditions with associated cooldown time intervals. As event statuses change, functionalities are enabled, disabled, or slowed.

Quick and helpful overview of these topics.

  1. http://www.journaldev.com/1827/java-design-patterns-example-tutorial
  2. https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker
  3. https://dilankam.medium.com/understanding-retries-exponential-backoffs-and-circuit-breakers-in-distributed-systems-4355db103505

Engineering: The Twelve-Factor App

Summary of Twelve-Factor App Principles:

  1. Codebase - One codebase tracked in revision control, many deploys.
    • A single codebase is used for an App.
      • E.g. - a specific repo in GitLab, BitBucker, Subversions, Azure Team Foundation, or GitHub.
    • Use a versioning system to track different deployments and changes.
    • Use a version control tool like Git or Subversion.
  2. Dependencies - Explicitly declare and isolate dependencies.
    • All dependencies are explicitly and expressly declared in a manifest.
    • Examples: package.json, pom.xml, requirements.txt, etc.
    • Use a package manager.
      • Dependencies should be versioned and modularized so their exact contents can be specified.
  3. Config - Store config in the environment.
    • App Configuration is kept with the App.
    • It should be stored as Environmental Variables within the Environment itself.
    • Example: passing Docker parameters that are exposed as Environmental Variables within an App Container.
    • App Configuration should be grouped by (Staging) Environment.
  4. Backing Services - Treat backing services as attached resources
    • Any Service or infrastructure dependency is treated as a Resource that's attached (containerized or configured along with the Service in question).
    • Examples: docker-compose.yml, Terraform, Cloud Formation
  5. Build, Release, Run - Strictly separate build and run stages.
    • Separate a deployment into Build, Release, and Run stages.
    • All Releases should have a unique identifier.
  6. Processes - Execute the app as one or more stateless processes.
    • Apps should be run as Processes.
    • They should be Stateless.
    • Example: containerizing and App in a Docker Container that spins up and initializations the App state each time it's run (state isn't preserved).
  7. Port Binding - Export services via port binding.
    • Preference for directly binding an App to a Port via a Webserver library (Jetty).
    • As opposed to using a Web Container (Tomcat).
  8. Concurrency - Scale out via the process model.
    • While one may need to spawn or multithread, these processes should use established tools and infrastructure to do so (Node exec, JVM Threads, Windows Process management).
    • Don't create/spawn Daemons when the above can be used out of the box.
  9. Disposability - Maximize robustness with fast startup and graceful shutdown.
    • Processes should be easy to terminate.
    • Deployed Services shouldn't be unnecessarily dependent on or entangled with other systems. They should be sufficiently decoupled so that they can started and stopped easily.
  10. Dev/Prod Parity - Keep development, staging, and production as similar as possible.
    • (Staging) Environments should be 1:1.
  11. Logs - Treat logs as event streams.
    • All logging events are printed to STDOUT and never routed within the app.
    • All log organization is handled at the Enivronment level.
    • Example:
      • Set Logging Levels within an AWS Environment so that it can be collated into a log ingestion Service.
      • Rather than routing logs using distinct say, Log4J LoggingAppenders.
  12. Admin Processes - Run admin/management tasks as one-off processes.
    • They should have a finite execution interval (should not be ongoing or occasion indefinite access to say PROD).
    • They should always use the most up-to-date config and code (not ad-hoc scripts or old code versions).
  1. https://12factor.net/

Engineering: Producers and Consumers

  1. Consumer - The resource that consumes, receives, and/or uses Messages or Events sent by the Producer.
  2. Producer - The emitter or sender of a Message or Event to the Consumer

Publish Subscribe and Message Queues

Several commonly encountered patterns that involve Producer and Consumer topics:

  1. Publish-Subscribe
    • Subscribers listen to, wait for, poll, or otherwise directly subscribe to a Publisher (specifically, a Publisher's Topics) which emits an Event or Message to all appropriate Subscribers.
    • Examples:
      • AWS Simple Notification Service (AWS SNS)
      • Apache Kafka
      • WebSockets
  2. Message Queue
    • A Message Queue sits in-between a Producer and Consumer. Often explicitly uses an intermediary called (or appropriately likened to) a Message Broker.
    • Examples:
      • ActiveMQ
      • RabbitMQ
      • AWS Simple Queue Service (AWS SQS)
  3. SOAP
    • A Consumer requests a WSDL from the Producer to mirror the Domain entities of a Producer (generates the shared contract for the Consumer).
    • This allows SOAP requests to be correctly made from the Consumer to the Producer and for the Consumer to correctly handle the corresponding SOAP responses.
  4. REST
    • A Client (Consumer) makes an HTTP Request to a Server (Producer) which returns an HTTP Response.

Note that this distinction isn't strictly mutually exclusive. While Quastor refers to RabbitMQ as thoroughly Pub-Sub, the actual RabbitMQ documentation refers to it as Publish/Subscribe (since it can be configured as such).

  1. https://blog.quastor.org/p/tech-dive-apache-kafka
  2. https://www.rabbitmq.com/tutorials/tutorial-three-spring-amqp

Code samples:

  1. https://github.com/Thoughtscript/java-reactive-pubsub/tree/main
  2. https://github.com/Thoughtscript/java_soap_wsdl_2023/tree/main

Engineering: Systems Design

Topics

  1. Separation of Concerns - Access, Visibility, proper decomposition of a Monolithic app, database access, subnets, private networks
  2. Concurrency - Thread, Thread and Worker Pooling, database Connection Pooling, use of Mutex and Thread-Safety, Asynchronous and Non-Blocking, Load Balancing, Sharding
  3. Events, Messaging, and Event Priority - use of Cron Jobs, Queues, Event Scheduling, Event Workflows
  4. Fault Tolerance - High Availability and Disaster Recovery, ProdEng manual overrides, data auditing and integrity, Graceful and Exponential Backoff, Dead Letter and Retry Queues
  5. Performance - Caching, Views and Indexes, Algorithm optimization, Elasticity, duplex connections
  6. Best Practices - Security, compliance, Integration Testing and environments, End to End Acceptance Tests, etc.

Some Approaches

Some common, famous, and or high-performance approaches:

  1. Decouple Events, Consumers, and Producers - Apache Kafka remains a top choice at many firms (PayPal, Uber, Dosh, etc.).
  2. A move from Relational Databases to Document Store Databases reducing overhead from Sharding, need for Cross-Database JOINS, SQL Table Normalization, and IO/latency.
    • Modifying SQL Table Columns is very expensive on large datasets that are distributed over many Shards.
  3. Extensive use of Sharding, Consistent Hashing (keeps Hash invariant as number of Shards increases), and Load Balancing across different Application Layers.
  4. Lots of Caching:
    • Clear distinction between READ and WRITE intensive resources (and corresponding resource responsibilities - READ Replica, etc.).
    • Materialized Views, etc.
    • Extensive use of Memcached, Redis, and/or Couchbase for READ intensive operations.