ch-3 Improving the way neural networks learn

The techniques we’ll develop in this chapter include

• a better choice of cost function, the cross-entropy cost function;
• four so-called “regularization” methods ( L1 and L2 regularization, dropout, and artificial expansion of the training data ), which make our networks better at generalizing beyond the training data;
• a better method for initializing the weights in the network;
• a set of heuristics to help choose good hyper-parameters ;

ch-2 How the backpropagation algorithm works

discuss how to compute the gradient ( $\nabla C$ ) of the cost function

Heart of backpropagation

• an expression for the partial derivative ∂ C /∂ w of the cost function C with respect to any weight w ( or bias b ) in the network
• tells us how quickly the cost changes when
  we change the weights and biases

ch-1 Using neural nets to recognize handwritten digits

Perceptrons

• a type of artificial neuron
• a device that makes decisions by weighing up evidence

[CS:APP-ch8] Shell Lab

http://csapp.cs.cmu.edu/3e/labs.html

http://condor.depaul.edu/glancast/374class/hw/shlab-readme.html

https://github.com/zhoudiqiu/Shell-lab/blob/master/lab5/tsh.c

Target

implement

• eval, builtin_cmd, do_bgfg, waitfg
• sigchld_handler, sigint_handler, sigtstp_handler

SICP-ch2 Building Abstractions with Data

building abstractions by forming compound data

Intro

• 2.1 data abstraction erect abstraction barriers
• 2.2 data == procedures, sequences, closure, conventional interface
• 2.3 symbols as elementary data
• 2.4 generic operations, data directed programming, additivity
• 2.5 implement a package

SICP ch-1 Building Abstractions with Procedures

We are about to study the idea of computational process

1.1 The Elements of Programming

Language combine simple ideas to form complex ideas by three mechanisms

1. primitive expression
2. means of combination
3. means of abstraction

CS:APP ch-1 A Tour of Computer Sytems

A computer system consists of hardware and systems software that work together to run application programs.

Tracing the life time of hello program.

1
2
3
4
5
6
7
8

#include <stdio.h>  
  
int main()  
{  
        printf("hello world!\n");  
  
        return 0;  
}

CS:APP ch-9 Virtual Memory

Virtual Memory

• modern systems provide an abstraction of main memory

three important capabilities

1. It uses main memory efficiently by treating it as a cache for an address space stored on disk, keeping only the active areas in main memory, and transferring data back and forth between disk and memory as needed.
2. It simplifies memory management by providing each process with a uniform address space.
3. It protects the address space of each process from corruption by other processes.

CS:APP ch-8 Exceptional Control Flow

control flow

• sequence of control transfers ( from the address of instruction to the next )

Exceptional Control Flow ( ECF )

• Abrupt changes in control flow
• Occurs all levels
  1. Hardware: Transfers to exceptional handlers
  2. Operating System: Context switch
  3. Application: Signal
  4. Individual Program: Nonlocal jump

CS:APP ch-6 The Memory Hierarchy

6.1 Storage Technologies

SRAM

• cache, transistor, stable, fast, expensive

DRAM

• main memory, capacitor, sensitive( periodically refresh every bit )
• d * w cells --> d supercells == r rows * c columns ( two dimension array )
• data pins ( w bits ) & addr pins ( row addr i, column addr j )
• Memory modules: Each supercell stores 11 byte. Each address represent by 8 supercells
• Enhanced DRAM