An activation function determines a neural network node’s output from its inputs and their weights. It helps a network represent useful relationships in data, and its mathematical properties influence learning.
A node first combines its inputs, commonly by weighting them and adding a bias. The activation function then transforms that combined value into the node’s output. In a simple notation, the output is f(z), where z is the weighted input and f is the activation function.
All flashcards
Activation function
Transforms a node’s weighted input into its output.
Universal Approximation Theorem
A nonlinear network with two layers can approximate a broad range of functions; this capacity result does not guarantee easy or successful training.
Finite activation range
Can support more stable training because a pattern presentation may significantly affect only a limited set of weights.
ReLU
Returns zero for negative inputs and the input itself for positive inputs.
Quantum activation construction
Certain quantum circuits approximate activation functions using Taylor-series methods while preserving quantum properties such as superposition.
Quiz questions
What does a neural-network node do with its weighted input before producing an output?Answer: It applies an activation function to the combined input.
Why can an activation function with an infinite output range call for a smaller learning rate?Answer: A pattern presentation may significantly affect most weights, so smaller updates may be needed.
Why can the binary step function not support progress with gradient-based methods?Answer: Its derivative is zero away from zero, and it is not differentiable at zero.
What determines the response of a radial basis function?Answer: The distance between the input and a center vector.
What operation characterizes folding activation functions?Answer: They aggregate values from multiple inputs.