All commonly used activation functions in deep learning are bounded, meaning their output values fall within a specific range. This is important for several reasons, including:
- Stability: Bounded activation functions help prevent the network from becoming unstable during training due to exploding gradients.
- Vanishing gradient problem: Unbounded activation functions can suffer from the vanishing gradient problem, where gradients become very small for large inputs, making it difficult to train the network.
- Computational efficiency: Bounded activation functions are often computationally cheaper to evaluate than unbounded ones.
Here are some examples of commonly used activation functions and their boundedness:
- Sigmoid: Bounded between 0 and 1.
- Tanh: Bounded between -1 and 1.
- ReLU: Not bounded below, but bounded above by infinity.
- Leaky ReLU: Not bounded below, but bounded above by infinity.
- ELU: Bounded below by negative infinity, but bounded above by infinity.
- SELU: Bounded below by negative infinity, but bounded above by infinity.
Therefore, none of the commonly used activation functions are truly "not bounded" in the strict mathematical sense. However, some, like ReLU, ELU, and SELU, are unbounded in one direction, meaning their output can theoretically grow to infinity.
