The Algorithm's Objective Function

When a decision tree algorithm evaluates a dataset, it does not possess human intuition regarding which variables are "important." Instead, it relies on a strict mathematical objective function: the minimization of impurity. To achieve this, it relies on fundamental concepts from Information Theory—specifically Entropy and Information Gain.

Quantifying Chaos with Entropy

Entropy is a quantitative measure of the unpredictability or "chaos" within a specific subset of data. If a dataset contains 1,000 users and exactly 500 churned while 500 renewed, the probability of either class is 0.5. Mathematically, this yields an entropy score of 1.0 (maximum impurity). The algorithm is completely uncertain about the outcome of a random sample.

Conversely, if a subset contains 100 users and all 100 renewed, the probability of churn is 0. This yields an entropy score of 0.0 (perfect purity). The primary objective of the algorithm is to transition the data from a state of high entropy at the root node to a state of minimal entropy at the terminal leaves.

Information Gain: The Metric of Entropy Reduction

Information Gain is the calculated difference in entropy before and after a proposed split.

• The algorithm exhaustively evaluates every possible feature (e.g., Age, Income) and every possible threshold value within the continuous feature space.
• For each candidate split, it calculates the weighted average entropy of the resulting child nodes.
• The split that yields the lowest resulting entropy—and therefore the highest Information Gain—is deterministically selected as the optimal split for that specific node.

While this exhaustive search guarantees a mathematically optimal split at that exact local node (a greedy approach), it often leads to overfitting. This is why interactive tools expose these Information Gain metrics visually, allowing human experts to overrule the algorithm and select splits that offer superior real-world generalization.