What Is Machine Learning? An Introduction

Machine Learning (ML) is the branch of Artificial Intelligence (AI) that builds systems capable of learning from a dataset without being programmed step by step for every task. These systems infer techniques and algorithms from examples, then produce predictive models and behavior patterns on new data.

ML overlaps heavily with statistics. The key difference is emphasis: statistics focuses on inference and uncertainty; ML focuses on computational complexity. Most interesting ML problems belong to the NP-complete or NP-hard class, so the discipline centers on designing tractable solutions to problems that cannot be solved optimally in reasonable time.

Two Classic Definitions

Arthur Samuel (1959): Machine Learning is a field of study that gives computers the ability to learn without being explicitly programmed.

Tom Mitchell (1998): A program learns from experience E with respect to task T and performance measure P if its performance on T, as measured by P, improves with experience E.

Regression vs. Classification

ML systems produce output in one of two forms.

Regression maps input data to a continuous output. Both the input domain X and the output domain Y are arbitrary. A simple example is the function f: R → R that predicts calories burned given weight, run time, and speed.

Classification maps input data to a finite output set. The input domain X is arbitrary; the output Y is a small finite set such as {0, 1}, {win, draw, loss}, or {yes, no}.

After a learning phase, the trained system acts as an expert system: given new input it applies the learned function and returns a prediction, recommendation, or classification.

How a System Learns: The Hypothesis

Every ML system learns by finding a hypothesis (model or function) that fits a labeled training set well enough to generalize to unseen test data.

The learning goal is to minimize empirical error on training data while keeping test error low. A hypothesis that memorizes training labels exactly tends to fail on new data. A hypothesis that is too coarse misses real patterns. Finding the right balance is the central challenge.

Overfitting and Underfitting

Two failure modes occur when a hypothesis does not generalize.

Overfitting happens when the hypothesis fits the training data too closely, capturing noise rather than signal. In a cricket match predictor, if the training set contains only 3 India-vs-Bangladesh matches (all Bangladesh wins), an overfitting model predicts "Bangladesh wins" even though India is historically stronger.

Underfitting happens when the hypothesis is too coarse to capture the real pattern. An underfitting model might predict "home team always wins" and fail to account for team strength at all.

Optimal generalization sits between the two: the hypothesis predicts India wins because India is one of the world's strongest teams, it wins the majority of its matches, and home teams win about 60% of the time in the training data.

These failure modes appear visually in both classification and regression:

The chart below shows how test error rises at both extremes of model complexity:

6 Types of Learning

The type of available training data determines which learning approach applies.

1. Supervised learning. Training data includes both input and output labels. The system learns a mapping from inputs to outputs. Produces the strongest results when labeled data is plentiful.
2. Unsupervised learning. Only input data is available. The goal is to discover structure in the input domain, such as clusters or latent dimensions.
3. Semi-supervised learning. A hybrid: a small labeled set combined with a large unlabeled set. Useful when labeling is expensive.
4. Adaptive learning. Starts from an existing model and updates its parameters as new training data arrives, rather than learning from scratch.
5. Online learning. No fixed training-then-test split. The system learns continuously during prediction, with a human validating or correcting each output in real time.
6. Reinforcement learning. A hybrid of online and semi-supervised learning. Supervision is incomplete: the system receives a sparse reward signal ({+1, -1} or {reward, penalty}) and must discover the policy that maximizes cumulative reward.

4 Evaluation Methods

After training, the hypothesis must be evaluated on held-out data. Four standard methods control how training and test sets are divided.

1. Validation (resubstitution). All available data serves as both training and test. Optimistic: error is underestimated because the hypothesis is evaluated on the same data it was trained on.
2. Hold-out (partition). Data is split into a training subset and a test subset. Simple, but test data is unavailable during training, which shrinks the effective training set.
3. Cross-validation. Data is split into N random blocks (folds). Each fold serves as the test set once; the model is trained on the remaining N-1 folds. Test error is averaged across all N runs. Drawback: training set size drops as fold count rises.
4. Leave-one-out. Each single data point serves as the test set once; the model trains on all remaining points. Equivalent to N-fold cross-validation where N equals the dataset size. Computationally expensive for large datasets.

ML Techniques and Problem Categories

Classifying ML techniques by the problem they address:

<table> <thead> <tr><th>Problem type</th><th>Supervised methods</th><th>Clustering methods</th></tr> </thead> <tbody> <tr> <td> <ul> <li>Classification</li> <li>Clustering</li> <li>Regression</li> <li>Anomaly detection</li> <li>Association rules</li> <li>Reinforcement learning</li> <li>Structured prediction</li> <li>Feature engineering</li> <li>Feature learning</li> <li>Online learning</li> <li>Semi-supervised learning</li> <li>Unsupervised learning</li> <li>Learning to rank</li> <li>Grammar induction</li> </ul> </td> <td> <ul> <li>Decision trees</li> <li>Ensembles (Bagging, Boosting, Random Forest)</li> <li>k-NN</li> <li>Linear regression</li> <li>Naive Bayes</li> <li>Neural networks</li> <li>Logistic regression</li> <li>Perceptron</li> <li>Relevance vector machine (RVM)</li> <li>Support vector machine (SVM)</li> </ul> </td> <td> <ul> <li>BIRCH</li> <li>Hierarchical</li> <li>K-means</li> <li>Expectation-maximization (EM)</li> <li>DBSCAN</li> <li>OPTICS</li> <li>Mean-shift</li> </ul> </td> </tr> </tbody> </table>

<table> <thead> <tr><th>Dimensionality reduction</th><th>Probabilistic models</th><th>Anomaly detection</th></tr> </thead> <tbody> <tr> <td> <ul> <li>Factor analysis</li> <li>Canonical correlation (CCA)</li> <li>Independent component analysis (ICA)</li> <li>Linear discriminant analysis (LDA)</li> <li>Non-negative matrix factorization (NMF)</li> <li>Principal component analysis (PCA)</li> <li>t-distributed stochastic neighbor embedding (t-SNE)</li> </ul> </td> <td> <ul> <li>Probabilistic graphical models (PGM)</li> <li>Bayesian networks</li> <li>Conditional random field (CRF)</li> <li>Hidden Markov model (HMM)</li> </ul> </td> <td> <ul> <li>k-nearest neighbors (k-NN)</li> <li>Local outlier factor</li> </ul> </td> </tr> </tbody> </table>

<table> <thead> <tr><th>Neural networks</th><th>Recommendation systems</th><th>Theoretical frameworks</th></tr> </thead> <tbody> <tr> <td> <ul> <li>Deep learning</li> <li>Multilayer perceptron</li> <li>Recurrent neural network (RNN)</li> <li>Restricted Boltzmann machine</li> <li>Self-organizing map (SOM)</li> <li>Convolutional neural network (CNN)</li> </ul> </td> <td> <ul> <li>k-nearest neighbors (k-NN)</li> <li>Singular value decomposition (SVD)</li> <li>Latent semantic indexing (LSI)</li> <li>Probabilistic latent semantic indexing (PLSI)</li> <li>Latent Dirichlet allocation (LDA)</li> </ul> </td> <td> <ul> <li>Bias-variance trade-off</li> <li>Computational learning theory</li> <li>Empirical risk minimization</li> <li>Occam learning</li> <li>PAC learning</li> <li>Statistical learning theory</li> <li>VC theory</li> </ul> </td> </tr> </tbody> </table>

Further Reading

For hands-on practice, the Stanford open classroom ML course and the Caltech video library both cover the fundamentals in depth. Tom Mitchell's textbook Machine Learning (McGraw-Hill) remains a clear reference for the theory above.

If you are working through a course that includes ML assignments and need support, GeeksProgramming has worked on ML projects since 2014. See the Machine Learning Assignment Help page for details on how submissions work. For a practical Python walkthrough of scikit-learn, neural networks, and CNNs applied to these same concepts, read Machine Learning with Python: A Practical Guide.