Over the last two decades, the dominant paradigm in data science has been the expansion of data scale. The rise of distributed computing, cloud infrastructures, and machine learning created an environment where the central question became how to collect, store, and analyze ever larger datasets.
This paradigm, commonly referred to as Big Data, has delivered significant advances in prediction, modeling, and large-scale pattern discovery. However, as organizations increased their analytical capacity, an unexpected limitation emerged: decision latency.
Many operational decisions cannot wait for the complete analysis of large datasets. Fraud detection must occur during the transaction. Medical triage must happen during the consultation. Supply chain disruptions require immediate response. In such contexts, the value of a decision is often determined not only by its accuracy, but by how quickly it can be made.
This reality reveals a fundamental gap in the current data science paradigm. While Big Data optimizes the completeness of information, real-world decisions frequently require sufficiency of context. The discipline proposed here — Small Data — addresses this gap.
Small Data does not refer to small datasets. It refers to the minimum contextual information required to make a reliable decision in real time, within environments that may contain vast volumes of data.
Small Data therefore emerges as a complementary decision discipline to Big Data. Where Big Data seeks to understand the entire system, Small Data seeks to determine what is necessary to act now. This manifesto establishes the scientific foundations for this discipline.
Principle 1: The Principle of Contextual Sufficiency
For most operational decisions, there exists a minimum subset of variables that preserves the majority of the decision power of the full system. Let the full information space of a decision environment be defined as:
Small Data seeks a subset:
where MCS denotes the Minimum Context Set such that:
under acceptable operational thresholds.
This principle implies that decision quality is often nonlinearly distributed across variables. A small number of signals frequently carries the majority of actionable information. The role of the Small Data discipline is therefore to identify, validate, and operationalize these minimal sets.
Principle 2: The Principle of Decision Latency
The value of a decision is a function not only of accuracy but also of time. We define the Decision Utility Function as:
where
A represents decision accuracy
T represents decision time.
In many operational environments, the marginal value of additional information decreases as decision latency increases. Waiting for more information may increase accuracy but reduce the usefulness of the decision. Small Data addresses this trade-off by optimizing for timely sufficiency rather than informational completeness.
Principle 3: The Principle of Real-Time Context Compression
Complex decision systems often operate within high-dimensional information spaces. However, the decision boundary that separates actionable outcomes can frequently be approximated using far fewer dimensions.
Small Data therefore frames decision-making as a context compression problem. Given a decision function:
the objective becomes finding an approximation:
such that
while minimizing the number of variables and maximizing decision speed. This compression allows real-time decisions in systems where full-model inference would be computationally or operationally impractical.
Principle 4: The Complementarity Principle
Small Data is not an alternative to Big Data. It is a complementary discipline. Big Data is optimized for:
discovery
retrospective analysis
model training
system understanding
Small Data is optimized for:
real-time action
operational decisions
environments with limited context
latency-sensitive systems
In modern data infrastructures, Big Data systems often generate the models, while Small Data systems execute the decisions. This creates a two-layer architecture of intelligence:
Analytical Layer (Big Data) — learns the system.
Decision Layer (Small Data) — acts in real time.
The Central Research Question
The Small Data discipline revolves around a single guiding question: What is the minimum context required to make a reliable decision in real time?
This question has implications across multiple domains including: financial systems, healthcare diagnostics, digital products, supply chain operations, public policy, venture capital, cybersecurity, and personal decision-making.
Each domain presents unique trade-offs between information availability, decision speed, and acceptable uncertainty. The role of the Small Data discipline is to systematically study these trade-offs.
The Research Program
The research program of the Small Data discipline consists of three core objectives.
The first objective is identification. Determine the minimal contextual variables required for specific classes of decisions.
The second objective is validation. Empirically test whether reduced-context models preserve operational performance.
The third objective is operationalization. Design architectures capable of deploying minimal-context decision models in real-time systems.
Together, these objectives transform Small Data from an abstract concept into an applied scientific discipline.
Implications
The implications of this discipline extend beyond data science.
Organizations frequently suffer not from lack of information, but from excessive informational dependency before action is taken. This creates analytical bottlenecks that delay decisions in dynamic environments.
Small Data proposes a different approach: design decision systems that operate with minimal sufficient context, allowing organizations to move at the speed of events.
This shift reframes the goal of data science. The objective is no longer merely to analyze more data, but to act with the right data at the right moment.
Closing Statement
The emergence of Big Data expanded humanity’s capacity to understand complex systems. The next frontier lies in transforming that understanding into timely and effective decisions.
Small Data represents the scientific effort to identify the minimum information required for action in real time. By focusing on contextual sufficiency, decision latency, and real-time context compression, this discipline aims to bridge the gap between analytical knowledge and operational response. The future of intelligent systems will not depend solely on how much data we possess, but on how little information we truly need to act wisely.