I will review the various space-time structures associated with Galileian, Minkowskian, generally non-relativistic and generally relativistic theories, and show why generally covariant space-times are fundamentally different from their predecessors (hole argument). this difference puts into question the usual starting point of space-time theories: a fixed manifold composed of distinguishable points. I will show that this basic feature of generally covariant theories does not depend on the presence of a differentiable or even a topological structure, but can be generalized to any set subject to certain relations. Some proposals to use mathematical structures that do not start with point sets to describe space-time structures will be reviewed.