The theories of quantum mechanics and relativity dramatically altered our understanding of the universe ushering in the era of modern physics. Quantum theory deals with objects probabilistically at small scales, whereas relativity deals classically with motion in space and time. We show here that the mathematical structures of quantum theory and of relativity follow together from pure thought, defined and uniquely constrained by the same elementary ``combining and sequencing'' symmetries that underlie standard arithmetic and probability. The key is uncertainty, which inevitably accompanies observation of quantity and imposes the use of pairs of numbers. The symmetries then lead directly to the use of complex \sqrt{-1} arithmetic, the standard calculus of quantum mechanics, and the Lorentz transformations of relativistic spacetime. One dimension of time and three dimensions of space are thus derived as the profound and inevitable framework of physics.