Hybrid inorganic-polymer nanocomposites can be employed in diverse applications due to the potential combination of desired properties from both the organic and inorganic components. The use of novel bottom-up in situ synthesis methods for the fabrication of these nanocomposites is advantageous compared to top-down ex situ mixing methods, as it offers increased control over the structure and properties of the material. In this review, the focus will be on the application of the sol-gel process for the synthesis of inorganic oxide nanoparticles in epoxy and polysiloxane matrices. The effect of the synthesis conditions and the reactants used on the inorganic structures formed, the interactions between the polymer chains and the inorganic nanoparticles, and the resulting properties of the nanocomposites are appraised from several studies over the last two decades. Lastly, alternative in situ techniques and the applications of various polymer-inorganic oxide nanocomposites are briefly discussed.