Starting from a (small) rigid C$^*$-tensor category $\mathscr{C}$ with simple unit, we construct von Neumann algebras associated to each of its objects. These algebras are factors and can be either semifinite (of type II$_1$ or II$_\infty$, depending on whether the spectrum of the category is finite or infinite) or they can be of type III$_\lambda$, $\lambda\in (0,1]$. The choice of type is tuned by the choice of Tomita structure (defined in the paper) on certain bimodules we use in the construction. Moreover, if the spectrum is infinite we realize the whole tensor category directly as endomorphisms of these algebras, with finite Jones index, by exhibiting a fully faithful unitary tensor functor $F:\mathscr{C} \hookrightarrow End_0(\Phi)$ where $\Phi$ is a factor (of type II or III). The construction relies on methods from free probability (full Fock space, amalgamated free products), it does not depend on amenability assumptions, and it can be applied to categories with uncountable spectrum (hence it provides an alternative answer to a conjecture of Yamagami \cite{Y3}). Even in the case of uncountably generated categories, we can refine the previous equivalence to obtain realizations on $\sigma$-finite factors as endomorphisms (in the type III case) and as bimodules (in the type II case). In the case of trivial Tomita structure, we recover the same algebra obtained in \cite{PopaS} and \cite{AMD}, namely the (countably generated) free group factor $L(F_\infty)$ if the given category has denumerable spectrum, while we get the free group factor with uncountably many generators if the spectrum is infinite and non-denumerable.