In this paper a drift-randomized Milstein method is introduced for the numerical solution of non-autonomous stochastic differential equations with non-differentiable drift coefficient functions. Compared to standard Milstein-type methods we obtain higher order convergence rates in the $L^p(\Omega)$ and almost sure sense. An important ingredient in the error analysis are randomized quadrature rules for Hölder continuous stochastic processes. By this we avoid the use of standard arguments based on the Itō-Taylor expansion which are typically applied in error estimates of the classical Milstein method but require additional smoothness of the drift and diffusion coefficient functions. We also discuss the optimality of our convergence rates. Finally, the question of implementation is addressed in a numerical experiment.