We study gravitational perturbations around the near horizon geometry of the (near) extreme Kerr black hole. By considering a consistent truncation for the metric fluctuations, we obtain a solution to the linearized Einstein equations. The dynamics is governed by two master fields which, in the context of the nAdS$_2$/nCFT$_1$ correspondence, are both irrelevant operators of conformal dimension $\Delta=2$. These fields control the departure from extremality by breaking the conformal symmetry of the near horizon region. One of the master fields is tied to large diffeomorphisms of the near horizon, with its equations of motion compatible with a Schwarzian effective action. The other field is essential for a consistent description of the geometry away from the horizon.