We study experimentally and theoretically the process of nanostructuring burnishing by a spherical indenter and show that if the smoothing surfaces contain a localized unevenness (a drilled hole), a stable limiting cycle of self-excited oscillations may occur, which results in wavy permanent deformation of the surface. We propose a numerical model for computing the topography of a surface that has been subjected to nanostructuring burnishing and show that it is in a good qualitative agreement with the experimental observations. Based on the model, the critical value of the damping is determined suppressing the oscillating mode.