In this work there is carried out an investigation of tooth different pitches with the purpose to increase a technological system vibration resistance at processing by end mills with a variable pitch and using elements of an operation modal analysis (OMA). An experiment is carried out in the course of which there are performed tests for the level definition of vibrations and resonance frequencies of technological system oscillations at cutting by an end mill with a constant and varied pitch. The work results show that the average level of vibrations at the work with a mill with a constant pitch is higher than at the work with a mill with a varied one. The existence of some own frequencies in one direction for a mill with a constant pitch of teeth is defined that is possible in the systems with a great number of degrees of mobility. At the same time for a mill with a varied pitch the oscillations were observed only at one frequency of own oscillations in one direction. In this connection a possibility of resonance oscillations decreases. During the work of the mill with a varied pitch the vibration waves with different length appear on the surfaces of cutting. A developed physical model of a technological system oscillation process at the work with a mill with a varied pitch shows that with the increase of a circular pitch the resultant oscillation displacements decrease.