We study complex dynamics of two Rulkov neurons unidirectionally connected via a chemical synapse with respect to three control parameters: (i) a parameter responsible for the type of dynamical behavior of a solitary neuron, (ii) coupling strength, and (iii) noise intensity. The coupled system exhibits various scenarios on the route from a stable equilibrium to chaos with respect to the coupling strength. We observe a variety of dynamical regimes, including mono-, bi- and tri-stability, order-chaos transitions and vice versa, as well as the coexistence of in-phase and anti-phase synchronization. We also study transitions between in-phase and out-of-phase synchronization with statistics on the duration of synchronization intervals and transitions from order to chaos. In addition to numerical simulations, we demonstrate the effectiveness of the analytical confidence ellipses method based on stochastic sensitivity approach.