In orbital mechanics, Rayleigh instability results from radial potentials where the angular momentum of circular orbits is a decreasing function of distance from the center. This is not to confused with Rayleigh-Taylor instability, which is an instability in fluid mechanics.
Mathematically, the Rayleigh instability results from the epicyclic frequency becoming imaginary
where is the angular frequency at radius from the center. The epicyclic frequency is the frequency of radial motions caused by a small perturbation to a circular orbit: where is the radial perturbation relative to a circular orbit.
It is easy to see that if the angular momentum decreases with distance, the epicyclic frequency becomes imaginary: thus leading to an instability.
The physical mechanism for the instability can be understood as follows:
Consider a test particle on a circular orbit. Now imagine the particle is radially perturbed to a slightly larger radius. The angular momentum corresponding to circular motion at this new radius is less than the angular momentum of the particle, since the angular momentum is a decreasin function of radius. Therefore, the particle reaches its new orbit with too much angular momentum, the centrifugal force becomes larger than the inward force, and the particle escapes to infinity.
On the other hand, if the particle is perturbed to a slightly smaller radius, then it reaches its new orbit with too small angular momentum. The centrifugal force is not strong enough to support the particle against the inward pull, and the particle falls inward.
It is worth noting that for Keplerian motions, we have so such orbits are stable to Rayleigh instability, which is why radial perturbations to circular orbits result in stable, closed, ellipses.