The reason why UnityEngine.UI.Button doesn't fit your use case is because it implement IPointerClickHandler, and the behavior you described is exactly what PointerClick event designed to be. See ...