Fair vs Unfair Challenge

A challenge can come in many forms. From learning a new language to pushing yourself just that little bit harder at the gym, it serves as a way to test your abilities and provide insight into where you are at this current moment. Video games are no different. But one question that is often asked is this. When does a game go from a good challenge to an unfair, controller-breaking annoyance?