Is Zermatt worth the detour?
Zermatt is worth it if you give it enough time and accept that the Matterhorn does not appear on command. Go for the whole mountain setting, not just a single view you expect to collect between trains.
Answer
Zermatt can be one of the places where Switzerland feels almost absurdly scenic: car-free streets, high peaks, mountain railways and the Matterhorn sitting above the whole idea of the trip. But the detour has a cost. Accommodation, food and mountain transport can be expensive, and cloud can hide the very view many travellers came for. Zermatt works best when it is a real mountain stay, not a one-night gamble jammed between other famous regions. If the Matterhorn is central to the trip, give it room. If it is just one more name on an already busy route, the Bernese Oberland may be enough.