What you're saying is what probably would happen. Since all three branches comprise the federal government, then if all three branches agreed, they could absolutely change the cons ution or even abolish it. The likelihood of that happening is small, but it's within their power to do so.
There are instances in history where this kind of thing has happened. It could happen in the US. Otherwise the watchdogs would be nuts for sounding out when these rights get violated by new bills, and some that have even been made into laws that later get overturned.