(Gen. 38:1:) "And Judah went down." What is the meaning of "went down"? That they brought him down from his greatness. Why? Because they had appointed him king over them. When they sold Joseph, they came to their father to comfort him, but he refused to be comforted. They said: Now the old man holds a grievance against us. What did they do? They arose and banished him (Judah), as it is said, "And Judah went down."
Why Judah Went Down and His Brothers Deposed Their King
Midrash Tanchuma Buber, Vayeshev 12:1
וירד יהודה. מהו וירד שהורידוהו מגדולתו, למה שמינו אותו מלך עליהם, כיון שמכרו את יוסף נכנסו אצל אביהם לנחמו, [ולא] ביקש להתנחם, אמרו עכשיו הזקן מקפיד עלינו, מה עשו עמדו ונידו אותו, שנאמר וירד יהודה.