Why specifically when Joseph was born [did Jacob seek to leave]? Because Jacob our father saw that the seed of Esau would be delivered only into the hand of the seed of Joseph, as it is said, "And the house of Jacob shall be fire, and the house of Joseph a flame, and the house of Esau for stubble" (Obadiah 1:18). One raised an objection against him: "And David smote them from the twilight" and so forth (I Samuel 30:17) [implying David, of Judah, struck Amalek, of Esau's line]. He said to him: He who taught you the Prophets did not teach you the Writings, for it is written, "as he went to Ziklag, there fell to him men of Manasseh" [Joseph's son] (I Chronicles 12:21). One objected: "of the sons of Simeon went to Mount Seir" and so forth (I Chronicles 4:42-43) [Simeon, not Joseph, struck Esau's Seir]. Their leader Yishi too came from Manasseh, as it is written, "and the sons of the half-tribe of Manasseh... and Epher and Yishi" [text uncertain].
Esau Falls Only by the House of Joseph
Yalkut Shimoni on Torah 130:5
מַאי שְׁנָא כִּי אִתְיְלִיד יוֹסֵף אֶלָּא רָאָה יַעֲקֹב אָבִינוּ שֶׁאֵין זַרְעוֹ שֶׁל עֵשָׂו נִמְסָר אֶלָּא בְּיָד זַרְעוֹ שֶׁל יוֹסֵף, שֶׁנֶּאֱמַר (עובדיה א, יח) "וְהָיָה בֵית יַעֲקֹב אֵשׁ וּבֵית יוֹסֵף לֶהָבָה וּבֵית עֵשָׂו לְקַשׁ". אֵיתֵיבֵיה, (שמואל א ל, יז) "וַיַּכֵּם דָּוִד מֵהַנֶּשֶׁף" וְגוֹ' אָמַר לֵיהּ, דְּאַקְרְיֵךְ נְבִיאֵי לָא אַקְרְיֵךְ כְּתוּבִים, דִּכְתִיב "בְּלֶכְתּוֹ אֶל צִיקְלַג נָפְלוּ עָלָיו מִמְּנַשֶּׁה" מֵיתֵיבֵי (שם ד, מב מג) "מִן בְּנֵי שִׁמְעוֹן הָלְכוּ לְהַר שֵׂעִיר" וְגוֹ' יִשְׁעִי נַמִּי מִמְּנַשֶּׁה קָאָתִי, דִּכְתִיב "וּבְנֵי [חֲצִי שֵׁבֶט] מְנַשֶּׁה וְגוֹ' וְעֵפֶר וְיִשְׁעִי".