Our Rabbis taught: if a man married a woman and stayed with her ten years and she did not bear, he is not permitted to remain idle. He must divorce her and give the marriage settlement, for perhaps he was not granted the merit to have children from her. And although there is no proof of the matter, there is a hint of it, as it says, "at the end of ten years of Abram's dwelling in the land of Canaan" (Genesis 16:3). This teaches you that dwelling outside the Land does not count in the reckoning. Therefore, if he was sick, or she was sick, or both were imprisoned in confinement, those years do not count for them in the reckoning. Rava said to Rav Nachman: let us learn instead from Isaac, of whom it is written, "And Isaac was forty years old" and "Isaac was sixty years old when she bore them" (Genesis 25:20, 26). He said to him: Isaac was sterile [and could not father earlier]. If so, Abraham too was sterile? Rather, that verse is needed for the teaching of Rabbi Chiyya bar Abba in the name of Rabbi Yochanan: why were the years of Ishmael counted? In order to reckon by them the years of Jacob.
The Ten-Year Rule for Childlessness Learned from Abram in Canaan
Yalkut Shimoni on Torah 79:2
תָּנוּ רַבָּנָן, נָשָׂא אִשָּׁה וְשָׁהָה עִמָּה עֶשֶׂר שָׁנִים וְלֹא יָלְדָה, אֵינוֹ רַשָּׁאי לִבָּטֵל, וְיוֹצִיא וְיִתֵּן כְּתוּבָּה שֶׁמָּא לֹא זָכָה לְבָנִים מִמֶּנָּה. וְאַף עַל פִּי שֶׁאֵין רְאָיָה לַדָּבָר זֵכֶר לַדָּבָר, שֶׁנֶּאֱמַר, מִקֵּץ עֶשֶׂר שָׁנִים לְשֶׁבֶת אַבְרָם בְּאֶרֶץ כְּנָעַן לְלַמֶּדְךָ שֶׁאֵין יְשִׁיבַת חוּצָה לָאָרֶץ עוֹלָה מִן הַמִּנְיָן לְפִיכָךְ הָיָה הוּא חוֹלֶה אוֹ הִיא חוֹלָה אוֹ שֶׁהָיוּ שְׁנֵיהֶם חֲבוּשִׁין בְּבֵית הָאֲסוּרִין, אֵין עוֹלֶה לָהֶן מִן הַמִּנְיָן. אָמַר לֵיהּ רָבָא לְרַב נַחְמָן, וְלֵילַף מִיִּצְחָק דִּכְתִיב, (בראשית כה, כ) "וַיְּהִי יִצְחָק בֶּן אַרְבָּעִים שָׁנָה" וְגוֹ' (שם כו) "וְיִצְחָק בֶּן שִׁשִּׁים שָׁנָה בְּלֶדֶת אֹתָם", אָמַר לֵיהּ, יִצְחָק עָקוּר הָיָה אִי הָכִי אַבְרָהָם נַמִּי עָקוּר הָיָה, אֶלָּא הַהִיא מִבַּעְיָא לֵיהּ לִכְדְרַבִּי חִיָּא בַּר אַבָּא אָמַר רַבִּי יוֹחָנָן, לָמָּה נִמְנוּ שְׁנוֹתָיו שֶׁל יִשְׁמָעֵאל, כְּדֵי לְיַחֵס בָּהֶן שְׁנוֹתָיו שֶׁל יַעֲקֹב.