If a man adjured witnesses five times outside the court, and afterward they came into court and admitted the truth, one might think they are liable; Scripture says "if he does not tell" (Leviticus 5:1). The rule applies only where, if this man tells, the other becomes obligated for money - and where is that? In court. From here the sages said: if he adjured them five times outside the court and they denied knowledge, then came into court and admitted the truth, they are exempt. If they admitted the truth outside but denied it in court, they are liable. If he adjured them five times before the court and they denied knowledge, they are liable only once. Rabbi Yishmael said: what is the reason? Because they can no longer retract and confess. If both witnesses denied together, they are liable. If one after the other, the first is liable. If one denied and the other confessed, the one who denied is liable. If there were two sets of witnesses and the first set denied and then the second set denied, both are liable, because the testimony could have been established through either of them. Rav Pappa said to Abaye: they are not liable until they deny knowledge in court. If so, even the oath itself must be in court - in court yes, outside court no? That cannot enter your mind, for it was taught that "in one" (Leviticus 5:5) comes to make one liable for each and every oath. But we learned: if he adjured them five times in court and they denied, they are liable only once - and Rabbi Shimon said the reason is that they can no longer retract and confess. Learn from this, then, that the oath may be outside the court while the denial must be in court.
Five Oaths Outside Court and the Window Left Open for Confession
Yalkut Shimoni on Torah 470:3
הִשְׁבִּיעַ עֲלֵיהֶם חֲמִשָּׁה פְּעָמִים חוּץ לְבֵית דִּין וּבָאוּ לְבֵית דִּין וְהוֹדוּ, יָכוֹל יְהוּ חַיָּבִין, תַּלְמוּד לוֹמַר "אִם לֹא יַגִּיד", לֹא אָמַרְתִּי אֶלָּא בְּמָקוֹם שֶׁאִם מַגִּיד זֶה, נִתְחַיֵּב זֶה מָמוֹן, וְאֵיזֶה זֶה בְּבֵית דִּין. מִכָּאן אָמְרוּ הִשְׁבִּיעַ עֲלֵיהֶם חֲמִשָּׁה פְּעָמִים חוּץ לְבֵית דִּין וְכָפְרוּ, וּבָאוּ לְבֵית דִּין וְהוֹדוּ פְּטוּרִין. בַּחוּץ הוֹדוּ וּבְבֵית דִּין כָּפְרוּ חַיָּבִין, הִשְׁבִּיעַ עֲלֵיהֶם חֲמִשָּׁה פְּעָמִים לִפְנֵי בֵּית דִּין וְכָפְרוּ אֵין חַיָבִין אֶלָּא אַחַת. אָמַר רַבִּי יִשְׁמָעֵאל מַה טַּעַם, מִפְּנֵי שֶׁאֵין יְכוֹלִין לַחֲזֹר וּלְהוֹדוֹת. כָּפְרוּ שְׁנֵיהֶן כְּאֶחָד חַיָּבִים. בָּזֶה אַחַר זֶה הָרִאשׁוֹן חַיָּב, כָּפַר אֶחָד וְהוֹדָה אֶחָד הַכּוֹפֵר חַיָּב. הָיוּ שְׁתֵּי כִּתֵּי עֵדוּת, כָּפְרָה הָרִאשׁוֹנָה וְאַחַר כָּךְ כָּפְרָה שְׁנִיָּה שְׁתֵּיהֶם חַיָּבוֹת, מִפְּנֵי שֶׁהָעֵדוּת יְכוֹלָה לְהִתְקַיֵּם בִּשְׁתֵּיהֶן. אָמַר לֵיהּ רַב פַּפָּא לְאַבַּיֵּי אֵין חַיָּבִין עַד שֶׁיִּכְפְּרוּ בְּהוּ בְּבֵית דִּין. אִי הָכִי אֲפִלּוּ שְׁבוּעָה נַמִּי, בְּבֵית דִּין אִין שֶׁלֹּא בְּבֵית דִּין לָא. לָא סָלְקָא דַעְתָּךְ דְּתַנְיָא "לְאַחַת" לְחַיֵּב עַל כָּל אַחַת וְאַחַת. וְהָתְנַן הִשְׁבִּיעַ עֲלֵיהֶן חֲמִשָּׁה פְּעָמִים בְּבֵית דִּין וְכָפְרוּ אֵין חַיָּבִין אֶלָּא אַחַת, וְאָמַר רַבִּי שִׁמְעוֹן מַאי טַעְמָא הוֹאִיל וְאֵין יְכוֹלִין לַחֲזֹר וּלְהוֹדוֹת, אֶלָּא לָאו שְׁמַע מִינַהּ שְׁבוּעָה חוּץ לְבֵית דִּין כְּפִירָה בְּבֵית דִּין.