Jacob draws the line. Targum Pseudo-Jonathan on Genesis 42:38 preserves his refusal: "My son shall not go down with you; for his brother is dead, and he alone remains of his mother; and if death should befall him in the way that you go, you will bring down my age with mourning to the grave."
Three sentences, three wounds
The first: his brother is dead. Jacob still believes Joseph is gone, and he cannot face a second loss from Rachel's line. The second: he alone remains of his mother. Rachel is also dead (Genesis 35:19). Benjamin is the only living thing left of the wife Jacob loved most. The third: you will bring down my age with mourning to the grave. This is the same phrase Jacob used about Joseph when he first believed him dead (Genesis 37:35) — he has not moved on. He is the same man, trapped in the same grief, twenty-two years later.
The weight of the word "you"
The Aramaic paraphrase, which took its final form in the Land of Israel around the seventh or eighth century CE, preserves Jacob's pronoun: the way that you go. Not "the way you travel." The way that you — specifically you, the sons who came back without Joseph once already — are going. Jacob is not simply afraid of bandits. He is afraid of his own sons' track record with lost brothers. The rabbinic tradition in Bereishit Rabbah 91:10 reads Jacob's hesitation as a quiet indictment: he does not fully trust the ten men standing in front of him, and he has reasons.
The takeaway
A father's no is sometimes the last protection a favored child has. Jacob's refusal will hold until hunger forces the family's hand — but the grief behind it reveals how much has been unsaid in his household.