For these are the men, through whom the light of Christ's gospel shone on you, O Rome, and through whom you, who wast the teacher of error, wast made the disciple of Truth. These are your holy Fathers and true shepherds, who gave you claims to be numbered among the heavenly kingdoms, and built you under much better and happier auspices than they, by whose zeal the first foundations of your walls were laid: and of whom the one that gave you your name defiled you with his brother's blood. These are they who promoted you to such glory, that being made a holy nation, a chosen people, a priestly and royal state, and the head of the world through the blessed Peter's holy See you attained a wider sway by the worship of God than by earthly government. For although you were increased by many victories, and extended your rule on land and sea, yet what your toils in war subdued is less than what the peace of Christ has conquered.
Leo the Great, Sermon 82. St. Leo, of course, is right: the rise in eminence of the bishops of Rome is closely associated with the fact that Rome was seen as the location of the martyrdom of both Peter and Paul under Nero; not only did this more closely link the See of Rome to Peter than other sees associated with Peter, it (combined with the relative ease of getting to Rome and the sheer number of saints who had been martyred at Rome in that and other persecutions) made Rome a primary pilgrimage site, and also made the bishops of Rome take seriously the role of successor to both Apostles -- they could hardly avoid being reminded of it. Thus we have symbolic association, lines of influence, and motivation to exercise influence. All that needs to be added is longstanding reputation for orthodoxy and the collapse of the Empire in the West to have the full suite of factors that builds the medieval papacy; and it quite literally all starts with Peter and Paul and what rippled out from their deaths.