This Ayah contains Allah's order to those who stand at `Arafat to also move on to Al-Muzdalifah, so that they remember Allah at Al-Mash`ar Al-Haram. Allah commands the Muslim to stand with the rest of the pilgrims at `Arafat, unlike Quraysh who (before Islam) used to remain in the sanctuary, near Al-Muzdalifah, saying that they are the people of Allah's Town and the servants of His House. Al-Bukhari reported that `A'ishah said, "Quraysh and their allies, who used to be called Al-Hums, used to stay in Al-Muzdalifah while the rest of the Arabs would stand at `Arafat. When Islam came, Allah commanded His Prophet to stand at `Arafat and then proceed from there. Hence Allah's statement:
﴿ مِنۡ حَيۡثُ أَفَاضَ ٱلنَّاسُ ﴾
(...from the place whence all the people depart.)
This was also said by Ibn `Abbas, Mujahid, `Ata', Qatadah and As-Suddi and others. Ibn Jarir chose this opinion and said that there is Ijma` (a consensus among the scholars) for it.
Imam Ahmad reported that Jubayr bin Mut`im said, "My camel was lost and I went out in search of it on the day of `Arafah, and I saw the Prophet standing in `Arafat. I said to myself, `By Allah he is from the Hums. What has brought him here''' This Hadith is also reported in the Sahihayn. Al-Bukhari reported that Ibn `Abbas said that `depart' mentioned in the Ayah refers to proceeding from Al-Muzdalifah to Mina to stone the pillars. Allah knows best.