Ibn ‘Umar (may Allah be pleased with him) reported that he heard a man saying: “No, by the Ka‘bah.” He said to him: "Do not swear by other than Allah, for I heard the Messenger of Allah (may Allah’s peace and blessings be upon him) say: 'Whoever swears by other than Allah has disbelieved or associated partners with Allah.'"
The Prophet (may Allah's peace and blessings be upon him) informs that whoever swears by other than Allah and His names and attributes has disbelieved in Allah or associated partners with Him. This is because swearing entails extolment of the one by whom swearing is done, whereas all greatness belongs to Allah alone. So, we must only swear by Allah, His names, and attributes. This type of swearing falls under minor polytheism. However, if the swearer extols what he swears by like the extolment of Allah or greater, then his swearing constitutes major polytheism.