"Privileges" are given to slaves, not freeman. A man of high rank doesn't acknowledge anyone above himself with the authority to grant him "privileges" (which implies the authority to take them away upon a whim). If you think of yourself as having "privileges" it means you belong to the servile classes. It is not in spite of those privileges that you are a slave; the fact that you are allowed certain privileges is proof that you are a slave.
Therefore it is the case that all social group of any significance, even if they are at the bottom of society, are allowed some privileges. The ruling class wants every group to accept some form of "privilege" because doing so only reinforces the authority of the ruling powers. It is a way of getting you to acknowledge the fact that they have the right to give permissions and privilges to different sections of society. So any group of people that is recognised as "privileged", is not the ruler of anthing.
To be the beneficiary of "privilege" is to be granted permissions by a superior power. People who are given "privileges" are by definition the subordinates of some higher authority. To believe that one is a recipient of "privileges" is to accept a subordination of rank in the social hierarchy. That is, if one regards oneself as "white privileged", one is tacitly admitting to being a slave.
The real rulers of society don't receive "privileges" from anyone; they have what they believe to be their own inherent rights, which cannot be given or taken away by a higher authority. Rights can only be be violated and transgressed, not taken away or abolished. Rights are actively asserted, privileges are received passively by servile castes. They can be taken away arbitrarily.
In summary, there are no "white people"; consequently, there is no "white privilege". And privileges, in any case, are accorded to subordinates and slaves. If you are "privileged", then you are not the ruler of anything.