You have the constraint just deferrable, not deferred.
You defer the constraint using the "SET CONSTRAINT ... DEFERRED" command.
After that, you will be able to swap the values.

Changing values in a primary key is not a good idea, by the way.