Partition map exchange is an absolutely necessary procedure, that cannot be disabled. Functionality of all caches depend on it.
I checked, and a cache destruction is performed as a part of a partition map exchange, and not the opposite. If you see, that nodes join the cluster fast, but cache destruction is slow, then exchange is not the reason of the slowness, since PME also happens every time when nodes join and leave the cluster.
How much data do you have in the cache? Cache destruction requires iteration over the whole dataset, so it may take quite a while.