Tax-free inbound sales at over 80 department stores across the country fell by 41% in May from a year earlier.