Only remove from queue if it's our download