If you want to remove user from the Assignment process, then you will need to remove the user from the queue (that you added in the Assignment Rule record). When you remove the user from queue, the user workload record of that user automatically gets deleted, and records will no longer be assigned to that user.