What if spam runs out of my mailbox quota? Will the new messages be returned to the sender?

No, if the quota is full, it will try to delete the oldest messages in the "Spam + virus" folder before returning a new message to the sender as undelivered.