Does osTicket support agents replying to tickets by email without logging in?

No. This is not natively supported. Agents must log into the web interface to post replies that are sent to the customer. Replies to notification emails are not processed as outbound customer responses.