Can agents reply to ticket notification emails from their email client to update the ticket and notify the customer, without logging into osTicket?

No. Agents must log into the osTicket web interface to reply to tickets. There is no native configuration or plugin to enable agent-side email replies that update tickets and are sent to customers. Replies from agents to notification emails are treated as internal data only.