Moodle can be made to perform very well, at small usage levels or scaling up to many thousands of users. The factors involved in performance are basically the same as for any PHP-based database-driven system. When trying to optimize your server, try to focus on the factor which will make the most difference to the user. For example, if you have relatively more users browsing than accessing the database, look to improve the webserver performance.