Your site isn’t accessible when it hasn’t loaded, but is it accessible once it has? Are you thrashing the main UI thread, impeding user interaction? Did you disable zooming, making your site illegible to many? To make your clients happy, your site must be accessible: it has to make it onto to device quickly, has to work once there, and has to have good usability no matter the current physical and cognitive status of your user.
The goal of web performance is to make your customers happy. If your customers can’t access your site because it’s too slow, they’ll leave your site and become someone else’s customers instead of yours. Similarly, even if your site does load quickly but is not accessible, it will result in your visitors being unhappy, potentially losing them as potential customer for life.