I have around 50 WordPress sites (all for clients). A couple are quite large websites; one’s an alumni community for a High School, one’s an eCommerce website.
Both of these are are plugin heavy. For example the alumni website has 65 plugins installs, 54 active.
During busy times they can both be a little sluggish (by that I mean takes a couple of seconds before the first visual elements load but still less than 10 seconds to fully complete loading).
I use server-side caching as well as W3 Total Cache, these do a great job on all my websites. My server’s pretty powerful too. The theme’s optimized well, as are all the images etc.
Unfortunately plugins can be a bit sloppy at times. Ideally a plugin should only load up site resources when needed — on one page, or pages with a certain element, etc. As someone who uses a lot of plugins as well, I can definitely tell you it’s the case that most plugins don’t abide by that.
What I’ve done to combat this issue is to add a series of conditional statements to my functionality plugin and dequeue CSS and JS myself. In the long term it makes more sense to submit a patch to the plugin developer, or at least try to solve the problem with them, but for now that’s what I’ve done.