Not really.  It's the content of the segments being translated that have the biggest effect on performance, so you could have a TM with less segments that contains very heavily tagged content. This could be less performant.  Another important point is that now we have fragment matching, so the larger the TM the harder it is to "upgrade" it to support fragment matching and some computers may even run out of memory trying to achieve this.

But typically I think once you get to around one million translation units for a file-based TM you are getting close to effective performance.  There is no set rule to this, and I know many translators who work with larger TMs without a problem.

In theory if you use Trados Live you should be able to work with larger TMs quite comfortably, but I have less experience to share on this yet.

It's probably also worth noting that unless you are importing TMs from other sources to grow your TM, or managing a TM for multiple translators, it would take a very long time indeed to get to one million TUs.