Do I need to bounce the parts of the stems that have no sound in them?

You can if you want, but isn't necessary. However, just make sure to check that the empty stems you bounced do not contain any sound which could interfere with the full track.