Can I leave a solo boost pedal on for a whole song?

Not a stupid question at all. Generally you just want your lead to be a bit louder than your rhythm setting. If you like the rhythm sound you get with the tube screamer then you could just slightly roll back the volume knob on the guitar (which lowers volume and cleans up the tone a little) and then when you start to play lead just roll the volume on the guitar all the way up. Leaving the pedal on will not drown out the other folks, you can set the volume at any level you want regardless of whether or not the pedal is on.