I am working on an article to be published in a future USSpecialist about that problem.
Throw away any brown albums from the USPS, particularly the early (brown on the inside covers) albums used for the Commemorative panels.USPS switched to other albums with white on the inside covers. These seem to be ok.
My guess is Souvenir panels also had/have their USPS album equivalent but I remember some collectors used the Commemorative panel albums for storage of Souvenir pages.
If your albums came with two brown sheets (for front/back "protection" of any panels, I guess) throw them away immediately.
My earliest unwrapped Commemorative panes (known to have been printed on acidic paper) have deteriorated. Some of my very early panels, still shrink-wrapped, are perfectly ok (although the wrapping says "non archival material"). Those opened and stored in the brown USPS albums have deteriorated as well, particularly those closest to the album cover/brown sheets.
Here is a picture of some page lifters from three albums:

At top: from a white album. About 30 years old
Others: from brown albums About 30-40 years old.
Unwrapped panels inside have the same brown color along the edges:

So the short answer is: keep the plastic wrapping (brown stuff kills your panels over time). But check regularly for any "browning" along the edges of the wrapper plastic/panel paper.
This warning also applies to "safe" hard pvc sheets. I just tossed 20 brown L* albums and over 400 pvc sheets in the trash because some of my booklet panes turned brown along the edges before I finally noticed the degradation.
The moral of the story: Things get ugly after a few decades...