Of course it is a concern. Any rust holes in the load bearing areas will eventually cause a problem. I hope you are planning to return this car to a safe condition. For holes thru a body part (especially frame rails) there is no substitute for cutting out the rust and replacing it with new steel.
However, this being said, it depends upon how bad the rust is and how big those holes are. I know people who drive classic cars of all kinds as daily drivers. Some of them are better than others. I guess as long as parts arn't falling off and the car will pass a safety check then you are probably OK. One of the guys I mentioned drove an unrestored 67
stang until the idler arm rusted off the frame! I expect you will know when your car needs to be parked and fixed.
What I AM suprised about is that the previous owner(s) did all the work you mentioned and didn't repair these holes.
Harry