Would the Ford Shop Manual have a good diagram of the carb?
Yes, the Ford manual has decent diagrams and descriptions of all the carb internals and how they work.
I have never dealt with Pony Carbs so I have no comment about them.
I agree there isn't any rational way for fuel to get into the secondary diaphragm unless possibly the secondary float level is so high that fuel is getting into the vacuum passage which runs through the carb top plate. Though if that were happening you would have other rich running problems.
To operate, the 4100 secondary has a finger sticking down into one, sometimes both, of the primary venturis as a vacuum pickup. From that finger the vacuum passage runs horizontally from near the vertical air cleaner bolt through the top plate, and down into the diaphragm body at the back of the 4100. There are a couple of holes in the top, main, bowl gasket that have to be open and properly aligned before any vacuum can make it to the rear. I think it is physically possible to put that gasket on upside down which will block at least one of those holes. The secondary diaphragm itself also has a small hole at the top that has to align and that one was iffy on my 4100 as well. Check to make sure none of those possible obstructions to the vacuum are causing your problem. Also, the diaphragm has a finger on the front, hidden side, that has to engage the plastic actuating arm or obviously it can't open anything.
The ball, if you have one, in the rear, angled vacuum passage at the secondary diaphragm is more likely to cause problems when closing the secondaries than when opening them. The ball is to slow the secondary throttle closing rate although it would slow the opening as well. Originally that passage had a small hole and a check ball which was spring loaded against the hole by a small spring and retainer. This caused engine over speeding on quick deceleration so the hole was enlarged and the spring and its retainer were eliminated in mid-1964. Many, I would say most, of the 4100s today don't even have that ball since it works nearly the same with, or without it. You get a similar check ball for the accelerator pump in any rebuild kit. The secondary ball is so (un)important that its not in any rebuild kit I have ever seen.