There is a huge variety of traditional toy soldiers, in many scales. The metal ones tend to be expensive, however, even if sold unpainted (which they often are not). There are some cost-effective ways to find good toy soldiers (as opposed to "miniatures," their smaller cousins) for wargaming, however, and some of these are explored here.
If wargaming with plastic soldiers is acceptable, one source which should not be ignored is Armies in Plastic. They sell sets of many types, and some of these are well-suited for playing With the Colours in the Late War. You can get WWI-era Germans, British, French, Americans, Serbs, Greeks, Turks, Russians, etc. complete with cavalry sets. They also have ranges for other conflicts (British colonials, Franco-Prussian War, Boxer Rebellion Russo-Turkish War) and some of these have artillery sets as well. Figures are 54mm (1:32) soft plastic, and so will require a good plastic primer before painting. (Gun barrels can be straightened using the steam from a tea kettle to apply heat while straightening, too!)
Another good source for plastics is Classic Toy Soldiers. And, for plastics and metal soldiers both, the Michigan Toy Soldier Company usually has a wide selection.
For those who prefer metal, cost becomes more of a consideration. Pre-painted toy soldiers are available from many manufacturers, but do not come cheap: it is sometimes useful to look on eBay for deals. There are a couple of companies which sell unpainted toy soldiers in appropriate scales, however, aimed at the wargaming market. These tend to be the least-expensive alternatives.
Imperial Miniatures has a line called Little Wars Revisted in 54mm metal, unpainted, covering everything you will need (and some things you won't, but will want anyway, notably a figure of H.G. Wells playing with toy soldiers). Their service is excellent, and they are happy to meet the specific needs of their customers. The soldiers are also available painted.
Another very useful source is Irregular Miniatures. They have ranges in 30mm, 42mm, and 54mm which fit the bill nicely, covering a wide range of periods suitable for With the Colours in the Late War. All figures are sold unpainted. One very notable 42mm line is titled Deutsche Homage, providing figures which indeed match many of the traditional German manufacturer's offerings in this scale.
There is also an American company called London Bridge Collector's Toys which sells old-fashioned Britains recasts in unpainted 54mm, termed replacement parts. These include complete soldiers of various types (scroll down to the "Figures" section), and also many useful parts for restoring damaged Britains soldiers purchased on eBay or elsewhere.
Langley Models is a bit hard to find, but they do traditional 54mm toy soldiers in both unpainted and painted form (along with lots and lots of other things). Soldiers are sold in sets of seven figures, but they are not too expensive as these things go.
For those who are on the creative side, there is a cheaper alternative for metal figures: casting your own. Prince August makes several moulds for traditional toy soldiers in 54mm, which are available in the US from The Dunken Company. A more complete listing of moulds is available here, along with a guide to scultping and casting, which covers the drop-casting process and discusses metals.
For those wishing to game in the smaller 25mm/28mm scale, British in home-service uniforms are available from Ironclad Miniatures (they also have some Prussians and French), and from Redoubt Enterprises (along with Russians, if you need an enemy). In this scale, the choice of continental foes is extensive: a listing can be found here for several 19th-Century conflicts.
Collecting toy soldiers is a fascinating hobby in its own right, and the subject is far too broad to cover here. If you want to see what is out there, try searching for W. Britain (Ceremonial and Archive lines), Tradition of London, Front Line Figures, the Dorset Model Soldier Company, Tommy Atkins, or Imperial Productions (not to be confused with Imperial Miniatures above). Most of these companies only sell painted soldiers, although some (like Tradition and Dorset) also sell unpainted castings. (There is a somewhat-out-of-date listing here, which at least gives a sense of how very many companies are out there.)
You cannot go far wrong with the suppliers linked to above, however, at least to get started (and assuming you are willing to paint your soldiers). If you need to find uniform plates for Edwardian-era British, there are a number of useful ones at Uniformology (Simkin is particularly recommended).