I am by no means an expert, but I do use different sizes together, because I buy what I can afford. If the blocks aren't too far off, it works pretty well. I also have found that, if you block the crown first and let it dry, you can then block the brim with an odd sized block by using a good steam iron and manipulating the brim and block to get the correct curve or angle where you want it. Sometimes it is necessary to work on one side of the brim and let it dry and then work on the other sides. The big thing is make sure you are not stretching the felt or straw crown out of shape.
You can block an old felt over a crown that's a little too small so that it fits smoothly and then, put your foil/gladwrap over this and block your crown proper. This will increase the size by up to 1/2 an inch. I've heard of people using 2 or more layers of felt.
I've used brims smaller than the crown (this is cartwheel brims, not flange brims) by just using a smaller collar - i.e. a collar which matches the crown size. If a bump appears near the headline, it can be pressed out with an iron - fiddly, but it works!