It depends. Sometimes I match it to hat color. Sometimes I just use black. And I have a gray fedora where I used a royal purple grosgrain ribbon just for the fun of it!
I like to match it to the trim/embellishments if I have that ribbon color on hand. If the trim/embellishments are the same color as the hat itself, then I try to match it to the hat color.