Primarily, but not exclusively, this is almost certainly going to come down to: your knives aren't sharp enough.
As other previous Answers have touched on, learning to get tools sharp is a journey, and few people (literally almost nobody, from what can be gleaned from first-hand reports) can get really good results immediately1 when sharpening freehand. And fewer still can do it consistently after that
2.
Even if you think your knives are sharp now there's likely a bit of improvement possible. Quite a bit. Like, if you're not actually scared of the edges because they're that sharp, they're definitely not as sharp as they could be :-)
Two other factors are that sideways cuts are cross-grain, and the basic methodology of how you do cuts that way when whittling or carving.
Cutting across versus with the grain
ALL cross-grain cuts are more difficult and tend to naturally leave a rougher surface3. This is because you're severing wood fibres.
By comparison with-the-grain cuts tend to split the wood between fibres, naturally leaving a smooth surface.
Listen to the difference when you cut across the grain and when you cut with the grain; you might be able hear a sort of crunching noise cutting across the grain (that's the fibres being severed), and something much smoother sounding cutting with the grain.
Methodology
From what I've noticed myself and from personal observations of others working (not just in wood), it's fairly common for people to try to take cuts that are too heavy working across the grain when they're new. This is just as true here as when chopping mortises and cutting out the waste in dovetails.
What you need to do instead is slow down a bit for these harder cuts. Well actually more than a bit.
Instead of trying to remove big chunks, think more along the lines of doing paring cuts, slowing working away material until the final cuts which can be (should be?) the finest, in order to leave the best surface quality. Sharpness is key throughout the process so one needs to get accustomed to the need to regularly top up edges; try to learn to do it before an edge needs it rather than wait until it's obvious that the edge is a little blunt and needs attention.
P.S. Species matters
As weird as this might sound, getting good results cutting sideways may actually be slightly easier in harder woods. It sounds paradoxical but here's why: harder woods have stronger fibres, so they stand up and provide more resistance, allowing them to be cut; by comparison soft woods, and especially very soft woods, tend to have limp fibres that fold over instead of allowing themselves to be cut cleanly.
1 Unless they're lucky enough to have first-hand instruction. Good first-hand instruction I should emphasise since there are plenty of bad teachers.
2 The eventual standard being Every. Single. Time. of course. That's a lot to ask of anyone new to anything!
3 Compare for example the simple tooth geometry of rip saws v. crosscut saws for example, and the typically quite smooth result of ripping and the rougher result when crosscutting (although fine teeth and different tooth patterns can greatly affect quality of surface here).
