I am a beginner myself with hand sawing, but the following tips worked for me for straighter saw cuts:
Mark your cuts either with a pencil or marking knife or both. This allows you to see if you follow the line. The added benefit of using a marking knife is that it severes wood fibres, so there is less tearing or splintering on the exit side.
Do not force the saw, just let it do its job. For Western-style saws, forward pressure is OK, but do not apply (a lot of) downward pressure. Relax your grip on the handle -- it may help if you imagine holding a tiny bird in your hand that you are trying not to crush.
As others have pointed out, your elbow should swing free of your body.
How you start the cut can make a huge difference. For particularly delicate cuts, I cut a straight V-groove with a marking knife or chisel that helps the blade to start right at the planned angle. Again, no forcing the saw and no downward pressure at all. I make a few pull strokes to get the blade catch in the wood (pros I've seen on YouTube start the cut with short back-and-forth movement -- didn't work for me, I guess it depends on the wood).
Another handy trick when starting a cut is to watch the reflection of the wood on the saw blade to see if you are holding it perpendicular to the surface.
Once the cut is started, control the saw by focusing on where the saw should go, as opposed to where it is. I know it sounds vague, but for some reason it works.
Use the full length of the blade. In my admittedly limited experience, itthis results in straighter, cleaner and faster cuts + the wear on the blade is more even. It helps if you imagine the saw is longer than is actually is.
If the saw starts to wander, do not bend the blade. Instead, track back to where the saw started wandering and try again.
Finally, practice a lot: a single scrap piece of wood offers plenty of opportunity. For what it's worth, I stopped beating myself over cuts that wandered off 1-3 degrees and bought a low-angle block plane, built a simple shooting board and now I can straighten a slightly skewed cut.