I can't afford a jointer right now, and I am wondering, what are the tricks to getting a good joint(on the edges, not on the surface)?
What I have tried in the past is simply nailing a straight board to the board I want to cut, and running the straight board along the fence of my table saw. I made sure to hold the straight edge firmly against the fence at the point just before the blade, and having a catcher helped as well.
What I have tried in the past is simply nailing a straight board to the board I want to cut
In essence that's a very good method to do it, and will even work even if the board you're working has very uneven edges, e.g. a live edge (UK: waney edge).
I would recommend you not use nails ideally, you can hold the board securely without having to mark it. Below are a few options.
Table saw
Additionally, you can use the mitre slot instead of the existing fence as your straight reference, this may help to reduce the risk of kickback.
Router
It's a shame you don't have a router as there are many good router-based options for edge jointing and a sharp router bit leaves a particularly good glue surface for a power tool. Here's one simple and effective design:
[Source: ShopNotes #8]
Handtools
This is of course doable just with hand tools, as that is how boards made flat, square and true in the past.
Traditionally a jointer would be used for this as its name would suggest (although they were also called try planes or trying planes). You can do this with a no. 4 or smoothing plane, but at only about 9" / 23cm long it's a little too short to do the job as effectively as one would like — a shorter plane can more easily ride up and down over irregularities rather than skating over the top of them as preferred.
When jointing edges by hand, in addition to the physical planing work it's important to first mark out your boards well, on both faces and both ends, and then work down to your line.
Even with marked/gauged lines to work to planing long board edges flat and square to the face is fairly challenging. So over the years various jigs and fence systems have been devised, especially to help less-experienced plane users do the job.
One particularly good solution is shown below, starting with just a common block plane. By effectively extending its bed length with the infeed and outfeed 'tables' it gives a true jointing effect, even though the plane at the heart of it is a mere 6" / 15cm long.
[Source: ShopNotes #19]
If you planks wiggle on your table saw, use your handheld instead. It is the same idea as you have already presented, but upside down.
In my experience, if you want a glue-able joint, a table saw cannot do it. There is too much 'wiggle' when pushing the wood through. A high end cabinet saw with a large carriage and a very good blade might manage it, but my table saw and certainly any portable shop saw just do not have the ability.
So, not being able to (yet) by a joiner you will need to go the old fashioned route. A hand plane. My recommendation is to get the longest one (in relation to your boards) that you can afford (find in your shop) and use that. The reason for the longer the sole, the easier it will be to make flat over a longer board. a 2' board can be a pretty small plane, an 8' board could use a longer one.
Most reading I've done has suggested planing/joining any cut on a table saw to make a nice surface to glue. The hand plane will need a bit more practice than a joiner but it is significantly cheaper (assuming to don't ruin lots of wood...)
The simplest method is with an edge guide made of 1/2 plywood clamped to the piece or screwed to the piece and a skill saw. Measure the distance between the blade and the outside edge of the skill saw foot and the inside of the blade. Put the edge of the guide that distance back from where you want to cut and have at it. Just let the saw cut don't push it, use a steady hand.
