I'm going to expand somewhat on my Comment above.
I don't think you're would be really happy with any smoothing option here so I strongly recommend you just sidestep the issue and line the box :-)
This isn't the cop out it might appear to be, small decorative boxes are commonly lined in various ways (baize, flocking, rouched fabric) and it finishes off the interior nicely. Certainly more attractive than a bland wood surface!
Thinking outside the box
The box bottom doesn't have to be a single piece with the sides.
Drill out the current bottom, or cut it out with a coping saw, and throw it away. Smooth off the sides with files (or sandpaper glued to sticks if you own no suitable files) and when you're happy with them you make a new bottom.
Mark through the hole onto the face of a board then cut it to shape and glue it in place. The new bottom can be a matching or contrasting wood as you prefer.
In addition to working around the current problem without having to make any more purchases, and struggling to refine a difficult-to-work surface in a difficult-to-access position, the bottom of the box will become a long-grain surface and not end grain as it currently is. End grain doesn't finish well that easily but long grain does. Additionally long grain is frequently more attractive if you want the bottom of the box to be wood and to be visible.