How to feel more comfortable in inline skates?

I used to like skating when I was a teen. I felt comfortable in my skates, and although I hadn't learned much about skating, I remember having fun building up as much speed as possible while skating back and forth on a long back porch/patio. I hadn't learned how to stop back then, and I think I'd gotten bored with only skating in areas where I could slam into a wall to stop.

After around twenty years, I thought I would buy a new set of skates, with the idea of skating around my neighborhood (nothing fancy, just thought I would be more inclined to go jogging if it didn't involve jogging). (Oh, and, of course, I also did an online search for different methods of stopping, haha.)

But I don't feel as stable in skates as I remember being. I can get somewhat comfortable on the glossy, level hardwood floor inside my house, although I always seem to feel just slightly off-balance; but each time I've set skate outside (only in my backyard so far, which has a lot of concrete), I start feeling shaky. The roughness of the concrete makes me feel more off-balance, even if I'm not moving. Plus, one part of the sidewalk goes very slightly downhill, and while on it, I can't seem to control my skates at all, I just keep accelerating until I can grab ahold of something.

At first I thought my legs might be getting jittery because the skates I'd ordered were too big, but when I ordered a smaller size that fits better, I still get that way. Like my legs have to work overtime to keep my balance. Is it just from a lack of confidence? How do you get more confident in skates, or transition from inside to outside? I'd wanted to skate around my neighborhood some, but when I need some kind of railing always within arm's reach, it's not looking possible so far.