Welcome to KH. Good planning! Yes, go ahead and join to work in the round. You've done well to knit a couple of rows first, it will help to avoid twisting your work. You can use the tail of the yarn to stitch any gap it leaves. Lots of knitters do this whenever they knit in the round. You're smarter than you realized.
ETA Oops! I just realize I missed the word "infinity"...unless you mean a twisted scarf, moebius style, then do join with a twist. If you want the twist, you can have one.