Give it a rub down with wire brush to remove all loose shyte, then use an old paint brush to remove all dust.

I would defo use primer. You'll get a better end result, even if it means 2 top coats. Allow primer 24 hours to go bone dry before adding top coat(s).

Good luck.