Bake in a preheated oven for 20–25 minutes, or until a toothpick inserted into the center comes out clean. Remove the cupcakes from the oven and let cool to room temperature.
Add the root beer extract, vanilla extract, milk, and half of the powdered sugar. Slowly mix until combined. Add the remaining powdered sugar and increase to medium-high speed.