Preheat oven to 350 degrees F.
Grease a 9" x 9" baking dish with nonstick cooking spray, or melted butter, and set aside.
In the bowl of a stand mixer, add the pumpkin puree, eggs, melted butter, sugar, and vanilla. Using the paddle attachment, mix until combined. In a separate large bowl, whisk together the flour, baking powder, salt, and pumpkin pie spice. Add to the pumpkin mixture and mix on low speed just until combined.
Transfer about half of the batter to the prepared baking dish and spread evenly using an offset spatula.
To make the cinnamon swirl filling, combine the softened butter, brown sugar, flour, and cinnamon in a medium sized bowl and mix with a spatula until the butter is mostly mixed in—about pea-sized.
Sprinkle the mixture evenly over the top of the batter in the baking dish, then finish with the rest of the batter and spread evenly.
In the same bowl that you used for the cinnamon swirl filling, add the cold cubed butter, sugar, flour, and cinnamon for the streusel. Using a spatula, mix until combined and the butter forms pea-sized pieces. Sprinkle over the top of the batter.
Bake in your preheated oven for about 45 minutes, or until a toothpick inserted into the center comes out clean. Remove the coffee cake from the oven and let it cool completely on a cooling rack.
Meanwhile, make your glaze by mixing together the powdered sugar and milk until there are no lumps. Add up to 1 tablespoon more of milk as needed to reach the desired consistency.
Once the coffee cake has cooled, cut into squares and drizzle with the glaze. Enjoy!