Even with flies in and stock exhaust the bike with have decel pops. It's just not very audible. Usually, when making modifications like pulling the flies and/or replacing exhaust will make more audible.
As suggested, plugging that line should fix it.