I would leave it alone. The "J" spring is what is causing it to be hard to remove, but I suggest you leave it alone.
You could get a 1/4" dowel, flatten one side of the tip and put the flatten side down on the barrel and push under and pry up to remove.