As far as I am aware Commercial Road was in Coxside, Plymouth so Arthur may not have been born in Devonport.
Hospitals [and the workhouse] usually had an arrangement with cemetery owners [eg Plymouth council or [private] Ford Park Cemetery] and whilst you say he died in Commercial Road, is it possible that was the address given when he was [possibly] admitted to an institution?

The Coxside area had a big following at Embankment Road Methodist Church and Cattedown Methodist Church and is possible the funeral service was held at a non conformist so you you may need to check those records as well as C of E parish records.

To lose a baby so young and to arrange a burial would have been a heavy financial burden and there was a custom of undertakers /clergymen to see if a grave opened for a burial of an adult from a different family, with that family's permission, to place the tiny coffin within the grave prior to the main coffin being rested.

Secondly it is possible that if the family were in poor circumstances that a public grave [sometimes referred to as a pauper's grave] was used. This with the name of the deceased may have been recorded in the log or minutes of the workhouse / hospital but the cemetery records usually under the entry for the workhouse or hospital would show the date and 'one male child'. The community grave sometimes had a wooden Christian cross but no names were shown on the grave to respect any offence it might assume being associated with a workhouse. The option of it being a community grave is possible but often it would contain numerouse bodies so very hard to identify an individual's burial.