The interior quality in the Passat is that bit higher - to me personally I think it's quite a significant improvement over the Golf.
The thing that amazes me about this generation Passat is that it's a really large practical car on the inside, but so manoeuvrable and with such good visibility, that it feels as easy as driving a Golf - the MQB platform really has worked its magic. Where the 5 series and A4 do feel big and a little bit intimidating when you drive them, the Passat feels really easy. Manoeuvring in tight spaces and parking is a breeze for a car with such a spacious practical interior.
I'd say there are no downsides - go for it!