A suggestion, building on what was said. Install the wiring etc out of site as much as poss. Mount the charger on a back board screwed into the mortar not the brick. When you leave, take the board off and repair the holes - no indication of what was there. If you're able to feed the cable through a mortar joint, there might not be any evidence anything was there at all in which case you can take everything away. If you can't put the cable through a mortar joint, then when you leave, do as chris suggested and put a cheap socket there.
To hide any signs of what is there, enclose it in a meter box (or a home made box), again, fitted to a backboard screwed into the mortar. Then when you leave, take the box away and no-one wil ever know what used to be there. They'll only see what you left
I'm aware the property may be inspected at regular intervals and it may be noticed that there was a charger fitted, and the tight fisted landlord may see the opportunity to improve his property by demanding your charger is left if he sees it. Putting it in a box stops it from being seen and if the box has a proper lock, he won't be able to look inside.
If asked about the box you can always say that you store a granny lead in the box rather than taking it inside with you. In use, all anyone will see is a lead with the car plug on the end. The charger will not be seen.
If the existence of a charger is ever discovered, remember, chargers don't last forever, and it would be typical bad luck if yours failed a month or so before you left forcing you to replace it with a 13A soocket so you can continue to charge the car with a granny lead ... of course, you'd leave the socket because you know you have to leave permanent installations.
If worse comes to worse, I'm not sure he will be able to demand the car charger is left, as long as you don't cause any detriment to the property. Repairing any installation in an invisible way would satisfy a court about that, but it's best to avoid that kind of escalation.
If you don't feel you can install and remove the charger without leaving some evidence, or if you feel teh existence of a charger will be found out, then get permission to install a 13A socket in a box to stop others stealing your electricity. Then, once it's been inspected, upgrade it to a charger mounted on a backboard. When you leave and remove the charger, you're leaving him with what he expected to have. If you replace the 40A cable to the charger with 13A, and replace the MCB, he won't know any different, and if everything looks neat and undamaged, the person doing the routine property inspection won't have a clue, they're simply not qualified.
Property management companies are the worst for this because they demonstrate to the owner how good they are by clawing back as much as they can from the tenant regardless of how fair it is. One company tried to charge me for the cost of mowing the lawn. Even when I pointed out that they inspected the garden one week after I left, and in a warm damp summer, grass actaully grows and need cutting. They still wouldn't relent, so I told them I'd start proceedings to get my deposit back and prime exhibit would be a photo of a neatly mowed lawn with my neighbour's son finishing it off, on the day of departure. I suggested they spoke to the owner to check he was happy to be sued about the matter... the deposit was handed back, without an apology.