Peter D,
As well meaning as a lot of the folks on here are, you have to be very careful listening to their advice without backing it up with solid fact.
Physics is presently not a requirement, but until about 2 years ago was a requirement for the QF Cadet Program. For some courses around the world it remains a requirement.
You also have to remember that while flying an aircraft is based on multiplying and dividing by 3, the University course entrance requirements are not, with most people now learning to fly through uni courses that have their own minimum requirements.
I think, for now, that you are safe in dropping physics, but just a warning that as well meaning as a lot of people on these forums are, they are not necessarily all familiar with current requirements and practices
.