One objection I have heard to an ADS is that it takes away the "right" to drive. Driving is not a right. In every declaration of human rights I have read nowhere do any of them say that driving is a right.
For example in the Canadian Charter of Rights and Freedoms there are two sections that, in my admittedly limited legal understanding, would deal with the actual rights that people have. Section 2 has a portion about Freedom of Association. Section 6 deals with mobility rights. Essentially Canadians have the right to go where we want and associate with who we want, but the Charter says nothing (to my knowledge) about how we are able to exercise those rights.
Another document is the Universal Declaration of Human Rights, to which Canada is signatory. Article 13 states that people have "right to the freedom of movement", but does not state how that freedom can be exercised.
There is also the fact that everywhere you have cars you have a drivers license, which can be revoked if you violate the local laws governing driving.
Does this mean that in a fully implemented ADS nobody would, or could, ever drive manually. Of course not. Some people would have to learn how to drive and maintain a certain level of experience. Firefighters, police, ambulance/paramedic staff, and electrical line workers, to name a few, would need to know how to drive manually as their jobs would take them into areas without ADS infrastructure from time to time.
In a fully implemented ADS the general public should not ever drive manually on public roads. However, there is nothing to stop people from joining a driving club, and driving manually on roads maintained by the club. The club could be just a racetrack, but it could also be a fairly large area, think square miles, with roads winding through it. Regardless of whether it was a track or winding roads, it would be on private land. I see no need for any government to maintain driving areas for the use of manual driving enthusiasts.
So while one may not have the right to drive on public roads, there is no reason one cannot drive as part of a club.
Privacy is an issue, who wants Big Brother able to determine on demand where we have driven, but that is for another post.