
Walker's education began with a governess at the family home, followed by spells at preparatory schools around the country. The family later moved to Coventry in 1928, when Graham worked as the Sales and Competition Director for Rudge-Whitworth. In 1925 Walker and his family moved to Wolverhampton when Graham became the Competition Manager for Sunbeam.

His mother, Elsie Spratt, was the daughter of Harry Spratt, a businessman from Leighton Buzzard, Bedfordshire. His father Graham Walker was a despatch rider and works motorcyclist for the Norton Motorcycle Company who participated in the Isle of Man TT.

Walker was born at 214 Reddings Lane in Hall Green, Birmingham, England, on 10 October 1923.
