Ernest's father, Henry, and mother, Henrietta Letitia Sophia Gavan, were of English-Irish ancestry. Ernest was the second of their ten children and the first of two sons. In childhood he was described as "bright, good-natured,... and confident".

In 1880, at six years old, Ernest moved to Dublin with his father, who was studying medicine. Four years later, the family moved from Ireland to Sydenham in suburban London to seek a better income and because their Anglo-Irish ancestry made them afraid to stay in Ireland after the assassination of Lord Frederick Cavendish by Irish nationalists.

Shackleton spent the remainder of his childhood in Sydenham.