Whites Castle is now a private residence. It was built in 1417 by Sir John Talbot, Viceroy of Ireland, to protect the bridge over the river Barrow and the inhabitants of the Pale. Built into the wall on either side of the original entrance doorway are two sculptured slabs.

On the right of the former doorway is the Earl of Kildare's coat of arms, signifying the Earl's ownership of the castle in former days. The slab on the left bears the date 1573, and the name Richard Cossen, Sovereign of Athy. It is believed to have been taken from a mill which once stood on the site of the present Castle Inn.

Nearby is Crom-a-boo bridge, from which you can look across the river Barrow to Woodstock Castle. "Crom-a-boo" was the war cry of the Geraldine family, and the inscribed stone in the bridge shows that it was built in 1796 by "Sir James Delahunty, Knight of the Trowel"