The probable site of the motte and bailey castle of the Norman Morville family was used to create a large artillery fort during the English occupation of 1548. This was dismantled shortly after, but the site was reoccupied in 1590 when the Maitland family built the present castle and named it after their ancestral tower on the east side of Lauderdale. The house was enlarged by William Bruce for the 1st Duke of Lauderdale in 1673 and incorporates some superb plaster ceilings by workmen who had previously decorated Holyrood Palace.