OK - A couple of months ago I posted under the heading "It looks obvious - but don't jump to conclusions" and I got it wrong. It seems possible that young James Holliday (1787-1879) stated to work for James Smith (1768-1843) shortly after James Smith moved to Ashlyn's Hall, Berkhamsted, in 1801. By 1841 James Holliday was James Smith's farm bailiff. Two of James Holliday's sons worked for the Smith family - at some time working in the Scilly Isles, which was "ruled" by Augustus Smith, of Ashlyn's Hall and Tresco. My initial mistake was a failure to find Charles Holliday in the 1841 census as he was listed as a domestic servant in London - but was probably there because he was a personal man-servant to James Smith, who was also staying at the same address.
More Information on the Holliday Familya