Some prisoners were held on the Retribution hulk at Woolwich prior to transportation.
Retribution Hulk - National Maritime Museum
Others were sent to the Portland Hulk in Langstone Harbour. The Lancaster Gazette reported on 2 January 1813......The following convicts under sentence of transportation were last week removed from Lancaster Castle, and put on board the Portland Hulk at Langstone Harbour viz - John Gleming William Houghton, John Milburne, William Parr, William Rigby, James Nuttall, William Mevee, Henry Waters and Thomas Kitnan. 
Hulks in Langston Harbour
James Neils published a report of the Portland Hulk in 1807; probably little changed in the following six years......
In consequence of La Fortunee having been broken up as unserviceable, the Portland, Captain Blackman, has been put into condition as a Receiving-ship. Divine Service is performed here on every other Sunday between-decks, there being no better accommodation as a Chapel. The robust and healthy go on shore to work; and those who are employed as carpenters, bricklayers, etc in Cumberland Fort, have the extra allowance of a biscuit, and tobacco. Their provisions are dressed on board the Portland, and sent in a cart to the mess-house on shore, about a quarter of a mile distant. My visit happening to be at the dinner-hour, twelve o'clock, gave me an opportunity of examining the beef when dressed, and of which the Convicts complained sadly: and, indeed, not without cause; for it was very lean, full of bones, and of the coarsest quality. A few had vegetables, which they purchased for themselves. There is nearly an acre of ground enclosed for the growth of vegetables; but so very poor is
the soil, that its produce will not afford a sufficient supply for one day in the week throughout the year. The bread, beer, barley, and oatmeal, appeared to be good: the cheese, however, was so execrably the contrary, that I saw a large heap of it put by, to be returned. The present number of Convicts belonging to the Portland, is 310. Of these 142 slept on the lower deck, 122 on the middle one; and 32 on the upper deck: fourteen were in the Hospital-room, of whom eleven were disabled by sore legs. Within the last nine months, twenty had been punished by additional irons, and sixteen flogged.
The Caledonia Mercury mentioned in June 1813 that among the convicts transported on the Earl Spencer were Capt. Davison, Mr. Lindsay Crawford, several bankers' clerks, the men called Luddites and the smugglers of Christchurch who were convicted of aiding French prisoners to escape to France.
The Smugglers of Christchurch included John Childs alias John Giles, Francis Long and Jacob Martin.
The Earl Spencer departed Portsmouth on 2 June 1813 and sailed via Madeira.
They arrived in Port Jackson on 9 October 1813.
Governor Lachlan Macquarie recorded the arrival in his journal....... On Saturday 9th October - This forenoon The Earl Spencer Convict Transport Ship commanded by Capt. Wm. Mitchell anchored in the Harbour from England, from whence she sailed on the 2d. of June last, touching only at Madeira which she left 2d. of July. - She brings 196 male Convicts - wt. a Guard of one Subn. (Ensign Bicknell)and 38 men of the 73d., together wt. several Free Settlers, and one asst. Surgeon (Mr. Young) for the Colony. I have received Public Dispatches by the Earl Spencer, and also a great many Private Letters from my Friends at home, all of which contain good news. By this conveyance I am informed, tho' not officially, of the certainty of my being appointed a Major General in a large Brevet made by the Prince Regent on the 4th. of June last.
The free settlers mentioned in Governor Macquarie's journal included
Mr. and Mrs. Younge,
Mr. John Dixon a civil architect who intended to erect water mills;
Thomas Barker and Richard Hobden also arrived free (Colonial Secretary's Index)
Later Governor Macquarie referred to Mr. Younge in correspondence to Earl Bathurst........ I am sorry to be under the disagreeable necessity of reporting to Your Lordship that Mr. Henry St. John Younge, the last Medical Officer sent out to this Country, and who arrived by the Earl Spencer transport, in October last, is by no means a desirable acquisition in the line of his profession. He is exceedingly ignorant as a medical man, being almost destitute of common understanding, and very low and vulgar in his manners. From this description Your Lordship will perceive that he is a very unsuitable person to fill a situation of such responsibility as that which has been assigned to him. I feel the more necessity for making this communication to Your Lordship in regard to Mr. Younge's qualification, in the hope that Your Lordship may direct a more minute investigation of Medical Candidates for this colony to take place, and that they shall be certified as duly qualified by the Army Medical Board previous to being appointed. 
He was more pleased with John Dickson - Agreeably to Your Lordship's Commands, I have located a liberal portion of land, namely three thousand acres to Mr. John Dickson, free settler, and civil architect, who arrived in October last by the ship Earl Spencer. I have also assigned him the services of ten government men on the stores for the term of eighteen months and I have also given him a most convenient and eligible situation in the town of Sydney, having a run of fresh water thro it, for him to erect his mills, steam engine and various machinery on. Having brought a considerable capital with him, and being of an enterprising spirit and persevering industry, I look upon Mr. Dickson as a very great acquisition to the colony. 
Those who died on the passage from England were John Robson, a youth, belonging to the ship; and John Hogg, William McLeod, Edward Whitford and Jarvis Copely, prisoners.
The prisoners were landed on Thursday 14th October and mustered in the presence of Governor Macquarie prior to their distribution to various employments. They were reported to be a healthy set of men; and appeared thoroughly sensible of the kind treatment they experienced from Captain Mitchell and his Officers during the passage. (Sydney Gazette) Find out more about the disembarking of prisoners.
In November the Sydney Gazette reported on the drowning in Sydney Harbour of a steward of the Earl Spencer ......
On 21st November at about six o'clock in the evening the jolly boat of the Earl Spencer with the boatswain, captain's steward and a boy on board unfortunately filled when opposite Bennelong Point and sunk. The steward and boy immediately grasped at the boatswain who they knew to be an expert swimmer and maintained their eager hold with all the force that terror could inspire, until they had several times gone down together, and were on the very verge of one common destiny, when the steward, exhausted relinquished his hold and sunk to rise no more. The boatswain had still sufficient power to raise the younger companion of his peril to the surface, and animate him to personal exertion as he could swim a little and just sufficiently to gain the shore with his assistance. The body of the young man whom the will of Providence had thus selected as an evidence of the extreme uncertainty of human life, was found yesterday, and conveyed to the hospital wharf, where an inquest found he died an accidental death. The deceased William Sampson. was a native of London aged 21 years
Life in the Colony
When the Earl Spencer arrived in Sydney the colony was in the grip of a devastating drought. Crops had failed and livestock was depleted.
Governor Macquarie issued a Government Order for an increase in the price of wheat for the season in consideration of the middling and lower class of settler who must be considerable sufferers by the great deficiency in their crops and he holds out this inducement to their going on with unremitting industry in cultivation and improvement of their farms and to alleviate their present distress.
Under pressure to mount an expedition to discover a way across the Blue Mountains to what was believed would be fertile land beyond, Governor Macquarie commissioned Blaxland, Wentworth and Lawson to attempt to find a passage. This they achieved in May and by November surveyor George William Evans had already been dispatched to follow in their tracks.
Earl Spencer Re-Fitted
Earl Bathurst had intended that the Earl Spencer, General Hewitt and Windham would convey troops to Ceylon. He informed Governor Macquarie that the ships had been chartered at a great expense and they were to be detained at Port Jackson as short a time as possible. He expected that the 73rd regiment would be ready to embark on the arrival of the Earl Spencer. By early in November Government Carpenters were employed in constructing the necessary berths, partitions and hammocks, and the decks intended for the troops were properly cleared and cleaned before the Government white washers proceeded on board to white wash the decks. Separate accommodation was provided for married couples and single men slept in hammocks.
Three hundred and thirty-four soldiers as well as seven Officers of the 73rd regiment were supposed to depart on her accompanied by 36 women and 70 children. The Officers were Major Alexander Gordon, Captain Haddon Smith, Captain John Pike, Lieutenant Thomas Atkins, Lieutenant Martin Murphy, Lieutenant D'arcy Wentworth, Ensign Anthony Pooke and assistant surgeon Martin however the Earl Spencer and General Hewitt were found inadequate for taking the troops to Ceylon, being too crowded, and a number of people were removed including 152 men, thirty women, sixty three children and six officers who were all to remain in Sydney until more suitable vessels could be procured. 
Departure from the Colony
The Earl Spencer departed the colony on 25th January 1814
1). Richard Stockwell arrived as a prisoner on the Earl Spencer. In 1814 he was given special permission to join Rev. Samuel Marsden on a missionary expedition to New Zealand.
2). On arrival Jacob Martin, one of the smugglers of Christchurch, was sent with thirty-four other men of the Earl Spencer up the river to Parramatta to work in the service of Rev. Marsden. In June 1824 he was sentenced to six months in the gaol gang on half rations for absconding from employment at the Barracks being sent up from Sydney. He died in 1860.
3). Convict John Lindsay Crawford was sentenced to 14 years transportation for committing forgery in an attempt to make claim to an Estate. He later returned to England. After his death Crawford's heirs continued to make claims to the estate.
4). Other Convicts transported on the Earl Spencer included artist Richard Read, surgeon Henry Ravenscroft; former Lieutenant in the Dragoons and Newcastle's first school master Henry Wrensford; and Jeremiah Butler who accompanied John Howe's 1820 expedition.
5). The Luddites.....May 25, 1812. The Special Commission for the trials of the rioters at Chester was opened by Judges Dallas and Burton. Their Lordships attended Divine Service the following day and afterwards proceeded to the Shire Hall, when the Grand Inquest of the County being sworn, Judge Dallas addressed the Grand Jury in an eloquent and impressive speech..... On the general feature of the cases for your consideration, I have no reason to form any very pleasing conjectures. Distress, I fear, did not operate as some would wish us to believe. The characteristics of the system of disturbance are of a very different colour. Handbills were printed and circulated in the manufacturing districts, holding out hope to the disaffected, and threats to the well disposed. A secret oath was administered - the crest and consolidation of conspiracy - all these denote the intrigues of wicked and designing men, to create riot and partial injury. Justice has been
for a while withheld, but the law must not unfold its terrors, always remembering, as we shall, that confidence and consideration so necessary in its execution.
The trials commenced on the 27th; when James Renshaw, a weaver from Winslow, was indicted for a riot, with several others, at Etchells, near Stockport, on the 15th of April, and obtaining from John Norris 2s. and from other persons money, bacon, bread, and cheese. Verdict, not guilty. The following persons were found Guilty: Edward Renfern and Nancy Hurst, for accompanying a large mob to the granary at Staley, belonging to the Huddersfield Canal Company, and for breaking open and entering the same, and stealing and destroying upwards of 1000 bushels of flour and meal. Thomas Burgess, a collier, from Bredbury, for being, in company with several others, assembled at and entering the corn-mill of Joseph Clay, and burning and carrying away flour. Samuel Lees, Thomas Etchells, and James Ratcliffe, for rioting at the same place. William Greenhough, for a riot at Tintwistle; John Temple, for a burglary, and robbery at Addington. Foster Roach, Collin Linden, James Wilson, James Bennett, Richard Wood, William Thompson, and James Tomlinson, for unlawfully assembling together in Etchells, and by numbers and intimidation, obtaining from J. Parker, Esq. 7s. J. Torkington and J. Henshall add R. Lowns, for rioting mid robbing in Pownall Fee and Styall. William Walker, dignified with the name of General Ludd, for a riot, and seizing flour from Ralph Booth at Gee Cross. James Crossland, for destroying machinery, and threatening the life of Robert Thomiley, a manufacturer of cotton. [Wm. Greenhough and Abraham Broadbent, tried on the same indictment, acquitted.] John Haywood, the younger, for a riot at Tintwistle. The Special Commission for the county of Chester remains in force till August. -
6). Thomas Whitaker was convicted on the evidence of one Parnell, of administering an unlawful oath to 18 or 20, in which they entered into a solemn obligation to destroy steam-looms, etc. Whitaker was sentenced to seven years transportation - Annual Register
7). Thomas Etchells (alias Brunt) former hatter and rioter at Joseph Clay's corn mill near Chester by 1828 was married, aged 50 and running his own hatter's business in George St. Sydney; working for him another Luddite Samuel Lees, presumably an old acquaintance as he had also been a hatter at home and had become involved in the same riot as Etchells. ... Rude, George F.E. Protest and Punishment: The Story of the Social and Political Protesters Transported to Australia, 1788-1868
9). One of the prisoners on the Earl Spencer was Thomas Collicott.... The Australian Dictionary of Biography includes an entry for John Thomas Collicott who was..... the only son of Thomas Collicott, who was transported for failing to affix duty stamps to bottles of medicine. In January 1816 John Thomas Collicott, with his stepmother, formerly the wife of Richard Allen, a physician to the Prince Regent, and members of his and her families, arrived in Sydney in the Mary Ann. They had a letter of introduction from influential friends to Governor Lachlan Macquarie, and John Thomas Collicott was granted 200 acres (81 ha) of land at the Five Islands.  Mary Collicott became Matron of the Female Orphan Institute.
11). Captain John Davison of the Marines, age 32. Born in Durham and convicted in Oxford of having forged notes. Admitted to the Retribution hulk 6th January and transferred to the Earl Spencer on 5 April 1813. 
 New South Wales Government. Main series of letters received, 1788-1825. Series 897, Reels 6041-6064, 6071-6072. Item 4/3491, p. 618. State Records Authority of New South Wales. Kingswood, New South Wales, Australia.