At 20.02 hours local time on 2 Nov 1944 the unescorted Fort Lee (Master Ottar Marius Andersen) was hit by one torpedo from U-181, while she proceeded on a nonevasive course at 15.5 knots. The torpedo warning device of the tanker did not sound until after the torpedo hit. The torpedo struck on the port side in the fireroom about 12 feet below the waterline. The explosion likely caused the port boiler to explode, stopping the engines and killing two men on watch below. A distress signal could not be sent because the radio antenna was broken. As the ship settled by the stern, the boiler room and the engine room were flooded and the crew´s mess filled with smoke and steam. When the ship lost headway, the master gave order to prepare the boats for lowering. These orders were misunderstood by some men and three boats were lowered.

At 20.18 hours, a second torpedo struck on the starboard side between the engine room and #9 tank. The explosion sent a ball of flame 200 feet in the air and shattered two of the lifeboats, killing six men and throwing the remaining eight men into the water. The tanker caught fire, but a wave washed over the well deck and put the fire out. The most men of the complement of ten officers, 39 men and 26 armed guards (the ship was armed with one 5in, one 3in and eight 20mm guns) abandoned ship in four lifeboats. U-181 surfaced and questioned the men in one of the boats before leaving. The tanker sank stern first at about 21.10 hours. One officer, six crewmen and three armed guards were lost in the sinking. The men in the water were picked up by the boats and provisions were taken from four rafts that floated free of the wreck. The four lifeboats sailed in company for three days and then became separated during strong winds that lasted for 48 hours. On 7 November, the British motor merchant Ernebank picked up the master and 15 survivors in one boat and landed them in Fremantle on 14 November. On 9 November, the American steam tanker Tumacacori rescued 17 men a second boat and landed them at Albany, Australia after five days. On 16 November, the American steam merchant Mary Ball fired 12 shells on a boat with 17 survivors before realizing it was not a submarine. The men were picked up and landed eight days later at Colombo, Ceylon (Sri Lanka).

The fourth boat with two officers, eight crewmen and six armed guards was never seen again. More recent research indicates that the boat made landfall in Japanese held territory at Soemba Island, Indonesia on 13 Jan 1945. Amazingly, three men were still alive after traveling over 2800 miles and taken prisoner, but one of them died shortly after reaching the shore and the Japanese reported that the other two later died in hospital, but it is believed that they were executed in April/May or September 1945.