At 02.52 hours on 31 May 1942, U-506 began to shell the unescorted Fred W. Green (Master Arthur G. Sampson) from her starboard side about 200 miles southeast of Bermuda. The U-boat had spotted the small freighter steaming at 6.5 knots about seven hours earlier and attacked with the deck and AA guns in the moonlit night as they had no torpedoes and only 34 rounds for the deck gun left. The first rounds carried away the mainmast and brought down the aerials, preventing the radio operator from sending distress signals. The gunner on watch opened fire with the 20mm Oerlikon gun aft (the ship was also armed with four machine guns), but after firing a few rounds he was killed by a hit that blew away the gun platform. At 03.06 hours, the U-boat fired the last incendiary round that caused a fire on the fore ship and gradually spread over the whole vessel. On order of the master, the engines were stopped and the two lifeboats were prepared to abandon ship while she was still gunned. However, the U-boat checked fire when the majority of the crew left in the port lifeboat. The master was last seen jumping overboard from the forecastle after checking if all men had left, but was not found when the boat searched for him afterwards. U-506 ceased fire after firing the last shell for the deck gun and then approached the lifeboat to question the occupants, who told them the name of the ship and that she was bound for Capetown and declined when asked if anything could be done for the survivors. The Germans apologized for having to sink the ship, returned to the burning Fred W. Green and used the AA guns to fire holes into the waterline with tracer rounds bouncing off the solid concrete ballast. At 05.42 hours, the ship sank by the bow. The master, three crew members and one gunner were lost and six others were wounded. At daylight, the wounded were placed underneath an awning stretched over the forward part of a lifeboat to protect them from the sun and the boats set sail independently. On 1 June, 31 crew members and one gunner were picked up from a lifeboat by USS Ludlow (DD 438) (LtCdr C.H. Bennett, Jr., USN), escorting convoy AS-3, after they attracted the attention of an aircraft by burning red flares. Shortly thereafter two crew members and two gunners were rescued from a raft by USS Bernadou (DD 153) (LtCdr R.E. Braddy, Jr., USN) escorting the same convoy. On 4 June, USS Ludlow transferred 24 survivors to USS Texas (BB 35) (Capt L.W. Comstock, USN) during refueling and on 17 June both destroyers transferred all remaining survivors to the battleship, which landed them at Bermuda later that day.

The survivors later reported that they had been shelled by two or three U-boats, but this was not the case.