Fulk the 5th was the next Count of Anjou, and later the King of Jerusalem in 1131. On August 24, 1113, he and his wife Ermengarde du Maine gave birth to their son Geoffrey Plantagenet.
Like the Fulks before him, Geoffrey Plantagenet did not have much of an effect on the castle. However, he did give the castle a Romanesque doorway. In 1128 Geoffrey married the widow of Henry the fifth, Queen Matilda of England, who in 1134 gave birth to Henry Plantagenet. In 1132 the main building of the castle, the Great Hall also known as Aula, was burned down. It was restored later on in the 1130's and continued to be a place of business for the rulers of Anjou. .
Henry Plantagenet, who was by now the ruler of Anjou, married Eleanor of Acquitain in 1152, gaining Acquitain only 2 years before becoming the king of England. Since the Count of Anjou was now the king of England, Anjou became an important location. There was now much more money in Anjou, so the castle was continually expanded during Henry's reign. He also continued to use Aula as part of his headquarters, making the castle a much more significant meeting place.
After the Anglo-French wars from 1180-1206, the kings of France took control of Anjou, as well as many significant locations in England. Philip Augustus was the first French ruler of Anjou, but he made no changes to the castle. In 1228, Anjou was ruled by Count Peter of Dreux, who once again made Angers a significant location as the western center of royal authority. After the young death of Louis the 8th, Blanche of Castille became the regent for her son Louis the 9th. Blanche of Castille decided to build a much larger fortress in Anjou, so along with Louis the 9th the fortress was built from 1230-1240 taking up ¼ of the city. The new fortress had very defensive purposes, with only one undefended side on the north. Its towers were closer together than most of the castles of the time, with 17 towers each only 10 meters apart.