Problem description: after growing the topology of the network, some OSPFv3 neigbhbors were stuck in the Loading phase
Solution: the size of the LSA update didn't fit anymore in the IPv6 MTU of the link and the router (Brocade MLXE Ironware 5.2f and 5.4c) didn't handle the case correctly. Increasing the IPv6 MTU of the link fixed the issue.
More information: The problem with OSPFv3 was that suddenly some adjencies between HP and Brocade routers were stuck in LOADING on the HP side, while the Brocades were reporting the adajcencies as FULL. There was no problem between routers of the same vendor. The problem was probably triggered by the size of the OSPF area which have been slowly increasing by adding routers.
The Brocade TAC has reported: “…we found that the problematic lsa was larger than 1500bytes, so the MLX was not able to transmit the LSA, so LSA with zero LSA was going out. After we increased the mtu on the virtual interface of MLX that was connected to HP, the MLX was successfully able to transmit the LSA, and the HP router could form the ospf adjacency. …”
Brocade is claiming there's no bug, citing this part of the OSPFv3 RFC:
"OSPF does not define a way to fragment its protocol packets, and depends on IPv6 fragmentation when transmitting packets larger than the link MTU. If necessary, the length of OSPF packets can be up to 65,535 bytes. The OSPF packet types that are likely to be large (Database Description, Link State Request, Link State Update, and Link State Acknowledgment packets) can usually be split into multiple protocol packets without loss of functionality. This is recommended; IPv6 fragmentation should be avoided whenever possible."