[Kyle]

At approximately 5 AM PST, during a scheduled maintenance window, our load balancer went down. As I write this, engineers are looking into the issue, but do not have an ETA as to its return.

[Kyle]

And we're back.

[Kyle]

Post mortem from our host:

Originally Posted by SoftLayer

For the DAR/MX upgrade on the 17th an engineering decision was made to implement the VRRP configuration for the Load Balancer farm as an integrated routing and bridging (IRB) interface rather than a routed port, to allow for more flexibility/less downtime for subsequent maintenances to improve the LB switching design. During the maintenance, the new config was applied and connectivity to the Load Balancers were re-established, but an issue was identified where the VIPs were no longer reachable due to a lack of arp resolution between the MX/LBs. After attempting multiple troubleshooting steps to re-establish connectivity, the configuration on the MX was implemented as a routed port to re-establish full connectivity.

Engineering recreated the LB/DAR configuration in the SoftLayer lab, and it was identified that the issue was due to a functionality discrepancy between the hardware in place and the code version on the DAR. The DARs in dal01 are currently running JUNOS 10.4R6, but the trio-based line cards that are in use don't support the IRB/vlan configuration style implemented in 10.4R6. The chipset uses a configuration style that was implemented in JUNOS 9.1, and after testing this implementation style Engineering was able to successfully implement the IRB configuration in the lab.