Hi Team - Over the past few days, IT has not observed packet drops on the San Jose network. While a root cause was not identified/communicated to IT by AT&T, the lack of drops means this incident will now be closed. IT will continue to monitor the network moving forward in case these issues resurface, but for now, the issues appear to be resolved. Thank you to everyone for your flexibility and patience as we have navigated these network challenges.
Posted Nov 18, 2022 - 10:19 PST
Monitoring
Good Afternoon Team - AT&T made some configuration changes on their end last night and we have not seen any drops or instability with the network today. IT is cautiously optimistic that the issues we have been seeing are resolved but will be keeping this incident open and monitoring the network intensively through the remainder of the week. Please continue to keep IT in the loop if you experience any network drops or issues. Thank you!
Posted Nov 16, 2022 - 15:22 PST
Investigating
Troubleshooting of the network has concluded for today. The plan remains the same for tomorrow.
If you are in San Jose and can come onsite tomorrow, please do so. We need to test the network with more users on, ideally to cause some further packet drops that can be used in troubleshooting. IT will be communicating as we monitor the network and see drops occur and if/when users should move to working from their hotspot. We thank you for your patience and flexibility as we continue to troubleshoot this issue.
Posted Nov 15, 2022 - 15:17 PST
Update
IT is troubleshooting the network with AT&T currently. Please expect some network downtime.
Posted Nov 15, 2022 - 14:04 PST
Update
IT has been troubleshooting the network with AT&T and we need to replicate the networking issues that were seen yesterday. Unfortunately, we have not been able to replicate that to perform more in-depth, live, troubleshooting with the AT&T specialists. With that in mind, we need all San Jose users to please come back on-site tomorrow. The intention of this is to add more users to the network and see if the network/packet drops pick up again so we can give AT&T the opportunity to dive in while the problem is occurring.
IT is still monitoring the network closely through today and tomorrow. If you see any issues on the network, slow connections, or drops in connectivity, please let IT know immediately. It is crucial that IT be informed right away so the troubleshooting can, ideally, identify the root cause of this issue.
Posted Nov 15, 2022 - 09:30 PST
Update
IT is currently working with AT&T and will be performing brief maintenance on the San Jose network. The network will drop momentarily but return to normal operation shortly after.
Posted Nov 15, 2022 - 08:04 PST
Update
IT has worked with AT&T specialists and the current thought shared by both teams is that the issue rests "upstream" or outside Westpak's infrastructure. Currently, AT&T is engaging the local telecom in San Jose to investigate the hardware/cables that connect Westpak San Jose to the regional AT&T node that provides internet access. Unfortunately, this infrastructure is not controlled by Westpak and we will need to wait for the local telecom and AT&T to troubleshoot and repair any damage.
For the time being, users in San Jose that have the ability to work remotely, should plan to work from home for the next 48 hours. We should have an update before then.
Users that need to be onsite, continue to do so and work on the Westpak network. If the network drops or shows signs of instability, please switch to your hotspot. If you see any network issues or strange behavior, please reach out to IT to investigate and relay up to AT&T if needed. IT will be monitoring the network and communicating with AT&T regularly throughout the next couple days. Any applicable updates will be relayed to the company through these alerts. IT appreciates everyone's patience and adaptability as we navigate these issues.
Posted Nov 14, 2022 - 15:59 PST
Update
Hi Team - IT is connected with AT&T's specialist team to troubleshoot the network. Can everyone please get back on the westpak network if you are still onsite? We want to put load back on the network to see if we can replicate the issue. We do not anticipate the issue to be resolved, this is a step to help us drill down to the exact cause of the recurring issues. Please reconnect your devices to the network and resume working.
Posted Nov 14, 2022 - 14:52 PST
Identified
IT has identified the issue is with the AT&T equipment once again. IT is reaching out to AT&T support to get this resolved. If you have the ability to work remotely for your job role, please transition to remote work for today. If that is not an option for you, please transition to your phone's hotspot for internet access and use the VPN on your system to connect to Westpak shared drives. If you have access issues or need the VPN configured on your system, please reach out to IT for assistance.
Posted Nov 14, 2022 - 10:06 PST
Investigating
Good Morning Team - IT is aware of the network drops currently being seen in San Jose. IT is investigating the cause and will provide updates as more information becomes available. While the network is still up, performance degradation and slow speeds are expected.
Posted Nov 14, 2022 - 09:37 PST
This incident affected: Site Locations (San Jose Site).