Resolved: Degradation of Off-Campus Internet Service
Incident Report and Final Update (10/1/13, 4:15pm)
During the time frame 9:00am Monday 7 January to 3:00pm Wednesday 9 January, off-campus Internet access was very slow due to a hardware fault in equipment responsible for managing off-campus network traffic flows.
The nature of the fault meant that lengthy and detailed work by both our own network engineers and external service providers was required to identify it. As soon as replacement components were delivered they were tested and installed. Full Internet access was restored by 3:00pm on Wednesday. Our network engineers continue to monitor the situation closely to ensure that the service remains stable.
We appreciate that this incident caused significant disruption to those needing access to off-campus services and also those remotely accessing on-campus services (e.g., Otago Connect). Telephone connectivity from the University’s Auckland Centre was also affected. We offer our apologies for this disruption and our thanks for your patience during the resolution process.
Updates
10/1/13 12:00pm: There is no change to the previous update posted at 9:00am this morning. Constant monitoring is continuing and we will advise of any changes. Next update will be at 4:00pm.
10/1/13, 9:00am: All indications are that on & off Campus Internet access and services have returned to normal operation and are functioning as expected. We will continue to monitor the situation and advise of any changes. Next update 12:00 noon.
9/1/13, 4:00pm: Monitoring of on & off Campus Internet access indicates that services have returned to normal. We will continue to monitor throughout the evening and provide an update tomorrow morning. Next update due Thursday 10th, 9:00am
9/1/13, 3:00pm: The first 2nd level steps towards fault resolution have been completed and initial indications are that traffic flows have returned to normal patterns. Further monitoring is now required to evaluate outcomes of the resolution steps taken to date. Next update 4:00pm
9/1/13, 2:00pm: The replacement unit issue has been rectified. Revised time period for first stages of resolution steps has now been scheduled to take place between 2:30pm – 3:00pm. During this period all on & off Campus Internet Services will be periodically unavailable. Next update due 3:00pm
9/1/13, 1:00pm: Unfortunately final preliminary testing of replacement unit failed which prevented the planned work between 12:30pm – 1:00pm from proceeding. Our Network Partners along with the ITS are working on this latest issue and we will advise of revised implementation time once available. Next update due 2:00pm
9/1/13, 12:00noon: The final part has been delivered and verified by our Engineers. This will allow the scheduled work, along with the associated impact on Internet Services, between 12:30pm – 1:00pm to proceed as planned. Next update due 1:00pm
9/1/13, 11:00am: We are still waiting on courier delivery of the final spare component. ETA from the Courier Company is 11:30am. If the part arrives at 11:30am and is confirmed as correct then the first set of changes involved towards resolution of issue will occur between 12:30pm – 1:00pm. During this period all on & off Campus Internet Services will be periodically unavailable as testing continues. Next update due 12:00noon
9/1/13, 10:00am: Spare components are continuing to arrive. An implementation plan for the replacement of units along with indicative impact/outage times to complete associated tasks is nearing completion and will be finalised, once all the required parts are received and verified. Next update due 11:00am
9/1/13, 9:00am: Spare parts have arrived this morning. ITS is now confirming that the spares are correct including the ‘Firmware versions’. At the same time we are formulating the correct process to replace the suspect components in the Cisco chassis. Next update due at 10:00am
9/1/13, 8:50am: No further information to report - diagnostic work is on-going. Next update at 10:00am.
8/1/13, 4:30pm: Continued diagnostic work has indicated that the likely cause of this issue is the 'supervisor card' within the network hardware that handles our external fire-walling.
Our network support partners are shipping replacement/spare components overnight. This will allow us to replace any components that may fail as part of further troubleshooting and resolution steps that will be taken first thing tomorrow morning. We will provide ongoing updates hourly from 9:00am tomorrow (Wednesday 9/1/12).
8/1/13, 3:55pm: No further information to report - diagnostic work is on-going. Next update at 4:30pm.
8/1/13, 2pm: No further information to report - diagnostic work is on-going. Next update at 3pm.
8/1/13, 12:30pm: Further analysis has eliminated the Firewall module as being the cause of our internet access issue. This points us at other hardware elements (‘supervisor card’ and the Cisco 6500 chassis) and diagnostic work on these is now being undertaken. Once the cause has been identified we can then determine the action required to resolve the issue. We will advise this as soon as possible.
Next update: 2pm
8/1/13, 11am: The slow internet access issue has been traced to a fault with a module in the main network firewall. We are preparing a replacement module and planning for the work to swap it for the faulty one. We will advise the plan and impact as soon as possible.
This information will be updated again at midday - or sooner if the replacement plan is available before then.
Our continued apologies for the disruption – we are aware of the significant impact this is having for many people and are working to resolve it as soon as possible.
8/1/13, 9am: We are continuing to experience slow off-campus internet access. The issue has been escalated to our internet services provider and they are investigating possible causes. We have also engaged our network services partner to assist with resolution.
Time Frame: Monday, 7 January 2013, approx 9.00am - on going
Audience: Staff and Students accessing Off Campus Internet Service
Description: There is currently a significant degradation of Off Campus Internet Service. This is can be either very slow loading of web pages etc with sometimes the page not loading at all.
Services affected: Off Campus Internet Service.
We apologise for any inconvenience caused by this service interruption.
Please contact the ITS Service Desk if you have any questions regarding this outage.
ITS Service Desk Reference: HD0459798