- 3 May 2010
- 1:03 pm
- [update] USA3-PN outage
The network issue this morning on usa3-pn lasted a few minutes and appears to have only affected some routes.
The network issue this morning on usa3-pn lasted a few minutes and appears to have only affected some routes.
Please ignore the last message re. usa3-pn reboot. I have been informed that the outage is due to a network failure which is being worked on currently.
usa3-pn requires a reboot, which will be performed shortly. The server will be back online within a few minutes.
USA3-PN seems to have ground to a halt – an engineer is on their way to check this and I’ll post any updates here.
UPDATE: Server back on within a few minutes.
The maintenance is now complete and we expect no more problems. Streams have started to come back online automatically.
After the server crash a short time ago, we have arranged for the RAM to be replaced to avoid any future problems. The maintenance is expected to be carried out within 2 hours of this post and will cause another outage of 10-20 minutes.
We have decided to do this asap both because we don’t want to risk any more outages on the server.
All customers on USA3-PN will be entitled to compensation under the SLA because as a result of this outage, the downtime has been over 0.1% for the current month.
The server has been rebooted and is now functioning again. Shoutcast streams should start to come back online automatically, however if you find that your stream fails then you will be able to do a manual restart from the control panel.
Sorry for any inconvenience caused.
updates to follow
We are aware some clients are still having connectivity issues. To rectify any remaining issues we are working on adding a connection to Cogent who is On-Net within our datacenter. They are currently not part of our bandwidth providers.
Abovenet has updated us they are trying to reroute the cut fiber using an alternate path because they are unable to gain access to the fiber for repair due to fire.
We do not have any ETA/ETR estimates at this time. As soon as we have more information we will send another update.
Last night, usa3-pn went down. It was put back up within 30 minutes, however when the server was originally set up, http wasn’t added to autostart when the server boots – this is normally done when a server is set up as it is required for shoutcast streams to start.
As a result, the small number of customers on the server would have experiences a long outage, and we will be offering a partial refund under the service level agreement to all affected customers.
We apologise for any inconvenience caused by the downtime and have fixed the error that caused it to last for longer than necessary.