Status Notice

UPDATE 2020-02-13 10:00 Mountain Time.

As of this morning, all web sites and email messages have been restored. If you have any problems, please contact support as usual.

Important: If your web site is still down, it means that Lexicom does not manage the DNS records for your domain. You will, therefore, need to arrange for the DNS to be updated to use 51.77.110.153 instead of 198.161.90.112.

Archive of posted updates

We are currently expieriencing a major outage due to a fibre cut in the Vancouver region. With the repair for the fibre scheduled for several days from now, we chose an alternative plan. We are working dilligently to get service restored as soon as possible. However, we do not have an ETA on when full service will be restored. We will restore service as quickly as we can.

UPDATE 2020-02-12 04:40 Mountain Time. Almost all web sites are back online as I write this. If yours still isn’t, try doing a forced refresh (SHIFT + Refresh). If that doesn’t help, then you may have a DNS issue. In that case, you can contact us so we can investigate and fix it or provide you with appropriate instructions.

Saved email messages are still importing as I write this. Some of you have over a quarter million messages in your mailboxes!

With the email restoration underway, there are only a few outlier services that affect relatively few customers that are still offline. We are, of course, now turning our attention to those as well as ongoing work stabilizing our upgraded hosting platform. (Yes, all this chaos has a silver lining: a major software upgrade. Although some of you who have been bitten by old code that isn’t compatible likey don’t think so.)

UPDATE 2020-02-11 17:50 Mountain Time. Webmail is back! Also, most cPanel hosted sites have now been restored. Meanwhile, our system continues churning through the absolutely monunental collection of files our customers have hosted with us to bring all the sites back online.

UPDATE 2020-02-11 03:35 Mountain Time. We have made more progress with restoration. Highlights include:

• Most aspects of the account management area here have been restored and verified to work. There may still be some lingering glitches.
• We are finally in a position to being restoring sites hosted on cPanel and have restored the first few accounts. We are waiting for the account data files to transfer to the new server to complete the rest.

Of course, the lowlights include:

• Webmail has not yet been restored.
• Stored emails are still waiting for bandwidth to be restored.
• Transferring sites and emails is going to be a long plod until the fibre cut is repaired and we can leverage the bandwidth there.

As a side note, we received an update that the repair to the fibre cut that started all of this chaos is delayed by weather. What a shocking surprise. Hence why we undertook this project. Better chaos and an improving situation than a total shutdown, we think.

And now it’s time for #napforwilliam since you don’t want your friendly neighborhood server monkey going stir crazy and undoing everything, do you? 🙂

UPDATE 2020-02-10 15:35 Mountain Time. We are continuing to restore web sites. At this time, the important things to know are:

• Your old email that appears to have vanished is not gone for good. It will be restored eventually after we have restored most of the web sites.
• Domain registration and management should be functional here.
• Email and other aspects of hosting management are currently not operational.
• There has been a marked uptick in spam since email returned. Some may be spam that was queued during the email outage that finally arrived. But it is partly due to one of the technologies we use to combat spam refused to install properly and we just don’t have time to investigate properly during the crisis.
• If your web site is down with a “not found” error, it means we haven’t restored it yet. See the previous update if you have some other behaviour.
• Your friendly neighborhood server monkey who is orchestrating all this recovery work needs a nap. #napforwilliam

UPDATE 2020-02-10 03:30 Mountain Time.Email continues to be operational. We are continuing to rstore web sites. If your web site is still down but is showing a “Not Found” page, then your it will be restored as we continue. If, however, you have a connection timeout, it probably means DNS for your domain is being provided by your registrar or some other third party. In that case, you will have to take action to replace 198.161.90.112 with 51.77.110.153. You will also likely need to make a similar change again not long afterward.

UPDATE 2020-02-09 22:15 Mountain Time. Email service should be operational. Bear in mind that we have not yet had time to import all your stored email. Those will appear over time as the import process continues. Please also be aware that the email management interface and webmail are not yet online and our spam filters still need to be re-established. However, we felt enabling email as soon as possible was the best option.

We are continuing to work on bringing web sites back online. There have been some unexpected software issues which we are working through. Also, be aware that like email, there is a lot of data to move into place.

---

We are proceeding in the following pattern:

• Restore DNS service (mostly done)
• Put a notice on our web site (this page)
• Restore email service but without any saved email messages. We deem this to be better than having no email service at all for much longer. It will mean you can resume collecting messages and sending them. (done)
• Restore web sites and content (underway)
• Reload saved email messages into email accounts (underway)
• Simultaneously with data loading, we will be testing and correcting problems with the system.

Due to the volume of data we need to load for recovery, some sites will be down for longer than others.

Watch this space for further updates.