Author Topic: Arca Noae and 2rosenthals.com web and mail services status  (Read 1496 times)

Lewis Rosenthal

  • Jr. Member
  • **
  • Thank You
  • -Given: 1
  • -Receive: 13
  • Posts: 62
    • View Profile
    • Tales from the Trenches of IT
Arca Noae and 2rosenthals.com web and mail services status
« on: December 13, 2019, 07:17:27 am »
Hi, all...

I just wanted to give everyone a quick update on where we stand after a catastrophic RAID failure on Tuesday night (December 10).

The main Arca Noae website remains up, and the store is accessible, as is all wiki content, and most downloads from the customer portal. New ArcaOS ISOs are not being spun, and subscription driver downloads are not available. Downloads from the Resources | Downloadables pages are not available (ANPM, eCo Runtimes, etc.).

Email remains offline, as does Mantis. This includes the ecs-isp, os2-wireless_users, os2-netware_users, virtualized_ecs_users, and usb-dev mailing lists. In addition, all VOICE mailing lists are offline. All posts sent to those lists since we went offline are queued at the firewall, waiting for the server to come back online, so they should be delivered in time.

As a result of email being offline, the Arca Noae store is unable to send order confirmations and renewal reminders. If anyone has a subscription renewal coming due, be assured that we will extend the due date for you (though you may certainly renew without a reminder email by logging into your account and selecting the subscription order itself).

Web sites other than Arca Noae remain offline, including Rosenthal & Rosenthal, my personal blog, VOICE, PMMail wiki, etc.

FTP is unavailable, as well, as are the Arca Noae YUM repositories and the US mirrors of the Netlabs YUM repositories hosted at 2rosenthals.com.

For those using Subversion, including the PMMail devs, this, too, is offline.

We are working hard to bring things back up again, and hope to have mail and other websites running at least in some rudimentary fashion in the near future. As you can imagine, it's been a rather stressful few days around here. Let's just say that the chain of events which led to this array failure was beyond my wildest expectations, and while I really did plan for disaster, it seems that there's always the one set of circumstances which gets left out of the equation, and this was it.
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS
Managing Member
Arca Noae, LLC                               www.arcanoae.com

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 449
  • -Receive: 91
  • Posts: 2562
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #1 on: December 13, 2019, 01:55:13 pm »
Hi Lewis.

Sometimes this catastrophic events happens. I still have the scar of the 2012 HDD disk crash on OS2World.

Good luck restoring the services, let's hope you get back in service as soon as possible.

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Lewis Rosenthal

  • Jr. Member
  • **
  • Thank You
  • -Given: 1
  • -Receive: 13
  • Posts: 62
    • View Profile
    • Tales from the Trenches of IT
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #2 on: December 13, 2019, 05:39:09 pm »
Thanks, my friend. Work continues. As you know, it's a process.
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS
Managing Member
Arca Noae, LLC                               www.arcanoae.com

Alfredo Fernández Díaz

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 4
  • Posts: 18
    • View Profile
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #3 on: December 13, 2019, 05:52:18 pm »
I hope it will all be over ASAP -- stay strong and keep going!
Thanks for the update,
A.

Sigurd Fastenrath

  • Sr. Member
  • ****
  • Thank You
  • -Given: 16
  • -Receive: 69
  • Posts: 434
    • View Profile
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #4 on: December 13, 2019, 09:42:07 pm »
All the best from me as well! I hopte you can enjoy the christmas holidays even though!

Gregg Young

  • Jr. Member
  • **
  • Thank You
  • -Given: 1
  • -Receive: 6
  • Posts: 55
    • View Profile
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #5 on: December 13, 2019, 11:31:36 pm »
Good luck Lewis. Thanks for the update.

Eugene Tucker

  • Jr. Member
  • **
  • Thank You
  • -Given: 25
  • -Receive: 5
  • Posts: 84
    • View Profile
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #6 on: December 14, 2019, 07:00:36 pm »
Lewis, thank you for keeping us updated. You will get it done.

Mathias

  • Full Member
  • ***
  • Thank You
  • -Given: 25
  • -Receive: 9
  • Posts: 148
  • using ArcaOS
    • View Profile
    • IRC
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #7 on: December 14, 2019, 11:13:02 pm »
Indeed! Thanks for the status update! Was asking myself what s going on, as my ArcaOS test result e-mail came "back". I`ll resend when services are back up again.
And, I hope there are backups of Mantis.. ; )

Good luck and fingers crossed! :D

Mathias


ivan

  • Hero Member
  • *****
  • Thank You
  • -Given: 2
  • -Receive: 36
  • Posts: 1078
    • View Profile
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #8 on: December 15, 2019, 01:43:31 am »
Just wondering if the 'fail over' server also failed.  I had a client with a complete Raid failure but they continued without problems by using the 'fail over' server, it allowed us to replace the failed Raid array from the daily backups and then top up from the running server.

While it can be devastating when something goes down it does help if you are prepared.

Like the others I do wish you well and hope everything be up and running soon.

Lewis Rosenthal

  • Jr. Member
  • **
  • Thank You
  • -Given: 1
  • -Receive: 13
  • Posts: 62
    • View Profile
    • Tales from the Trenches of IT
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #9 on: December 16, 2019, 07:18:53 am »
Thanks for all the good wishes and positive thoughts.

Mail services are back online as of 1:00am EST, though anyone retrieving mail from the server will need to accept the new self-signed cert (yes, I should have added a real cert to the box, but there has been so much other stuff to do).

Recovery of prior data continues.

Web services are next, and we should have ArcaOS ISOs spinning in the next day or so.

As for standby servers, let's just say that that was not an option in this case.
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS
Managing Member
Arca Noae, LLC                               www.arcanoae.com

Lewis Rosenthal

  • Jr. Member
  • **
  • Thank You
  • -Given: 1
  • -Receive: 13
  • Posts: 62
    • View Profile
    • Tales from the Trenches of IT
Re: Arca Noae and 2rosenthals.com web and mail services status
« Reply #10 on: December 17, 2019, 06:59:52 pm »
Public-facing services should be back up and running, with the caveat that much public FTP content still needs to be restored and updated. The Arca Noae bugtracker is back up, and we are able to build new ISOs and sign driver downloads, so subscription content should be available. YUM repositories should be back up as they were last week, as well.

Still more to do, but we're making steady progress, finally.
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS
Managing Member
Arca Noae, LLC                               www.arcanoae.com