Wednesday, July 14, 2010

Gratia Upgrade status update 20:00 UTC

A little after 15:00 CDT (16:00 EDT, 20:00 UTC), the status of the gratia upgrades was as follows:

1) The legacy redirector service was deactivated at 09:00 CDT (10:00 EDT, 14:00 UTC) as previously announced.

2) OSG-DAILY and OSG-ITB services have been upgraded successfully and are receiving incoming probe data.

3) Based on a very rough estimate of progress (the relative sizes of the upgrading table file and its temporary counterpart on disk), OSG-PROD is expected to come online around 18:30 CDT (19:30 EDT, 23:30 UTC) this evening.

4) At the time OSG-PROD comes back online, we will swap the reporting-DB IP over to the collector DB and the reporting will start to catch up. Meanwhile the upgrades will be replicated to the reporting DB. All service downtimes except OSG-TRANSFER will be considered complete at this time.

5)OSG-TRANSFER is expected to complete by the same token around or before 23:30 CDT (00:30 EDT, 04:30 UTC), at which time it will quietly start receiving data without human intervention.

6)The reporting-DB IP will be switched back to the reporting DB at such time as the replication has caught up, which is likely to be a day or two from now. This will be transparent and will not affect user-visible service.

One particular note: please do not try to kill / restart probes until such time as that service has been marked available in OIM: A hang is expected until that time.