Hej alla.
Senaste uppdateringen från NORDUnet om Zoom Cloud Recording
Med vänliga hälsningar,
Peter Häggstrand
SUNET
Från: Jørgen Qvist <qvist(a)nordu.net>
Datum: fredag 22 november 2024 09:03
Ämne: [Mediagroup] Zoom on-premise recording issue update 20241122
OBS: Detta mail kommer från en avsändare utanför din organisation.
All,
The final problem, ref. last weeks update, has been resolved and everything is back to normal operation, with 0 recordings being lost this week.
It has also been possible to recover most of the old missing recordings and these have been processed as well.
For now we consider the issue to be resolved. A permanent solution will be deployed later.
Unless something changes this is the last update in the series.
Brgds,
Jørgen Qvist
email: qvist(a)nordu.net
mobile: +45-31621400
Please read NORDUnets Data Protection Policy - https://nordu.net/privacy-policy/ & Legal Notice - https://nordu.net/legal-notice/
On 16 Nov 2024, at 20.12, Jørgen Qvist <qvist(a)nordu.net> wrote:
All,
Sorry about the delayed update.
Things are continuing to improve. The changes made last week has uncovered an NFS permission problem on the recording servers that Zoom has been able to reproduce. A fix for this will be deployed over the weekend.
Again, we will monitor this closely,,,
Next update Friday November 22’nd.
Brgds,
Jørgen Qvist
email: qvist(a)nordu.net
mobile: +45-31621400
Please read NORDUnets Data Protection Policy - https://nordu.net/privacy-policy/ & Legal Notice - https://nordu.net/legal-notice/
Hej alla!
Se nedan.
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: mån 18 nov. 2024 kl 09:53
Subject: Status Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Incident Update NORDUnet Status
*Current Status:* Degraded Performance
*Started:* September 2, 2024 12:22 UTC
*Resolved:*
*Affected Infrastructure*
*Components:* Zoom
*Locations:* Recording
*Update:*
Things are continuing to improve. The changes made last week has uncovered
an NFS permission problem on the recording servers that Zoom has been able
to reproduce. A fix for this was deployed over the weekend.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
Hej alla!
Ny uppdatering gällande problemen med cloud-recordings.
All,
Sorry about the delayed update.
Things are continuing to improve. The changes made last week have uncovered
an NFS permission problem on the recording servers that Zoom has been able
to reproduce. A fix for this will be deployed over the weekend.
Again, we will monitor this closely,,,
Next update Friday November 22’nd.
Brgds,
Jørgen Qvist
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej alla!
Se nedan.
Mvh
Monika
Skickat från min iPhone
Vidarebefordrat
> Ämne: [Mediagroup] Re: Zoom on-premise recording issue update 20241101
>
> All,
>
> Unfortunately the change to the storage hardware and logical infrastructure did not resolve the problem.
>
> We did however uncover a problem where the recorders are started before the NFS mounts are fully setup. We have tried to implement a fix for this and we are currently seeing an improvement of the situation.
>
> We will monitor this closely.
>
> Next update Friday November 15, late as I will be travelling back from the US.
>
> Brgds,
>
> Jørgen Qvist
>
>
> Please read NORDUnets Data Protection Policy - https://nordu.net/privacy-policy/ & Legal Notice - https://nordu.net/legal-notice/
>
>
> _______________________________________________
> Mediagroup mailing list -- mediagroup(a)lists.nordu.net
> To unsubscribe send an email to mediagroup-leave(a)lists.nordu.net
Hej!
Det här kom i fredags:
All,
Unfortunately the new version of the Recording Connector did not resolve
the problem.
Zoom is continuing to work on a solution at their end.
NORDUnet has in a maintenance window Thursday October 31, made changes to
the Recording Nodes storage hardware and logical infrastructure, for two of
the operational Zoom Zones. We will monitor the impact before making
further changes.
Next update Friday November 8.
Brgds,
Jørgen Qvist
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
All,
enclosed is the Zoom Root Cause Analysis (RCA) of the issue encounter
between 17 and 21 UTC on 20241022.
The effect of the issue was that users on a subset of sub-accounts could
not start/join meetings and instead received 3010 errors.
The issue was triggered when NORDUnet was doing planned maintenance on a
Zoom Zone, removing that Zone from all sub-accounts.
Zoom had recently deployed a new security feature that had a flaw which
then returned a completely empty Zone list to these sub-accounts, that then
had no active Zoom Zone to connect to.
The workaround was to disable the security feature which restored
operations.
A fix has been deployed that permanently has resolved the problem.
Full Zoom RCA enclosed.
NORDUnet regrets the problems encountered due to this.
Brgds,
Jørgen Qvist
Please read NORDUnets Data Protection Policy -
https://nordu.net/privacy-policy/ & Legal Notice -
https://nordu.net/legal-notice/
_______________________________________________
Mediagroup mailing list -- mediagroup(a)lists.nordu.net
To unsubscribe send an email to mediagroup-leave(a)lists.nordu.net
Hej alla!
Se mejl från NORDUnet nedan. Om det efter helgen dyker upp problem så låt
oss veta det via media(a)sunet.se
All,
Finally good news.
A new version of the Recording Connector has been made available and will
be deployed during the weekend. The new version includes the fix for the
recording problem that we have been struggling with.
We are however reluctant to say that the problem has been fully fixed by
Monday.
During the next weeks we will monitor closely to verify that no recordings
are being lost and that the recording flow is stable.
If you encounter any issues please let us know immediately.
Thx,
Brgds,
Jørgen Qvist
mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
Hej!
Vi har för närvarande problem med Zoom och felsökning pågår.
mvh
Monika
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
---------- Forwarded message ---------
Från: NORDUnet Status Notification <
do-not-reply(a)d3l31203d313yz.cloudfront.net>
Date: tis 22 okt. 2024 kl 19:17
Subject: Status Notification from NORDUnet Status
To: <monika.allov-andersson(a)sunet.se>
Incident Update NORDUnet Status
*Current Status:* Service Disruption
*Started:* October 22, 2024 17:17 UTC
*Resolved:*
*Affected Infrastructure*
*Components:* Zoom
*Locations:* On-Premise, Zone 3 - SE #1
*Update:*
There are currently connection issues with Zoom. We're investigating.
Manage Subscription
<https://status.nordu.net/pages/subscriber/manage/6135c0c26fcce8052dde4f06/6…>
| Visit Status Page <https://status.nordu.net>
Powered by Status.io <https://status.io>
All,
some good news. Zoom believe that they have identified the root cause of
the problem.
It has to do with that the recording connector needs to migrate files
during processing.
(If the disk is busy at the time the file movement might fail and this is
resulting in a “dirty” file, and this also affects subsequent files and is
the reasoning for the generic slowness.)
Zoom is working on a new release of the recording connector, that will be
deployed as soon as available.
Brgds,
Jørgen Qvist
Det kan ta upp till 48 timmar innan en recording dyker upp men *vår
rekommendation (och NORDUnets) är att stänga av funktionen tills problemet
är löst. *Vi har kommunicerat detta i omgångar, men det trillar fortfarande
in ärenden om förlorade inspelningar.
Vill ni vara helt säkra på att det fungerar så stäng av på kontonivå. Vi
vet att både NORDUnet och Zoom jobbar på problemet.
Alltså, stäng av funktionen för cloudrecording i Zoom tills problemet är
löst.
Mvh
Monika Allöv Andersson
Systemförvaltare - Utbildningsstödjande tjänster
System administrator - Educational Services
Sunet - Swedish University Computer Network
Vetenskapsrådet | Swedish Research Council
Phone: +46 72 394 78 58
