Large scale Moodle support (IT)
26 June 2021
EL
00:09
Eloy Lafuente
They did it, I think?
BH
04:41
Brendan Heywood (Moodle)
Yes they wrote the excimer library
28 June 2021
BH
02:54
Brendan Heywood (Moodle)
In reply to this message
hey @nadavkav I appreciate the blog post - but this plugin is barely even alpha at this stage, might be a bit premature
Matt Porritt invited Matt Porritt
NK
09:07
Nadav Kavalerchik
In reply to this message
If you are referring to: http://moodlemagic.info/blog/
(which is my public bookmarks/favorites which I share with other colleagues on other universities, and not really a blog)
Than I am sorry 😞 as I was too excited.
I have installed it, locally, and it works very nice.
I am looking forward to see how it develops.
BH
09:28
Brendan Heywood (Moodle)
its all good :) I just don't want people raising issues in github for a while yet
30 June 2021
Temesgen E. invited Temesgen E.
2 July 2021
GH
16:43
Gavin Henrick
go catalyst !
3 July 2021
AS
01:31
Alistair Spark
Wahey
01:36
Yep it was quite a summer. New CIO, first big decision was to go cloud for Moodle, procurement within 2-3 weeks. 10 weeks to implement.

Major upgrade with less than a week’s handover.

On-prem scale up continued but we would not have had enough hardware in the datacenters in time for start of AY.

Brilliant relationship so far.
01:39
But most importantly, this summer, I can finally take a holiday 🏖🍹🧘‍♂️
MD
07:32
Martin Dougiamas
Sticker
Not included, change data exporting settings to download.
19.8 KB
NK
10:07
Nadav Kavalerchik
BBB conference team just released the video recording for their sessions. and if anyone of you is struggling with running large scale BBB clusters to support your large scale Moodle systems... you better look at BBB@Scale session (and other sessions too!)
https://bigbluebutton.org/global-conference/
5 July 2021
Delta004 invited Delta004
D
16:30
Delta004
Just joined the group Hi everybody
16:30
Uisng Moodle
one emerging invited one emerging
oe
19:23
one emerging
In reply to this message
would it be based on the m4e operator by chance? or any pointers to k8s operators/charts you're using?
AL
20:04
Avi Levy
In reply to this message
It not based on the m4e operator, We use the Zalando postgres operator for DB and Kiali operator. We manage all service mesh with Istio. other components are based on official images of nginx, php, redis, with modifications that we made and some of them are build from scratch like postfix and unoconv.
the solutions. for monitor and scalling we used the prometheus and promethues-adapter
JC
20:30
Job Céspedes
Developer and user of M4e operator here. As developers, the approach is for it to be a reliable way to handle the lifecycle of small to medium size Moodle installation in multiple cloud providers. As a user, our use case for it is to do so in a managed service.

However, large Moodle installations have particular requirements and challenges. The M4e operator approach is not intended for such use cases
oe
20:34
one emerging
thanks! any further pointers/info to share?
JC
20:47
Job Céspedes
There has been interesting information on the subject (k8s) in previous messages int this group. History search could help you
JC
21:04
Job Céspedes
One thing, may not be that useful, is that we are testing with a NFS operator with autoexpansion for shared storage (moodledata) in GCP. We have found that managed NFS services in main cloud providers are not ideal for our use case. GCP managed NFS service (filestore), for instance, requires a minimum size of 1 TB (costs and size is more than want we want/need). We believe deploying NFS servers in this way for each Moodle installation, reduces costs, help us keep shared-nothing concept, works in multiple cloud providers, all in the same Kubernetes environment. The autoexpansion feature helps with pvc size autoincrements when needed. This approach is not without challanges, though. We have to maintain/test our NFS operator implementation, for instance.
AS
21:08
Alistair Spark
I don’t remember this being discussed but if anyone else is curious - https://github.com/krestomatio/m4e-operator
AL
23:28
Avi Levy
In reply to this message
My golden rule is that any process you see in "ps" commend in Moodle cluster should have is own service/pod, then try to find the metrics for each one of them that should told you when this process should scale vertical or horizontal. (cpu and memory should not be part of the horizontal one)
oe
23:33
one emerging
In reply to this message
🙏 thanks.
23:45
In reply to this message
not sure if already posted, could've missed it somewhere in the history...
AL
23:48
Avi Levy
In reply to this message
no, I not able to get it yet, sorry. Upgrade season
6 July 2021
Monica Ferraro invited Monica Ferraro
MB
12:15
Martin Božič
How did you solve problems with auto backup and core\lock\mysql_lock_factory? Every night I have about 1k of courses to backup and I have about 50% failure rate. Should I reduce adhoc tasks concurrency?
BH
12:17
Brendan Heywood (Moodle)
what exactly are the symptoms of a failure?
MB
12:33
Martin Božič
Exception ignored in shutdown function core\lock\mysql_lock_factory::auto_release: can't read from the database
12:33
Potential coding error - existing temptables found when disposing database. Must be dropped!
BH
12:41
Brendan Heywood (Moodle)
that smells like a bug elsewhere, I'm not sure that has much to do directly with the locks and I'd guess whatever the root cause is breaking both the lock implementation and the temp tables
MB
12:56
Martin Božič
Hm, I'll try to catch some more information with adhoc debugging turned on.
AS
13:44
Alistair Spark
Have you got a scheduled task syncing user accounts by any chance?
MB
13:45
Martin Božič
In reply to this message
Actually yes, cohorts and users!
13:48
But there weren't any of those run during scheduled backup.
AS
14:17
Alistair Spark
I was thinking it could be something like https://tracker.moodle.org/browse/MDL-60666
M
14:17
MoodleBot
MDL-60666 - LDAP Sync (auth/ldap) - prevents access to site as transaction blocks writes to mdl_user
Status: Waiting for peer review
Assignee: Alistair Spark
Reporter: Joseph Cape
Votes: 2
Michele Malagnini invited Michele Malagnini
Deleted invited Deleted Account
7 July 2021
TH
13:55
Tim Hunt
We have a number of reporting queries that we would like to run against the logs table, which would be massively helped by an index on component, eventname (or possibly both combined). What do people think about the pros and cons of adding that?
BH
14:11
Brendan Heywood (Moodle)
I think that makes sense, and many other places have a similar dual index
TG
20:12
Toni Ginard
We've found that when there are millions of registers in log table, course reports go very slow when filtering by fields that don't have indexes. In that cases, indexes would help a lot, but this have cons: indexes use space in database and have a cost in CPU because the DB engine has to do maintenance on them. So you are speeding a lot some cases, but slowing down a bit all the other operations. The answer is not clear, but, anyway, my feeling is adding a combined index would be positive.
8 July 2021
Albert Thompson invited Albert Thompson
9 July 2021
IC
15:18
Iban Cardona
In reply to this message
👏🏻👏🏻👏🏻
12 July 2021
Jessica Shirley invited Jessica Shirley
13 July 2021
😋 invited 😋
14 July 2021
-dyt- invited -dyt-
AS
19:05
Alistair Spark
Anyone started looking at Moodle 3.11?
19:06
In particular, wondering if anyone else gets some noise from this upgrade step when using objectfs

https://github.com/moodle/moodle/blob/MOODLE_311_STABLE/lib/db/upgrade.php#L1952-L1990
NK
20:41
Nadav Kavalerchik
I have started using objectfs to store large backup files on aws s3, and I do not think smaller files like h5p libreries should be moved into objectfs, which might workaround this issue.
I am also, just about to extend local_objectfs to be able to filter which files (names) can be pushed to s3 using regex, so I do not relay only on filesize.
We are also thinking it should support mimetypes. that's next on the list.
15 July 2021
JC
02:20
Job Céspedes
In reply to this message
Not yet
Santhosh H K invited Santhosh H K
-
08:58
-dyt-
What's everyone's choice for on premise file server for moodledata? I tested 3 nodes glusterfs with replicated and arbiter configuration, seems it at least 20 times slower based on dd test compared to local data. Is this a tradeoff I have to live with in order to use file server cluster solution or you guys have better idea?
MŽ
09:02
Matej Žerovnik
We were using single server NFS, but migrated to NetApp NAS with NFS. This gives us vertical and horizontal scaling capabilities. We did not test performance per se, but we are saturating 10Gbit/s link when doing moodle courses backups (that is with spindles, no ssd in netapp). Latency is below 10ms, usually around 2-5ms. Happy so far
-
09:04
-dyt-
In reply to this message
This is promising. Thanks.
AN
09:24
Andrew Nicols (Moodle)
In reply to this message
dd doesn’t really give you good metrics because it’s a) contiguous writes, and b) not very realistic in terms of file write patterns (file sizes, clusters, etc)
MŽ
09:26
Matej Žerovnik
That is true. But as a rough estimation, if dd is slower than local, other operations will be even slower since dd is almost ideal write pattern
AN
09:29
Andrew Nicols (Moodle)
You also have to remember that any network file system will be slower than local. There's no two ways away from it.
M
09:48
Mathieu
In reply to this message
On the other side, GlusterFS is fast enough at reading files. Hopefully with caching, that should happen much much more often than writing.
-
10:25
-dyt-
In reply to this message
I agree with that. I just didn't expect it will be 20x slower than local. Maybe there is some optimization that can be made. Mind to share your setup for file server cluster?
10:30
In reply to this message
Hopefully. I'm in the midst of designing large moodle infra to cater 5000 users. Will focus on caching part once I decided on which file cluster system to use.
JT
11:10
Jordan Tomkinson
In reply to this message
Ew, it would have been nice if that upgrade step used the h5p libraries muc cache I created, would be near instant if it had...
AN
12:16
Andrew Nicols (Moodle)
Caching is disabled during upgrade
MJ
12:17
Mark Johnson
In reply to this message
We've always used an NFS server for shared moodledata on-premise, and it does the job. One thing to consider is cache configuration. Put shared caches on a dedicated cache store like Redis or Memcached, not on a shared filesystem. Put caches that dont have to be shared on the local filesystem, or a local cache store.
MŽ
12:45
Matej Žerovnik
In reply to this message
+1 at that. We saw major performance boost when moving caches and locking to shared and local redis instances.
12:45
If you can live with NFS only, that isnthe simplest and fastest way.
-
13:18
-dyt-
Thanks everyone. Will further explore the cache part. 🙏🙏
AS
17:52
Alistair Spark
was running the upgrade in a test environment that has read-only access to the bucket which sometimes throw errors because it expects to be able to write but did find some plugins incorrectly addressing files during an upgrade step. (https://github.com/gjb2048/moodle-format_grid/issues/143). So was mainly asking before I spent time digging into it, in the remaining hours before I go on leave.
AS
18:35
Alistair Spark
would definitely recommend the object storage approach as it also ticks off the course backup capability for us. We can spin up a Moodle instance with just the database backup & know all the files will be included without moving TBs of data around & recover the deleted activity or item. While that relies on actually forcing all files to be uploaded - which is not the default config. It then brings down files to serve them & keeps them around if they are frequently used.
18:37
Sort of a cached filedir but shared. Makes me wonder if that's something been looked at, probably would be too complex to have a "cached" filedir on each node with the source of truth on a shared store but seems like it could avoid the IO issues some of the HA nfs solutions have
MŽ
22:18
Matej Žerovnik
That sounds nice😀😀
22:18
And its easier to maintain minio cluster than glusterfs or ceph
22:19
Of use clouds provider object storage if one is in the cloud
JC
22:22
Job Céspedes
Arroz y pollo o carne
EL
22:24
Eloy Lafuente
Heh, Job ? Menu for today?
JC
22:24
Job Céspedes
😁 opss sorry
22:25
Launch time here
EL
22:25
Eloy Lafuente
🚀 , lol!
16 July 2021
BH
02:12
Brendan Heywood (Moodle)
In reply to this message
We have considered settings to keep files in shared filedir longer, and also a second cache on each local too. We would probably not use them but not opposed to them. The more you scale up the lighter and more stateless you want the frontends. It also feels like we are reinventing a new filesystem to work around issues in a filesystem because file systems are just hard
02:20
In reply to this message
I'm pretty sure using filename regex will not be performant, mimetype does make sense and we've considered it that but not got around to it, probably the better approach instead of regex is settings per component and filearea.
NK
14:32
Nadav Kavalerchik
In reply to this message
Yes, you are probably right. still, I am doing some performance tests to see what would be best, and even consider adding some indexes to mdl_files if necessary.
BH
15:23
Brendan Heywood (Moodle)
It also seems a bit weird to me from a functional point of view, I'm more likely to want to say 'all h5p files should behave like blah' rather than 'files called bar do that'. Got some concrete examples? We can move this into a github issue too
NK
16:45
Nadav Kavalerchik
In reply to this message
Again, you are probably right, and filtering for mimetype/component/filearea is enough. though I would probably like to use a comma separated list of items for each filter.
I have opened an issue, and I will upload a patch
https://github.com/catalyst/moodle-tool_objectfs/issues/424
AS
18:39
Alistair Spark
In reply to this message
Yeah I didn't mean that in objectfs necessarily - more as another file store plugin specifically for NFS driven systems since there's usually hard drive space going spare on app servers. Indeed .
18:41
In reply to this message
i'm not sure I understand the rationale. There is filtering if you only wanted some to be delivered by the cloudfront side though
NK
20:37
Nadav Kavalerchik
In reply to this message
I use objectfs with on-premise servers, to push all the automatic backup files and large media files to aws S3
19 July 2021
BH
09:05
Brendan Heywood (Moodle)
In reply to this message
ok, but why make a whole new plugin vs adding a config option?
20 July 2021
JT
12:35
Jordan Tomkinson
FYI - All our 3.11 upgrades are erroring out due to memory exhaustion.. tracked it down to a function committed as part of MDL-28452 - theres a patch in MDL-72025 that is working. our {course_modules} table was 2.3 million records and the original function was doing a SELECT * on that table, sigh. hopefully this helps other people with large sites..
M
12:35
MoodleBot
MDL-72025 - Memory exhausted on user_profile_social_update_module_availability()
Status: Waiting for peer review
Assignee: Juan Segarra Montesinos
Reporter: Juan Segarra Montesinos
Votes: 2
https://tracker.moodle.org/browse/MDL-72025
12:35
MDL-28452 - Convert user profile fields for messaging/networking into custom profile fields
Status: Closed - Fixed
Assignee: Bas Brands
Reporter: Martin Dougiamas
Integrator: Adrian Greeve
Fix Versions: 3.11
Votes: 27
https://tracker.moodle.org/browse/MDL-28452
NK
12:58
Nadav Kavalerchik
I think it was mentioned in the 3.11 release notes:
https://docs.moodle.org/dev/Moodle_3.11_release_notes#Warnings
JT
12:59
Jordan Tomkinson
the release notes mention it takes a long time, but it is exhausting all memory due to the function selecting all records at once. that is unrecoverable without the above patch
13:00
this is also unrelated to the column modifications portion of the code.
NK
13:00
Nadav Kavalerchik
Thank you @jtomkinson
AN
13:34
Andrew Nicols (Moodle)
Surely better to use a record set
21 July 2021
AN
07:11
Andrew Nicols (Moodle)
Hey folks, I've just opened MDL-72194 to discuss dropping support for the alternative admin feature in Moodle.

I doubt anyone here is using it, but if you are it would be useful to know your justification.
M
07:11
MoodleBot
MDL-72194 - Drop support for $CFG->admin
Status: Open
Reporter: Andrew Lyons
https://tracker.moodle.org/browse/MDL-72194
MV
21:36
Mallika Valluru
Please recommend us the maximum number of concurrent students that can login into moodle server of 32 GB RAM and 16 VCPU.
21:40
Also suggest what would be max count of concurrent students who can take multiple choice assessment with same configuration. Thank you.
MŽ
23:19
Matej Žerovnik
Thats a hard one as with all things in OT: it depends
23:20
You will get the best result with actual tests
22 July 2021
Darío invited Darío