Large scale Moodle support (IT)
28 January 2021
AN
08:53
Andrew Nicols (Moodle)
But the design allows for you to disable adhoc tasks from the main cron runner, and just have dedicated adhoc runners
MB
08:54
Martin Božič
In reply to this message
I was thinking of using the admin/cli/automated_backups.php
AN
08:54
Andrew Nicols (Moodle)
That's a different question
BH
08:55
Brendan Heywood (Moodle)
typical numbers are 60 task runners, and max 30 are backups:

$CFG->task_concurrency_limit = [
'core\task\course_backup_task' => 28,
'core_course\task\course_delete_modules' => 6,
];
AN
08:56
Andrew Nicols (Moodle)
The automated-backups CLI task is unrelated to the adhoc task infrastructure
08:56
I think it can work, but I'm not sure it can be safely parallelelised
08:56
You would be better to run as Brendan suggests and have dedicated adhoc task runners
BH
09:08
Brendan Heywood (Moodle)
To answer the original question, yes we do also run cron (all of cron and the adhoc task runners) on different boxes to the front ends, they are tuned and scale differently. Cron on your frontends is not great
MB
09:42
Martin Božič
In reply to this message
If I understand this right, you can set concurrency per task in config.php? And then you run split the tasks on different frontends?
AN
09:43
Andrew Nicols (Moodle)
You can set max concurrency per task type for adhoc tasks in config.php
BH
09:43
Brendan Heywood (Moodle)
yes you can throttle how many of the same type of task run at once, but you can't control much where they run
AN
09:43
Andrew Nicols (Moodle)
And then you can choose where you run the adhoc task runner
BH
09:44
Brendan Heywood (Moodle)
I just realised this isn't documented in config-dist
M
09:46
MoodleBot
MDL-64610 - Add support for per-task concurrency limits
Status: Closed - Fixed
Assignee: Mikhail Golenkov
Reporter: Andrew Nicols
Integrator: Jake Dallimore
Fix Versions: 3.9
Votes: 3
MB
09:54
Martin Božič
Ok, these are great answers, thank you both!
AN
09:54
Andrew Nicols (Moodle)
(Brendan and I wrote this stuff)
BH
09:54
Brendan Heywood (Moodle)
standby for a sec, editing the wiki and you can sanity check it
09:59
In reply to this message
Please review this, the bottom two sections https://docs.moodle.org/310/en/Cron_with_Unix_or_Linux
10:02
hahaha
AN
10:02
Andrew Nicols (Moodle)
Wrong window
BH
10:02
Brendan Heywood (Moodle)
I was expecting a moot talk on cron :P
AN
10:03
Andrew Nicols (Moodle)
In the style of FT? Might be worthwhile
MB
10:04
Martin Božič
In reply to this message
Ok, I'll check ASAP.
TG
10:40
Toni Ginard
In reply to this message
Thanks Job and Tobias for your suggestions. I'll check them.
MB
10:41
Martin Božič
In reply to this message
Brendan, I've read the wiki but I guess I'm still a bit confused by the naming. Is "task runner" a generic term for adhoc and scheduled tasks runner scripts in admin/cli/?
BH
12:07
Brendan Heywood (Moodle)
yeah task runner is just what I guess I've come to call it, I've rewritten that and used the word process instead and fleshed out a few more bits
MB
12:16
Martin Božič
Wow, this is now much more detailed, thanks! I guess I'll have some task processing reconfiguration to do 😊
BH
12:17
Brendan Heywood (Moodle)
it probably needs to be its own page not tied to unix
12:21
I've also just removed all the /dev/null piping, you shouldn't swallow the logs
MB
12:32
Martin Božič
Because the logging is in the database anyway?
BH
12:34
Brendan Heywood (Moodle)
we log everything to syslog, but each env is going to be different so I'm hesitant to offer a 'best practice' in the wiki - but at the very least I don't think it should be piped to /dev/null like it was and not logged anywhere
12:34
the db doesn't store them forever, it rotates them out
AN
13:20
Andrew Nicols (Moodle)
In reply to this message
Maybe just mention that it should be captured
JC
17:09
Job Céspedes
In reply to this message
👌
29 January 2021
GiDA invited GiDA
1 February 2021
A
03:36
Arif
Hi everyone


Anyone using WAF on moodle?
Any recommendation for WAF and how the performance?
AN
03:37
Andrew Nicols (Moodle)
What is WAF?
A
03:37
Arif
Web application firewall
03:38
Basically to protect web to attacker
AN
03:38
Andrew Nicols (Moodle)
Yup - just good to be clear on acronyms
03:39
Okay, I'd suggest posting in the performance and/or installation forums on moodle.org as they're more appropriate than this develoepr channel.

Lots of people do use a Web Application Firewall, but there are some gotchas. Some of them detect normal use as attacks, especially when a site is in dev mode.

I'd strongly recommend posting on moodle.org and detail which firewall you intend to use (it's an acronym, not a brand or product). Others will almost certainly use the same one that you're looking at, and will be able to tell you if there's anything to watch out for.
A
03:40
Arif
In reply to this message
Sure.
Thanks for the recommendation
DM
03:41
Dan Marsden
"WAF" also suggests an off-the-shelf pre-configured firewall which typically don't support Moodle out of the box.. I'm not aware of anyone sharing their "WAF" config for a specific product.
03:41
of course - we have firewalls in front of our web servers... but I wouldn't call them "WAF"
AN
03:41
Andrew Nicols (Moodle)
^^
A
03:42
Arif
We already use Azure Application Gateway previously.

Too bad the performance around around 95%

Using standard load balancer can get 100%

2K user Load test
Arif invited Yeow Chyuan Kwan and Munir Tajudin
oh
08:05
omer hameiri
We are using AWS's WAF
MB
10:30
Martin Božič
We use Cloudflare's WAF. AFAIK we used the defaults. The only problem were the SQL homework s from ICT students sent to the teachers via assigments module.
TH
10:53
Tim Hunt
We are intending to use AWS WAF for the OU's cloud install. However, we had to go in an disable a bunch of the rules. Some of it herusitics aren't very good for Moodle.
10:53
I means, sure, if you are a running an ecommerce site, then probably no-one ever needs to submit any POST data including '; DROP TABLE users;'
10:54
However, if you teach databases or security as an academic subject, then probably they do.
10:55
Over the years, in the Moodle quiz forums, there have been a steady stream of posts from people where their 'security' firewall causes quiz results to be thrown away.
10:55
If it wasn't so tragic, it would be funny that a in information security product was causing dataloss.
10:56
Really classic one recently, a quiz response 1=1 got rejected. I would love to know whcih heuristic caused that, any why anyone thought it was a sensible heursitic to add.
10:56
Before WAF and things, it used to be apache mod_security.
10:57
Anyway, in summary, if it was up to me, I would not use a firewall like this with Moodle. I would rely on Moodle's security design. The firewall will cause more false positives than it does good.
AN
11:00
Andrew Nicols (Moodle)
I know that we used to use such a firewall at Lancs and we had to disable bits periodically. We weren’t allowed to disable it entirely
TH
11:03
Tim Hunt
Yes. I expect that is the situation we are in.
11:03
When they were testing this, I had quite a lot of fun coming up with a list of strings that could reasonably be used legitimately in our of our forums or quizzes, which I thought would be wrongly blocked.
11:04
Also, search is worse, because that is a GET, and the heursitics are tougher there.
NK
15:37
Nadav Kavalerchik
For Moodle instances on AWS cloud: we use the AWS WAF, and had to tweak it a little bit to support Moodle, as it was blocking some standard Moodle quiz actions. It depends on the rule packages you use.

And for On premise Moodle instances: we use F5 WAF (ASM) with a lot of customization to allow for normal Moodle functionality to work properly. It was running in "learning" mode for 6 months, which helped us understand how to ignore some false positives.

We are currently considering two WAF alternatives for the cloud: ReBlaze & Imperva.
MŽ
19:30
Matej Žerovnik
Same experince here with CloudFlare WAF. Lots of uploads blocked due to sql injection or javascript xss reasons. Has to tweak it and now it works
2 February 2021
BH
00:27
Brendan Heywood (Moodle)
we also use the AWS WAF with a bunch of rules turned off
TH
10:51
Tim Hunt
What form does AWS WAF configuration take? Is it a .yaml file? I wonder if collectively maintaining one in a github repo would be worthwhile?
MJ
11:45
Mark Johnson
There are several ways to configure
it, including a CloudFormation template which can be YAML or JSON.
11:46
We could extract the relevant part of our template fairly easily
AL
22:19
Avi Levy
If someone use reblaze?
22:20
And could sure his waf rules template, I'll be more then happy😀
3 February 2021
MB
11:21
Martin Božič
MDL-51111 reared it's ugly head again on our site and I'm considering to migrate to https://github.com/catalyst/moodle-tool_forcedcache but reading MDL-70233 I'm not so sure it would help us enough.
M
11:21
MoodleBot
MDL-70233 - alternative_cache_factory_class isn't honored when caching is disabled
Status: Open
Reporter: Brendan Heywood
https://tracker.moodle.org/browse/MDL-70233
11:21
MDL-51111 - MUC config.php random reset
Status: Waiting for peer review
Assignee: Brendan Heywood
Reporter: Rikard E
Votes: 23
https://tracker.moodle.org/browse/MDL-51111
Vasileios Sotiras invited Vasileios Sotiras
oh
13:04
omer hameiri
I've had this bug last week on a few of our instances.
It was devastating.. And very very hard to debug
MŽ
13:07
Matej Žerovnik
Yes, it's hard to see what the reason is when it happens for the first time:) We are used to it by now and quickly recogise the effects. We just rolled back MDL-70137 as this gives us an option to set chmod 444 to the muc/config.php. With MDL-70137, we would need to set the directory to root:root owner, which raises an exception if muc folder is not writable.
M
13:07
MoodleBot
MDL-70137 - MUC config can become corrupt in certain cases
Status: Closed - Fixed
Assignee: Andrew Nicols
Reporter: Andrew Nicols
Integrator: Víctor Déniz Falcón
Fix Versions: 3.8.6, 3.9.3
https://tracker.moodle.org/browse/MDL-70137
BH
14:27
Brendan Heywood (Moodle)
in the grand scheme of things MDL-70233 is a not a huge deal. It means on a few select pages that legacy muc/config.php file is still touched, but it is ignored by all the main traffic so is irrelvant
M
14:27
MoodleBot
MDL-70233 - alternative_cache_factory_class isn't honored when caching is disabled
Status: Open
Reporter: Brendan Heywood
https://tracker.moodle.org/browse/MDL-70233
IC
18:55
Iban Cardona
In reply to this message
Could you share it with us?
4 February 2021
BH
01:06
Brendan Heywood (Moodle)
can I suggest bundling that into a moodle plugin, and the plugin has some runtime Check API checks with a bunch of known potential failure cases, things like a post request with sql in it
01:07
the checks would be the same across the board regardless of was vendor so the plugin could end up with some example config for a range of them
01:11
I'd be happy to help bootstrap that plugin and sort the boilerplate
MŽ
12:03
Matej Žerovnik
I have been thinking more about the issues we had yesterday with MUC config.php getting erased. I know rename operations should be atomic, but I was wondering if rthat is the case with NFS as well. I know this is not a 100% verified source, but looking at this post (https://stackoverflow.com/questions/41362016/rename-atomicity-and-nfs) it seems like there could be cases, specially with multiple clients accessing same file, that a file can be missing during rename
12:03
> Because NFS is stateless, it has to have some persistent for the client, so it can still read/write that file in the same way as it would if it had done an open/unlink on a local filesystem. And to do that - we get a double RENAME and a very brief (sub millisecond) interval whereby the file we're targeting isn't present for a LOOKUP NFS RPC to find.
MR
12:31
Michael Ruder Rapajic
Hi guy, we have an unusual problem as of this morning. One query from the cron started filling up /tmp disk : @sql1:/tmp# ls -alih

total 221M
524502 -rw-rw---- 1 mysql mysql 201M Feb 4 11:24 '#sql-temptable-264c-43954-16c.MAD'
524456 -rw-rw---- 1 mysql mysql 129M Feb 4 11:24 '#sql-temptable-264c-43954-16c.MAI'

The query in question is:

SELECT pairs.tagid, pairs.correlation, pairs.ocurrences, co.id AS correlationid
FROM (
SELECT ta.tagid, tb.tagid AS correlation, COUNT(*) AS ocurrences
FROM mdl_tag_instance ta
JOIN mdl_tag tga ON ta.tagid = tga.id
JOIN mdl_tag_instance tb ON (ta.itemtype = tb.itemtype AND ta.component = tb.component
AND ta.itemid = tb.itemid AND ta.tagid <> tb.tagid)
JOIN mdl_tag tgb ON tb.tagid = tgb.id AND tgb.tagcollid = tga.tagcollid
GROUP BY ta.tagid, tb.tagid
HAVING COUNT(*) > '2'
) pairs
LEFT JOIN mdl_tag_correlation co ON co.tagid = pairs.tagid
ORDER BY pairs.tagid ASC, pairs.ocurrences DESC, pairs.correlation ASC

We have run it on our test db and the query finishes in under a second, on the production no end. Any ideas at where should we focus?
12:33
when I kill the query in mariadb, *MAD files disssaper for a second and recreated again
MB
12:50
Martin Božič
In reply to this message
Missing indexes perhaps?
MR
13:56
Michael Ruder Rapajic
actually we found the culprit, massive tags input from a certain user
5 February 2021
Brendan Heywood (Moodle) invited Rex Lorenzo
Elearning Cloud invited Elearning Cloud
Madhu Vemana invited Madhu Vemana
G
17:34
Gillian
For large AWS installs - what are folks doing for a shared filesystem? We encountered major long-tail latency problems with EFS, currently using a third party NFS provider but would love to get back inside the AWS ecosystem for this. Don't really want to run our own single point of failure NFS server. Thought about something like Gluster or Ceph but that's a lot of work as well. Has the latency situation on EFS improved?
oh
17:56
omer hameiri
In reply to this message
EFS is terrible for PHP web apps..
However, you could have your web server load the files from efs to local ebs when spinning it up and work from local
MJ
18:07
Mark Johnson
We dont put PHP code on a shared FS, we build it into our container image. We are using EFS for moodledata, but offloading files to S3 via objectfs.
18:10
we've not migrated our main VLE yet, so this may be subject to further tweaking
G
18:12
Gillian
In reply to this message
Unfortunately, can't do that with Moodle because, even if you use localcache as much as possible, the moodledata directory still has to be shared between nodes in real time.
18:13
I've looked into objectfs but it seems to still require the moodledata directory for some things? Is there a way to get entirely away from needing any shared mounted filesystem at all?
MJ
18:14
Mark Johnson
You still need moodledata
18:15
Files go into the normal filedir before objectfs moves them on a scheduled task
G
18:16
Gillian
In reply to this message
We found that on 99.9% of file operations, EFS worked fine. But there was extreme latency on maybe 1 in 1000 or 1 in 10000 file operations, and even with using moodledata as little as possible that was still enough to make the site fall over even at low concurrency. Maybe that's improved, I hope so! What scale of concurrent users are you running on that environment today?
MJ
18:19
Mark Johnson
I dont have the figures to hand I'm afraid, we've not got it running in production on our main VLE yet, but the load testing we've done so far went well
oh
18:23
omer hameiri
In reply to this message
I was referring to the web app only.
We haven't had any issues with moodledata on efs
P
18:24
Pablo Javier Etcheverry
Do you know of any epic in the tracker where changes to file management in order to overcome these limitations are concentrated?
A
18:53
Arif
anyone facing this issue before?
18:54
basically i already upgrade the plugin.
somehow keep prompting need to update
G
19:05
Gillian
Hmm. I'm happy to see EFS doesn't seem to be breaking things anymore. I'll have to change a test site back to EFS and run load tests again - maybe the EFS service has improved. If you have an order of magnitude guess of the load testing that would be helpful, though we were seeing errors at even very low concurrency.
RL
19:56
Rex Lorenzo
Gillian We are moving our Moodle to AWS and running into problems with the slowness of EFS as well. Do you have it in bursting mode? Maybe your bursting credit ran out? Have you set a CloudWatch alarm and then changed EFS to provisioned throughput when you do run our of bursting credit?
19:57
Also, many places that are running Moodle in AWS are using Gluster... seems to be the best practice? But my team does not want to maintain a Gluster Cluster.
19:59
Also, how are people deploying their Moodle to AWS? I took Amazon's sample AWS CloudFormation scripts and updated them/modified them. CloudFormation is a nice way to handle deployments: https://github.com/ucla/aws-refarch-moodle
G
20:01
Gillian
In reply to this message
Burst credits remain high. This is a weird edge case of EFS performance where a small number of operations have unpredictable and extreme extreme latency even though the average latency remains well within SLA.
20:02
We are using containers on ECS.
RL
20:04
Rex Lorenzo
In reply to this message
Unfortunately there isn't an epic, but here is a search filter you can possilby use: https://tracker.moodle.org/issues/?jql=project%20%3D%20MDL%20AND%20component%20%3D%20Performance%20and%20status!%3DClosed%20and%20text%20~%20%22file%22
RL
22:22
Rex Lorenzo
How do you guys run cron in a clustered environment? I am thinking of having a single EC2 instance that is not a webserver, but connects to the same database/filesystem/MUC cache and have that run the Moodle cron every 2 minutes. Is that the best practice? Or do you task one of your webservers to run the cron script?
MŽ
22:24
Matej Žerovnik
that is how we do it and I think that is the preffered way
22:25
do not mix cron and webservers
oh
22:54
omer hameiri
Since we have around 1300~ separate instances - single cron server isn't enough.

We have a fleet of spot servers only running cron, and autoscaling up/down based on cpu
22:54
Using sopt.io
As cheap as it gets
22:54
Spot.io *
6 February 2021
AN
01:41
Andrew Nicols (Moodle)
In reply to this message
It needs a location on disk to temporarily store files. Does t have to be shared but you do need to have temp and cache dirs shared
01:41
In reply to this message
What limitations are you hitting?

I’m not aware of any active epic
01:42
In reply to this message
You have to have some kind of shared file system for the shared file dirs like temp, cache, etc
01:43
In reply to this message
This is what I’d recommend. You can then provision with different config, and scale from as required separately to web. You can also have webs running cron too during low activity times (overnight) if that helps
01:45
But generally speaking cron on separate nodes scales best

Don’t forget to oook at the concurrency features and the adhoc concurrency configuration.

I’ll also point out that you can have slightly different configurations for that on each node so if you have a mixture of node types for cron you can config scaling differently. You may have one main dedicated cron node and then additional smaller spots and have them determine concurrency based around the node type
BH
07:17
Brendan Heywood (Moodle)
In reply to this message
I've been working on MDL-70243 which provides the raw tools to expose and hunt down all the io. It provides very low level file io logging and well as high level UI summary of io broken down by config paths
M
07:17
MoodleBot
MDL-70243 - Add a file system performance summary into the footer and file IO debug mode
Status: Peer review in progress
Assignee: Brendan Heywood
Reporter: Brendan Heywood
Votes: 3
https://tracker.moodle.org/browse/MDL-70243
BH
07:17
Brendan Heywood (Moodle)
P
13:06
Pablo Javier Etcheverry
In reply to this message
Wow, great!
7 February 2021
Giovanni Desogus invited Giovanni Desogus
G
03:15
Gillian
That filesystem performance summary looks very useful! Look forward to it landing.
03:17
We are running cron as separate Docker containers and scheduling start through ECS Scheduled Tasks.
NK
15:07
Nadav Kavalerchik
FYI: https://tracker.moodle.org/browse/MDL-70038 PDF to PNG conversion several orders of magnitude quicker than using ghostscript, might be an interesting patch for anyone running large scale system with heavy load on student submitting assignments that need conversion to PDF
M
15:07
MoodleBot
MDL-70038 - PDF to PNG conversion serveral orders of magnitude quicker than using ghostscript
Status: Development in progress
Assignee: Jordi Pujol-Ahulló
Reporter: Jordi Pujol-Ahulló
Votes: 15
NK
16:08
Nadav Kavalerchik
Has anyone experience improvement in overall backup finished time, when using parallel CRON jobs?
If so, how much scheduled cron jobs and how much adhoc are you using?
AN
16:33
Andrew Nicols (Moodle)
Each individual backup will not be any faster but there are a lot of benefits to parallel cron. Often tasks are waiting (on IO, DB, network requests) and the thread is just waiting for responses.

There is no one size fits all - it depends on your entire stack and the work load but oook at the parallel limits, both for total concurrency and per-task concurrency.
NK
16:58
Nadav Kavalerchik
I am running 3 system level CRON jobs in parallel (and 3 adhoc) and did not notice any improvement in the time it takes for the overall nightly automatic course backup to finish.
BH
23:11
Brendan Heywood (Moodle)
as a data point we run 60 parallel adhoc task runners on our big sites
AN
23:31
Andrew Nicols (Moodle)
In reply to this message
Pretty sure they’re queued as adhoc jobs... Brendan may be able to confirm before I get to a computer
23:32
So you should see some form of performance improvement for the process as a whole if you have enough parallelisation.
BH
23:35
Brendan Heywood (Moodle)
there is a master scheduled task which spawns many children
8 February 2021
Manuel Tróia invited Manuel Tróia
10 February 2021
NK
12:00
Nadav Kavalerchik
In reply to this message
So it seems that a single main cron job for scheduled tasks is enough, as it can spawn several adhoc backup tasks that are controlled by admin settings?
BH
13:20
Brendan Heywood (Moodle)
the automated backups is very old code which predates the 2.7 task api rewrite, and it has its own internal scheduler and then it was retrofitted to use adhoc tasks for each backup. If it was rewritten today again it would be much simpler. The scheduler is admin settings but it only controls when the master task spawns the children
13:21
once the children tasks are queued they can run whenever they get processed which can be well outside when you set the schedule to be if it can't handle the load
13:30
\core\task\automated_backup_task by default runs once an hour, but in theory inside it's own scheduler logic it will only spawn children say once a night, and then the other runs each hour won't do anything
M
13:33
MoodleBot
MDL-70347 - Be able to register and schedule adhoc tasks
Status: Open
Reporter: Brendan Heywood
Votes: 3
BH
13:37
Brendan Heywood (Moodle)
what we've done in other plugins that have this problem is the master scheduled task runs and it slowly drip feeds adhoc tasks onto the queue as they get processed so you can control their schedule indirectly using the master tasks schedule. As soon as the master task stops no more new children
Sal Meneses (UTC/GMT -5:00) invited Sal Meneses (UTC/GMT -5:00)
NK
18:46
Nadav Kavalerchik
Thank you @brendanheywood for the explanation, now it is much more clear!
MDL-70347 looks an excellent way forward. (watching)
M
18:46
MoodleBot
MDL-70347 - Be able to register and schedule adhoc tasks
Status: Open
Reporter: Brendan Heywood
Votes: 4
https://tracker.moodle.org/browse/MDL-70347
11 February 2021
Mohamad invited Mohamad
13 February 2021
Mehdi Ghodrati invited Mehdi Ghodrati
NK
12:03
Nadav Kavalerchik
Is anyone using https://github.com/open-lms-open-source/moodle-local_redislock instead of File lock (or DB lock MDL-67052 )?
And how can I tell if I should be use it?
M
12:03
MoodleBot
MDL-67052 - Implement a Lock API factory using Mysql / MariaDB
Status: Closed - Fixed
Assignee: Brendan Heywood
Reporter: Brendan Heywood
Integrator: Eloy Lafuente (stronk7)
Fix Versions: 3.9
Votes: 2
https://tracker.moodle.org/browse/MDL-67052
BH
12:23
Brendan Heywood (Moodle)
we aren't using it (it was higher on our radar prior to that tracker above landing), the main thing is to make sure you aren't using file locks or db record locks. I'd consider the mysql / pg advisory locks and the redis locks as all roughly equal. The main reason in my mind you'd want to get away from the db advisory locks might be some marginal reduction in primary db load but in the grand scheme of things it is tiny
NK
12:38
Nadav Kavalerchik
Thanks @brendanheywood for the explanation and advise.
I am using Moodle 3.9 with the DB locking factory (using AWS RDS MariaDB)
Digging deeper into this, I will run some performance test "vendor/bin/phpunit lib/tests/lock_test.php"
to see how the different locking is performing.
NK
14:14
Nadav Kavalerchik
In reply to this message
Looking forward to see MDL-67648 solve this issue
M
14:14
MoodleBot
MDL-67648 - Cron task manager quality of service (version 3)
Status: Open
Reporter: Brendan Heywood
Votes: 2
https://tracker.moodle.org/browse/MDL-67648
Panos invited Panos
17 February 2021
Azhari Hasbi invited Azhari Hasbi
Michal משרד Wazana invited Michal משרד Wazana
18 February 2021
Iip Permana invited Iip Permana
Lynns90 invited Lynns90
Deleted invited Deleted Account
19 February 2021
Valeria invited Valeria
V
00:23
Valeria
Hi! I'm from Argentina.
00:24
May I ask some questions about a problem I have using moodle?
MŽ
00:25
Matej Žerovnik
Fire away
P
00:26
Pablo Javier Etcheverry
Hola Valeria, if your question is about large scale Moodles, then this is the best place. If not, you can always go to the Argentinian Moodle Channel, https://t.me/MoodleArgentina
V
00:31
Valeria
Thanks!
24 February 2021
P
13:02
Panos
Hello, after moodle migration there are some links/filenames failing, eg /moodle/pluginfile.php/56/block_cocoon_slider_7/slides/1/_1JB1279.jpg
is there anyway to resolve this issue?
thanks
n
13:04
nizam
Contact edumy theme support. This is theme issue.
P
13:06
Panos
I will, thank you!
AN
13:08
Andrew Nicols (Moodle)
In reply to this message
How do you work that one out?
TJ
13:10
Timotej Jazbec
In reply to this message
I'd guess from the component/filearea bit, it does not look like default Moodle
n
13:11
nizam
Edumy themes has lots of customized blocks. Which throws error while upgrading moodle or migrating. I had premium support with their theme. They supported to fix the issue.
P
13:13
Panos
I not very familiar with moodle, for sure there is a edumy theme, but the archive I am given does not include any of those images, unless they are generated dynamically or stored in db?
AN
13:14
Andrew Nicols (Moodle)
Just to note that Edumy are not a Moodle partner
TH
19:33
Tim Hunt
I can never remember when you should use @see or @link in PHPdoc. Does anyone have a good way of remembering?
19:36
@link is the one you can use inine, so I think that is waht I want.
19:37
Oops! wrong chat
25 February 2021
Hrishikesh J invited Hrishikesh J
HN
08:22
Huong Nguyen
In reply to this message
@dobedobedoh Ban this user pls
AN
08:33
Andrew Nicols (Moodle)
I don't have permission. This is Nadav 's group
AL
14:40
Avi Levy
Just reminded that Nadav ask if someone from Moodle HQ can join to the group admins
AL
14:40
Avi Levy
NK
Nadav Kavalerchik 13.01.2021 10:28:17
BTW, if anyone from Moodle HQ or others would like to be a co admin... please PM me.
As I think it would be better to have a few admins on this group.
AN
14:40
Andrew Nicols (Moodle)
Happy to do so
IC
15:10
Iban Cardona
Hello!

Do you have experience with large moodle websites and AWS cloudfront?
JT
21:51
Jordan Tomkinson
In reply to this message
yeah, don't do it.
IC
21:55
Iban Cardona
In reply to this message
Why?
JT
21:58
Jordan Tomkinson
among other things, cloudfront has an unconfigurable 2 minute backend timeout, which is not enough for big operations like report generation, backup/restore operations. it also is unable to cache many assets of moodle because they are delivered via cookies
21:59
you will get better performance out of Varnish and a tuned vcl config, or a cdn like fastly/cloudflare
IC
22:05
Iban Cardona
Ok. Thank you so much.
RL
23:02
Rex Lorenzo
In reply to this message
Yeah, running into that.. I can only get CloudFront to have a 60 sec timeout. How do you get it to 2 minutes? Anyways, currently running it off.
26 February 2021
BH
00:02
Brendan Heywood (Moodle)
worth noting that that 1 min / 2 min is a network inactivity timeout not a hard execution limit. While cloudfront is currently not viable I think we should all collectively be working to get moodle well under this limit so that it can be behind all of these 'normal' cdn's without issue
RL
00:07
Rex Lorenzo
Yeah and long running tasks should run as an adhoc task, like the backup/restore.
BH
00:13
Brendan Heywood (Moodle)
long running tasks in the ui are also fine as long as they have a regularly updated progress bar which keep the connection alive, it's just the genuinely slow things with a 2+ minute TTFB which are the show stoppers which need to be hunted down.
00:16
As of this morning, the top 5 worst offenders are:
GET /course/format/topcoll/duplicate.php?courseid=XXX&sectionno=XXX&sesskey=XXX
POST /course/delete.php
GET /course/mod.php?sesskey=XXX&sr=XXX&duplicate=XXX
POST /enrol/self/unenrolself.php
GET /course/management.php?categoryid=XXX

ie many of the top 10 are related to backup / restore related processes
JT
00:18
Jordan Tomkinson
Too many times I've seen customer sites where they'll click grade on a course with 10k+ students and the page takes 10 minutes to load heh
BH
00:19
Brendan Heywood (Moodle)
gradebook is up there there as one of the worst offenders too it's just not used as much, it will be later in the terms
JT
00:19
Jordan Tomkinson
Bit that was due to filters being enabled and there's an MDL currently being ignored for it
BH
00:20
Brendan Heywood (Moodle)
you mean this one? https://tracker.moodle.org/browse/MDL-35044
M
00:20
MoodleBot
MDL-35044 - Expensive purify_html used by grade_get_grades
Status: Open
Reporter: Tony Levi
Fix Versions: STABLE backlog
Votes: 4
JT
00:20
Jordan Tomkinson
You know it 😉
BH
00:22
Brendan Heywood (Moodle)
I can't see Tony's branch anymore from 9?! years ago, it should be an easy one to get over the line
JT
00:22
Jordan Tomkinson
It's more than just purify_html. All the text filters run too and then get thrown out. I should really update the issue
BH
00:22
Brendan Heywood (Moodle)
yeah if you can throw in some of the workaround stuff too please
JT
00:23
Jordan Tomkinson
I have the xhprof screenshots still
00:23
Will do tomorrow
G
00:23
Gillian
We are successfully running behind CloudFront. Had to move a few long running operations to "shell script only" but nothing user facing should take that long. User initiated backups are not part of our workflow though, and we don't use gradebook extensively.. so it sounds like we may have just dodged the bullets.
BH
00:25
Brendan Heywood (Moodle)
yeah it all comes down the site, the scale, and how it's used. For us it's totally unviable
JT
00:25
Jordan Tomkinson
Funny you should mention course format topcol Brendan we just had to turn off all the additional info settings because it was taking 4 minutes to load a (large) course with them on.. the quiz counter setting to be precise
00:25
There's an issue on the gh page for it
00:26
One user accessing the course initiated 600k SQL queries
P
01:45
Pablo Javier Etcheverry
In reply to this message
We faced a somehow related problem this week, when using webservices to remotely duplicate courses from some templates. We found that if two courses were requested from the same template in about the same time, the intermediate file generated to do the backup-restore cycle would have the same name, thus generating an exception.
01:47
The duplicate function looks like a nice candidate to get asynchronized 😊
AN
01:48
Andrew Nicols (Moodle)
Really we should try and keep the backup file for templates around
01:48
Not sure how we’d compare it
P
02:11
Pablo Javier Etcheverry
That would be a great option!
02:11
like a duplicate_from_cached_backup function.... beauty
AN
02:12
Andrew Nicols (Moodle)
Well under the hood it would just check for an existing backup before performing the restore
P
02:13
Pablo Javier Etcheverry
Yes, but it would be nice that if a teacher decided to make a new "version" of the course, she could decide to automatically update that cached backup file...
AN
02:13
Andrew Nicols (Moodle)
Well ideally they woudl have no idea
02:14
I.e. they make a change to the course, and next time it's needed a new backup is created and stored for future use
P
02:14
Pablo Javier Etcheverry
yes, definitely
02:14
that would make a huge improvement
02:15
we are creating 800.000 courses... that would improve speed a lot
AN
02:15
Andrew Nicols (Moodle)
As it is though we already perform fileless backups when doing course duplication - we've been doing that since about Moodle 2.2/2.3 I think
02:15
The biggest time sink is quiz/question backup
02:16
The problem with that kind of solution is trying to determine if/when something changed
02:16
And right now I don't believe that there is a way to do so
02:16
Not reliably at least
P
02:16
Pablo Javier Etcheverry
In reply to this message
but is it the function used by "core_course_duplicate_course" webservice?
AN
02:17
Andrew Nicols (Moodle)
Any course or activity duplication involves a backup and immediate restore. Any backup and immediate restore does not includes files, or zip the backup when complete.
02:18
We have never zipped duplicate backup files, and I removed files from those backups in MDL-37761 in 2013
M
02:18
MoodleBot
MDL-37761 - Improve backup/restore within Moodle (e.g. course and activity duplication)
Status: Closed - Fixed
Assignee: Andrew Nicols
Reporter: Andrew Nicols
Integrator: Dan Poltawski
Fix Versions: 2.6
Votes: 5
https://tracker.moodle.org/browse/MDL-37761
AN
02:18
Andrew Nicols (Moodle)
So 2.6
02:19
But we still have to spend time creating the various XML files for the backup/restore
02:19
If we could somehow skip that step then that would be useful
02:20
As another option, if you have your own course lifecycle management, you could do this now yourself by modifying the step to create a file-less backup once a tthe start of the process, and use it for all future restores.
02:20
But that depends on how you manage your course lifecycle
P
02:21
Pablo Javier Etcheverry
yes
02:21
In reply to this message
but i am puzzled about this...
AN
02:22
Andrew Nicols (Moodle)
What part?
P
02:22
Pablo Javier Etcheverry
We are still working on it, but it all started with a storedfilenotcreated error. After debugging we found this:

Cannot create file "1669/backup/course/0/backup.mbz" (ERROR: duplicate key value violates unique constraint "mdl_file_pat_uix"
DETAIL: Key (pathnamehash)=(b3167e0aa345de9c3a0ffaf137a3a5ca48c81183) already exists.
INSERT INTO mdl_files (contextid,component,filearea,itemid,filepath,filename,timecreated,timemodified,mimetype,userid,source,author,license,status,sortorder,contenthash,filesize,pathnamehash)
02:23
that was when sending multiple calls of that webservice to the same moodle server
02:23
so the intermediate file is being created... am i wrong?
AN
02:24
Andrew Nicols (Moodle)
Ah - it appears you are correct
02:24
That's the .mbz file
P
02:24
Pablo Javier Etcheverry
I thought it could be related to this: https://tracker.moodle.org/browse/MDL-55859
M
02:24
MoodleBot
MDL-55859 - Assignment pdf conversion doesn't clean up after failure
Status: Closed - Fixed
Assignee: Ankit Agarwal
Reporter: Jeffery Watkins
Integrator: Dan Poltawski
Fix Versions: 3.1.5, 3.2.2
Votes: 13
AN
02:24
Andrew Nicols (Moodle)
I could have sworn that we don't do that...
P
02:24
Pablo Javier Etcheverry
(at least a similar problem)
02:25
yeah... these cases are interesting, who would think of duplicating 10 or 20 courses at the same time?
AN
02:26
Andrew Nicols (Moodle)
It's not uncommon for course lifecycle
P
02:27
Panos
hello again, I am trying to migrate a moodle site from windows to a linux server and I get the following error:
"Coding error detected, it must be fixed by a programmer: Component was not valid: theme_edumy"
any ideas? thanks
AN
02:27
Andrew Nicols (Moodle)
So we do not convert to tgz for the activity duplication, but we do for whoel course
P
02:27
Pablo Javier Etcheverry
yes
AN
02:27
Andrew Nicols (Moodle)
In reply to this message
I'd suggest posting in the support forums
02:27
But it sounds like your theme is either invalid or incorrectly installed
02:30
So @pablo_javier It looks like the duplicate_course() API is being called with the wrong mode - MODE_SAMESITE instead of MODE_IMPORT.
When you use the course import feature in the UI we use MODE_IMPORT to create the backup, but the duplicate function uses SAMESITE which prevents the files included in the course from being created, but not the mbz file itself
P
02:30
Panos
In reply to this message
hm, it might be, someone else developed the site initially, if there is a way to disable that theme from cli, thanks
P
02:32
Pablo Javier Etcheverry
In reply to this message
WOW! Great finding! Switching the mode would suffice?
AN
02:33
Andrew Nicols (Moodle)
POssibly - it may be a little more work than that
02:33
It's been a while since I've looked into backup but I think so
P
02:35
Pablo Javier Etcheverry
will check that first thing in the morning, thank you!!
AN
03:10
Andrew Nicols (Moodle)
Alas, it isn't quite that simple. We may need a new backup mode
P
03:18
Pablo Javier Etcheverry
Oops
AN
03:18
Andrew Nicols (Moodle)
Whenthe backup mode is IMPORT it prevents the users setting from being set
03:18
So it would require a little bit of work to do such a change but it's worth doing IMHO
03:20
MDL-54719
M
03:20
MoodleBot
MDL-54719 - Cannot run multiple backups simultaneously with the same user and filename
Status: Open
Reporter: Skylar Kelty
https://tracker.moodle.org/browse/MDL-54719
P
03:45
Pablo Javier Etcheverry
Nothing new under the sun 😎
03:46
In reply to this message
Certainly worthwhile!
AS
15:26
Alistair Spark
In reply to this message
Topcoll duplicate’s existence is my fault
Seeing as it’s #1 in the list I should do something about it
IC
16:20
Iban Cardona
In reply to this message
Could you join use more details about it? How did you get it?
27 February 2021
Luke Carr invited Luke Carr
LC
02:33
Luke Carr
Hi guys, got a quick question and would love to hear people’s ideas. I’m looking at setting up a horizontally scaled Moodle using Docker. What are people thoughts on separating nginx and php-from into separate containers and scaling them independently, versus scaling a bunch of nginx + php containers? I guess that fewer nginx containers would be needed than PHP? But they’d all need to have the wwwroot and moodledata directories, correct?
AN
02:39
Andrew Nicols (Moodle)
Yeah you can do... be aware of the x-send file feature. You can use that to your advantage
02:39
Most things still need php though. Very few files are served directly. Almost everything goes through a php endpoint
02:40
Partially that’s due to access control, sometimes it’s to control caching, and sometimes it’s to fill placeholders in content (ie css), or to compile content
LC
02:48
Luke Carr
I’ll take a look at x-sendfile! 2nd question, in regards to AWS specifically, I seem to remember from my own previous testing and some forum threads that EFS is discouraged in favour of something like gluster. Anyone know if this is still the case?
AN
02:50
Andrew Nicols (Moodle)
Yes
02:51
It’s discussed heavily here regularly
02:52
But look at the alternative file system API. There are a few options and they move your user data to alternative systems like S3. You still need a shared file system for a few things which we can’t move off disk easily
G
02:54
Gillian
Do those systems take enough load and critical path off of the shared file systems to permit EFS use? Or are you still stuck having to do Gluster or a dedicated NFS server?
AN
03:22
Andrew Nicols (Moodle)
You still need some form of shared storage
03:22
I’ve been trying to work out how to love the remaining items off shared storage but they are non-trivial
03:23
Backup is the main one. It is disk intensive and complex
03:27
There’s also MUC caching. That defaults to using shared disk storage until you configure it with an alternative like redis
G
04:54
Gillian
What I'm wondering - and will probably load test soon - is if this ultimately moves enough stuff off that EFS becomes a viable option. The main problem with EFS is that it will just occasionally have extreme latency for a single operation, while the average latency stays within SLA. So if it can get down to shared storage not being hit on most page loads, the remainder can be lived with.
AN
05:23
Andrew Nicols (Moodle)
I couldn’t comment but others here may be able to
05:24
I haven’t had a chance to try and work at moving any more off from shared storage for a while
BH
07:26
Brendan Heywood (Moodle)
In reply to this message
I've more recently come to the much stronger conceptual position that anything served directly by nginx is a bug that needs to be fixed, and I'm tracking metrics around this and slowly stamping them out in core and plugins. The reasoning is simple: if it needs an ACL then it must go through php pluginfile to do that logic. If it is public then it *should* go through php to set it's caching headers correctly so that varnish / cdn can handle it properly and offload it completely from nginx even if xsendfile is is on. In generally probably 99.99% fall into these two cases already, the exceptions are rare
AN
07:55
Andrew Nicols (Moodle)
In reply to this message
Exactly. The thing you gain with a separate host in front is xsendfile which has an advantage but not sure how much of an advantage
BH
07:56
Brendan Heywood (Moodle)
we still have xsendfile setup, but it really doesn't get much use in our inf as the vast majority goes through objectfs and signed urls via cloudfront
08:00
taking that to it's logical extreme is why we need a general php routing framework so we can capture all the last edge cases and allow us to dynamically serve things like /robots.txt and /.well-known/foo and other similar bits
one emerging invited one emerging
28 February 2021
Android invited Android
João Freitas invited João Freitas
JF
14:13
João Freitas
Greeting to all. We support a 10k MOODLE with 1.5 simultaneous Quiz users during exams. Looking forward to follow this group.
Adit invited Adit
1 March 2021
Vitaly P invited Vitaly P
A
11:28
Adit
hello, please help me with haproxy config for moodle server cluster
11:28
global
tune.ssl.default-dh-param 2048
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 20000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 20s
timeout queue 2m
timeout connect 20s
timeout client 2m
timeout server 2m
timeout http-keep-alive 20s
timeout check 20s
maxconn 20000

frontend http-in
bind *:80
stats uri /haproxy?stats
mode http
redirect scheme https code 301 if { hdr(Host) -i production.example.com } !{ ssl_fc }
acl development.example.xyz hdr_end(host) -i development.example.xyz
use_backend dev_server if development.example.xyz

frontend https-in
bind *:443 #ssl no-sslv3 crt /etc/haproxy/certs/development.example.xyz.pem
mode tcp
option tcplog
option forwardfor
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
acl development.example.xyz hdr_end(host) -i development.example.xyz
acl production.example.com hdr_end(host) -i production.example.com
option forwardfor
use_backend prod_server if { req_ssl_sni -i production.example.com }
use_backend dev_server if development.example.xyz

backend dev_server
mode http
option forwardfor
balance roundrobin
server app-dev1 999.999.1.1:80 check
server app-dev2 999.999.1.2:80 check

backend prod_server
balance roundrobin
option forwardfor header X-Client-IP
mode tcp
server app-1 999.999.1.3:443 check
server app-2 999.999.1.4:443 check
server app-4 999.999.1.5:443 check
server app-5 999.999.1.6:443 check
server app-6 999.999.1.7:443 check
server app-7 999.999.1.8:443 check
server app-8 999.999.1.9:443 check
11:30
need development.example.xyz ssl handle in haproxy
AN
11:42
Andrew Nicols (Moodle)
Post into the installation forum
11:43
But looks like you’re terminating the ssl and then sending to an ssl connection to determinate
11:43
But it’s been a while since I’ve configured haproxy
A
11:45
Adit
bind *:443 ssl no-sslv3 crt /etc/haproxy/certs/development.example.xyz.pem

if i do this,
production.example.com won't work
2 March 2021
P
01:59
Pablo Javier Etcheverry
In reply to this message
On a followup on this: I found the temp directory for backups filled with... 65000 directories that were not removed after a failed course duplication operation. That's where my remaining disk was! 😱😂
Eloy Lafuente invited Eloy Lafuente
Nils Adermann invited Nils Adermann
3 March 2021
Naajii Zinaab%duchoo invited Naajii Zinaab%duchoo
P
23:26
Panos
Hello, I am getting an error trying to run the cron command:
sudo -u www-data /usr/bin/php /moodle/html/moodle/admin/cli/cron.php
!!! Exception - Class 'core\local\cli\shutdown' not found !!!
why it cannot find core\local... path?
4 March 2021
BH
00:14
Brendan Heywood (Moodle)
thats weird, ramp up the debugging
P
00:16
Panos
I've migrated the site to an other server, which by default using an older version of moodle, the code and dataroot are in different folders, so I am wandering if I had to remove caches too?
00:19
In reply to this message
is there any debug flag I can pass to that command?
BH
00:21
Brendan Heywood (Moodle)
https://docs.moodle.org/310/en/Debugging
00:21
// $CFG->showcrondebugging = true;
P
00:27
Panos
it displays the same message
BH
00:29
Brendan Heywood (Moodle)
it sounds like the code is completely messed up and you should restore it or update it
P
00:32
Panos
hm, is admin/cli/purge_caches.php going to damage the state of the site?
BH
00:33
Brendan Heywood (Moodle)
I would expect your entire site is broken in a myriad of ways, but no clearing the cache should be ok
AN
02:00
Andrew Nicols (Moodle)
In reply to this message
This is almost certainly your problem. You must must have the same version of code on all servers
02:01
And once you have done so, purge your caches.
02:01
Right now you could have all sorts of weird and wonderful things happening, corrupting data
P
02:10
Panos
Thanks, I followed the migration steps, the old installation is kept in different folders including the database. I hope the site is not that broken, the only apparent issue was the Cron job.
AN
02:11
Andrew Nicols (Moodle)
In which case I may be confused by your statement.

I would recommend purging cache at a minimum, but really this is an installation support question and shoudl be directed to the appropriate forum.
02:12
This channel is aimed at supporting Large Scale sites, primarily around performance, architecture, and design.
P
02:36
Panos
May be it's an installation support question. The site was migrated to a cloud installation and it can be the complexity of its architecture in which I do not have much control. I will ask the Moodle forum too, thanks for your answers.
JC
05:43
Job Céspedes
In reply to this message
Even though container best practices suggest a container should have only one responsibility, it is still quite common to have nginx (or apache) on the same container as php. We are currently trying separate containers for nginx and php-fpm in Moodle Kubernetes operator. Code and moodledata are mounted in the same path for both containers. What we want to figure out is if this config works best for resource optimization while keeping good performance. It's been mainly testing so far. So, I can't comment further.
05:52
In reply to this message
Interesting argument. I would think that anything that can be handle in an outer layer should not go to a lower one. Example, if Nginx/Apache do static files or caching, let's use that and avoid php load.
AN
05:53
Andrew Nicols (Moodle)
The issue is that Moodle is very keen on access control (and so are most of its users)
05:53
Which means that there isn't very much content which does not come under access control
05:53
Or have the potential to, even if your specific use does not
05:53
Additionally, many things are pre-processed (SCSS, CSS)
BH
05:54
Brendan Heywood (Moodle)
the problem is that we'd then have to ship a whole bunch of code for nginx and apache and varnish and cdn X Y and X, or we can do it all in one place in php
05:54
the application layer will always be able to make better decisions
05:56
and doing it this way is precisely to avoid php load, it's counter intuitive
JC
06:02
Job Céspedes
I wasn't considering all those aspects. Thanks.
AN
06:03
Andrew Nicols (Moodle)
There are definitely things you can offload, but mostly we're talking about X-Sendfile, which has been in Moodle for about 8 years.
Essentially that allows you to perform all of the access control in Moodle, but then for Moodle to return an appropriate header so that the web server serves the file and frees up the PHP process
JC
06:08
Job Céspedes
I was also thinking that sometimes letting moodle (app layer) handle some things is more expensive (web server + php + database) vs Nginx/Apache (just web server)
AN
06:09
Andrew Nicols (Moodle)
So yeah, it will be more expensive, but realistically there is little-to-no way to be correct without them.
06:10
And people get very upset when content is incorrectly released. Imagine if we did not correctly check access permissions to assignment submissions, or images used in quiz questions for an important test.
JC
06:18
Job Céspedes
🙈
AN
06:18
Andrew Nicols (Moodle)
Exactly ;)
NK
17:12
Nadav Kavalerchik
In reply to this message
Just added @dobedobedoh as admin too
(should have done so before going offline for two weeks Vipassana meditation 😉, me bad)
5 March 2021
A
05:34
Arif
Anyone using cloudflare?

What kind of plan you're using for the moodle?
MB
06:34
Martin Božič
In reply to this message
We do. We use business plan primarily because that one enabled us to redirect just one subdomain to CF instead of whole domain.
A
06:38
Arif
In reply to this message
🤔. You use cloudflare on all of your site or moodle only?
MB
06:40
Martin Božič
In reply to this message
Just Moodle.
A
10:13
Arif
In reply to this message
I was thinking to use Pro, however it doesnt stated come with custom rule WAF
10:14
+ Doesnt come with SLA thought
Nadav Kavalerchik invited IFTTT
NK
11:57
Nadav Kavalerchik
/start@IFTTT
I
11:57
IFTTT
In reply to this message
This group was successfully connected to your IFTTT account. You can now use it with Telegram Applets on IFTTT.
6 March 2021
AS
21:54
Alistair Spark
In reply to this message
We will flick over to CF in a couple of weeks.

We’ve been using AWS Shield Advanced. But CF is our strategic choice for the institution at large.
21:54
Enterprise plan iirc
8 March 2021
A
13:08
Arif
Thanks for the info.
I might go with business plan since it's a easy setup. Just cname dns only
9 March 2021
Ld
12:35
Luis de Vasconcelos
Hi. Does anybody have experience with running large Moodle installations (900+ courses 80000+ students) on Microsoft Windows Server OS and MS SQL Server databases?

If so, what's your experience like?
12:37
Using Isilon NAS storage for Moodledata.
DM
20:09
Dan Marsden
In reply to this message
I've had major issues with the performance of the php ms sql driver - so we moved the clients to Linux based tools.
10 March 2021
A
10:17
Arif
In reply to this message
Hi Martin,

Do you have any issue on session keep changing whenever user login when using cloudflare?
or you add few header on your web server?
MB
18:26
Martin Božič
In reply to this message
No problems whatsoever. We set session affinity to "By Cloudflare cookie only". And we've only set ' RemoteIPHeader CF-Connecting-IP' in the vhost config (httpd) but that's for logging, not session.
18:27
We only use SAML login (currently via builtin auth_shibboleth).
A
19:40
Arif
In reply to this message
🤔. Let me check first.
As i have issue with previous config running on azure front door
11 March 2021
Andrew Sand invited Andrew Sand
NK
09:35
Nadav Kavalerchik
Hello everyone,
I am organizing a public webinar together with GÉANT to all the NRENs and anyone who is interested about case studies for managing large scale Moodle system, and I am looking for presenters 😊
So if anyone of you is interested in showcasing your Moodle system, share some statistics, challenges and solutions, we would love to have you join the presenters team.
We are aiming for a 15-20min short presentation.
We will try to make it as diverse as possible considering the architecture and cloud solution you use.
Also, if anyone think we should add anything interesting to the agenda... please reply with ideas.

EDIT: Planned for the 20th of April 2021
12 March 2021
NK
09:48
Nadav Kavalerchik
Is anyone using this relatively new ReBlaze cloud native security product? (as WAF?)
https://www.curiefense.io/
If so, how easy to setup? and how easy to manage? and does it provide good protection against DDOS?
Viraj De Mendis invited Viraj De Mendis
13 March 2021
Nethan invited Nethan
15 March 2021
NK
08:38
Nadav Kavalerchik
What type of shared file system do you use for moodledata?
Anonymous poll
- Dedicated NFS server 11 votes
- AWS EFS 9 votes, chosen vote
- GCP Filestore
- Azure FIles 3 votes
- GlusterFS 1 vote
- NetApp 2 votes
- CephFS 1 vote
- Other... 5 votes
32 votes
08:42
What type of cache you use for MUC?
Anonymous poll
- REDIS 20 votes, chosen vote
- Memcache(d) 4 votes
- File 8 votes
- APCu 4 votes, chosen vote
- MongoDB
29 votes
08:45
If anyone have any ideas about interesting Polls we could share between us, to reflect the kind of technologies we use... please advise
AL
08:45
Avi Levy
For the first question I Vote other, becuse it should be multi choice😁
NK
08:47
Nadav Kavalerchik
In reply to this message
Oups, me bad, I should have though about it, as I am unable to change the Poll settings anymore 😞
AL
08:48
Avi Levy
I like the poll idea
08:49
What about, where you host your cluster?
NK
08:49
Nadav Kavalerchik
In reply to this message
Like which cloud? or on-premise?
AL
08:50
Avi Levy
👍
n
20:02
nizam
Poll is gud idea
G
20:29
Gillian
Very interesting to see so many people using EFS these days!
RL
20:29
Rex Lorenzo
For whoever voted for AWS EFS for Moodle data, are you using it in production? In benchmarking it is slow, but wondering if speed doesn't matter as much in practice
G
20:32
Gillian
We tried it a while back and found the problem to be that EFS has occasional but extreme latency spikes; burst credits are not exhausted and the _average_ latency is within SLA, so AWS says it's functioning as designed.
20:34
It would be on the order of (I don't remember the exact numbers) 99,999 reads run in .01 second each but then, at random, 1 read would take 20 seconds.
20:35
I'm hoping this has been fixed but at the time AWS didn't even consider it a defect.
IC
21:40
Iban Cardona
In reply to this message
Yes, we use it in production
RL
23:32
Rex Lorenzo
Iban How is performance? Nothing to complain about it?
23:32
Gillian Yeah and also it doesn't work well with small files... There are cases in which Moodle uses moodledata for some small files
MJ
23:39
Mark Johnson
We are using EFS for moodledata, but with objectfs to offload most files to S3, shared MUC caches in Redis, and local caches on the local filesystem
AN
23:50
Andrew Nicols (Moodle)
In reply to this message
Standard file system or a custom one?
MJ
23:50
Mark Johnson
for local caches?
G
23:51
Gillian
We have moved to redislock recently, I need to test if that's moved EFS back to acceptable performance
RL
23:52
Rex Lorenzo
How does objectfs work? I see https://moodle.org/plugins/tool_objectfs

So when a file that is not in EFS is requested it will get instead get that file from S3 if it is not found on EFS? How much latency does that introduce? Is that file then moved to EFS in case it starts to be more used?
MJ
23:52
Mark Johnson
Just standard at the moment, in /var/local. Considering putting it on ramdisk if that's not good enough
23:54
In reply to this message
Custom filesystem class, it knows if a file has been moved to S3 and either opens it and serves it from there, or redirects to a signed URL directly in S3
23:56
I dont have numbers on latency, but based on our testing, "not too much".
23:57
In reply to this message
Oh yes, we do locking with postgres not in moodledata
16 March 2021
BH
00:43
Brendan Heywood (Moodle)
I fleshed out a wiki page somewhere with each specific config path and recommendations for it but I cannot find it anywhere now
00:56
found it, a bit light but a start: https://docs.moodle.org/310/en/Server_cluster
IC
07:10
Iban Cardona
In reply to this message
No numbers to show and compare but we know it's slower than on premise NFS.
NK
09:52
Nadav Kavalerchik
In reply to this message
@brendanheywood, I noticed you updated wiki about $CFG->backuptempdir , but it seems it is already defined as $CFG->tempdir/backup (if not initially defined) and I wonder if there is a special case when it is useful to put it on s special shared FS that is not the one we use for $CFG->tempdir
BH
09:59
Brendan Heywood (Moodle)
we don't have anything definitive but we've been experimenting with efs for backuptemp as it is very bursty and putting temp on gluster as it's a more stable size
AN
10:15
Andrew Nicols (Moodle)
I'd still like to see us remove backup from temp at all
10:16
Honestly I'm not sure why it needs to be there any more. It's historically there because it was a multi-step process, but I think we can fix that so that it isn't. The only things put into that directory are the config for the backup, which is also in the DB I think
10:16
(from memory)
10:16
Restore uses it more but we may be able to only extract the moodle.xml file from the backup during the restore, and then restore the rest of the backup
NK
10:49
Nadav Kavalerchik
In reply to this message
+1
LC
14:27
Luke Carr
Hi all, is it safe for me to manually delete the moodledata/trashdir contents?
14:28
And is there any way to debug why my trashdir has grown from 4GB to 80GB in a few days? It’s never increased in size this fast.
AN
14:46
Andrew Nicols (Moodle)
There is a scheduled task which empties it regularly. From memory it waits until the mtime on the file is 7 days or so. I think it’s safe to delete manually.

Chances are it’s gotten large because someone has deleted a course, or a number of assignments with submissions, or video content
LC
14:52
Luke Carr
Hmm. I know for a fact that no video content is stored on the site, because we use a paid Vimeo subscription for all videos, so I’m just surprised that it’s grown in such size. Manually deleting has resolved immediate issues, but I’ll have to keep an eye on it. Is there any way to change the “lifecycle policy” of the trashdir files to something shorter?
14:52
I suppose I could setup a cron job on the machine to clear it daily if necessary.
EL
15:00
Eloy Lafuente
Related, but not 100% the same (just in case it’s backup’s old temp information), it’s being moved from within backup and restore operations to backup own scheduled cleanup task:

https://tracker.moodle.org/browse/MDL-71044

It doesn’t affect other temp contents.
M
15:00
MoodleBot
MDL-71044 - Extend the 'backup_cleanup_task' scheduled task to remove old files
Status: Integration review in progress
Assignee: Mikhail Golenkov
Reporter: Mikhail Golenkov
Integrator: Eloy Lafuente (stronk7)
IM
15:01
Ivica Matotek
You can change this in Administration -> Plugins ->Admin tools -> Recycle bin
AN
15:02
Andrew Nicols (Moodle)
In reply to this message
Recycle bin !== trashdir
15:02
Recycle bin is a backup which is stored in Moodle data for a period
15:04
The trashdir is a feature of the file system whereby files are moved into a different location once all references to them are gone. It’s only used to restore files in rare situations and it’s contents are not tracked
21 March 2021
NK
11:28
Nadav Kavalerchik
Where is your Moodle hosted?
Anonymous poll
- On-premise 17 votes, chosen vote
- AWS (Amazon cloud) 9 votes, chosen vote
- GPC (Google cloud) 2 votes
- Azure (Microsoft cloud) 4 votes
- DigitalOcean
- Other (you can elaborate in the reply) 1 vote
28 votes
11:32
How large is your Moodle? (users)
Anonymous poll
- Less then 50K users 10 votes, chosen vote
- More then 50K and less then 150K users 8 votes
- More then 150K and less then 250K users 5 votes
- More then 250K and less then 750K users 4 votes
- More then 750K users 1 vote
28 votes
11:33
As privately requested, more Polls 😊
11:37
Hi @brendanheywood, I am trying to offload large video files from our system (and backups) to AWS S3 using https://github.com/catalyst/moodle-tool_objectfs, can I specify that only video files (maybe by mime type) are moved to a remote S3 storage?
BH
12:25
Brendan Heywood (Moodle)
In reply to this message
No, but it would be an easy patch, pr's welcome. Using the tool_objectfs->sizethreshold setting would practically have the same effect anyway. If you've gone to the effort to set it up why not apply it to everything? If you want to offload them fully you want to setup signed urls so they are served from cloudfront (what we use) or s3 (we wrote this first but don't use it in prod so isn't as battle tested). Re backups is just about using file-less backups, or do you mean os / file level backups?
12:28
the killer with videos is the constant range requests and the overhead of the bootstrap / sessions lock with each chunk
NK
12:42
Nadav Kavalerchik
In reply to this message
Thank you for the information and advice. I will definitely make a PR, as soon as we patch the code.
At first, I am not planning on offloading everything as I am not sure what the performance will be, but I guess after we had some experience with it, we will probably consider it.
We use cloudfront as well, so I am going to use signed urls.
We use file-less backup on some Moodle systems, but for some it is still an issue (as Moodle admins insist on full backups)
I have made some tests with serving videos of S3 and it looks relatively fine, especially "streaming" them using progressive download (not via AWS Media services)
BH
12:47
Brendan Heywood (Moodle)
something else you might want to consider is https://github.com/catalyst/moodle-local_smartmedia which handles all the resizing and re-encoding to give the end user a more optimal version. This one doesn't have widespread usage yet, and has a lot more room to flesh out it feature set
AS
21:49
Alistair Spark
In reply to this message
There is the file extension whitelist
So if you only enable video file formats, only videos would be offloaded to S3
MJ
22:22
Mark Johnson
In reply to this message
I dont *think* that's correct. I believe the whitelist just controls whether it uses pre-signed URLs for that file type
AS
22:23
Alistair Spark
Yeah you’re right
22:25
It was one of multiple setting s which needed enabling for the signed urls which wasn’t obvious.
iirc the size threshold setting was also about the pre signed urls rather than the shifting.
MJ
22:25
Mark Johnson
Yeah I was thrown by it too
AS
22:25
Alistair Spark
we’ve held off enabling pre-signed as we uncovered a bug with scorm packages
BH
23:00
Brendan Heywood (Moodle)
from memory we added the file extension whitelist specifically to work around the scorm issue. eventually I'd like to see this all rewritten so the urls don't change and all the logic is moved into a cloudfront lambda
DM
23:09
Dan Marsden
...and I'd like to see that term "whitelist" removed... I've created a tracker... 😊
22 March 2021
AS
01:00
Alistair Spark
I’d be even better if it was an exclusion list. I assume for fixing scorm we’d want all except html & js files or something
Denny Hermawan invited Denny Hermawan
NK
12:00
Nadav Kavalerchik
Thank you @aspark21 & @marxjohnson for the information, I am checking those options.
MJ
12:41
Mark Johnson
Speaking of objectfs, would someone from Catalyst mind taking a look at https://github.com/catalyst/moodle-tool_objectfs/issues/409? We're currently having to use a slightly frankensteined version to use it with Moodle 3.10 :)
Parangoma Bhattacharjee invited Parangoma Bhattacharjee
23 March 2021
Chitra P invited Chitra P
CP
05:35
Chitra P
We are hiring for senior Moodle developer for Pune location. Interested candidates can share their CV on chitra.p@paradisosolutions.com
24 March 2021
Giovanni Desogus invited Giovanni Desogus
Ashraf Nawaz Qureshi invited Ashraf Nawaz Qureshi
Ashraf Nawaz Qureshi invited Ashraf Nawaz Qureshi
25 March 2021
BH
01:56
Brendan Heywood (Moodle)
In reply to this message
done, not sure that branch should exist at all
LC
17:57
Luke Carr
Not really a support issue, but just interested to hear any stories of infra outside of Moodle and PHP that people are currently using. For example, my Moodle installation (for a school) is integrated with a serverless microservices framework (Go and Rust) to synchronise with our MIS. Students subjects and classes (and subsequent cohorts and enrolments) are pushed into Moodle, and grade book data and access logs are extracted automatically (using webhooks).
MB
18:24
Martin Božič
I have a question about the file picker and "server files" feature: did anyone begin to have timeouts at a certain moodledata size/scale? We're having trouble with certain users unable to browse their files via "server files". We're guessing that the sheer size of the mdl_files table takes too long to list everything and hits the CF/Apache/php-fpm time limit. Just making those timeouts longer doesn't seem to do the trick.
NK
18:34
Nadav Kavalerchik
In reply to this message
We use blackfire.io to profile specific pages on specific times when we experience timeouts or long page delays.
NK
18:59
Nadav Kavalerchik
In reply to this message
We are running different Moodle systems for different communities.
One of our Moodle systems is controlled by the campus SIS that is connected to Moodle web services via an ESB proxy that is controlling all of the creation and update of courses and users. (we use the beautiful https://moodle.org/plugins/webservice_restful )
Another Moodle systems we run is using the Ministry of education IDM to authenticate users via SAML2 (we use the featureful https://moodle.org/plugins/auth_saml2) as our MOE is managing a single point of authentication to all the K12 teachers and students for all the LMS providers and content providers in the country via this IDM. About 1.7M users. but only small fraction of them uses our systems as we are serving STEM content and LMS services for K9-12.

And BIG thank you! to the Catalyst team for those excellent plugins.
MB
19:58
Martin Božič
In reply to this message
Ha, good point. I'll have to check with our New Relic instance.
26 March 2021
Kuking Ñahui invited Kuking Ñahui
31 March 2021
Hughes Dimka invited Hughes Dimka
1 April 2021
VS
22:50
Vasileios Sotiras
In reply to this message
are you in the moodle root folder when running the script?
P
23:03
Panos
In reply to this message
hm, I cannot remember probably not, I resolved the issue by running the web command, not ideal tho.
VS
23:04
Vasileios Sotiras
In reply to this message
remember class auto-loading depends on the root folder of the application, and the web interface is always based on the root folder of moodle :)
P
23:08
Panos
In reply to this message
thanks, I will check that, the older version of moodle cron used to run without problems
2 April 2021
TM
13:22
Tomek Muras
I think that running cron should work, no matter what's your current working directory
13:22
If it does not work for you - then double check it. If changing CWD is required in some edge cases, then we should update the documentation on https://docs.moodle.org/310/en/Cron
Satish Gupta invited Satish Gupta
SG
21:25
Satish Gupta
Hi, I am looking to deploy a highly available and scalable solution on Google cloud. I am using Moodle mainly as a backend with a custom frontend in ionic interacting with backend using webservices. Is there any documentation anywhere on how to deploy a scalable solution for this on Google Cloud for optimal performance? I found some material on AWS implementation on Internet but nothing for Google Cloud.
21:25
Any support will be highly appreciated.
3 April 2021
Kuking Ñahui invited Walter
AL
09:43
Avi Levy
In reply to this message
There is no documentation that I found, but we use GCP for our Moodle as a Service solution, We use GKE, with redis memory store, and filestore for shard NFS storage. Except human mistake 🤪 the product is solid, we are stand on 99.9% in the last 6 month
MT
09:44
Munir Tajudin
How much pod that it scale when let say few thousand of people access at the same time?
AL
09:51
Avi Levy
The question is not the pods, the nodes is the issue, it's depend on your autoscaler deffinitions but we already see the php pods scale to 100 pods, but we don't have yet few thousand RPS so it may be more. It somthing you should tune according to your requirmentes, the pods resource(requests and limitis) and the autoscalers, we work with custom metrics for our autoscales
MT
09:53
Munir Tajudin
my team use azure vm scale set instead of AKS in azure
AL
09:55
Avi Levy
We work with AKS, but GKE is better as managed k8s.
MT
09:55
Munir Tajudin
awesome
AL
09:55
Avi Levy
Sticker
Not included, change data exporting settings to download.
👍, 42.4 KB
SG
12:18
Satish Gupta
In reply to this message
Hi Avy, thanks for the response. are you using cloud sql or self managed instance for DB. and whats the size that you have allocated.
12:31
And how much cost are you bearing on monthly basis. or rather i should ask how much is the bare minimum you would pay if you had say only 100 users per day.
oh
16:22
omer hameiri
In reply to this message
What moodle version are you using for MOE?
NK
16:45
Nadav Kavalerchik
In reply to this message
3.8
oh
16:46
omer hameiri
Planning any upgrade soon? See any reason to?
NK
16:48
Nadav Kavalerchik
Lot's of customizations. The budget is invested in developing new features for K6-K10 subject matter teacher requests, that seems more important than upgrading.
Maybe next year.
JC
19:36
Job Céspedes
We use GKE and OpenShift for development of an Moodle Operator. GKE has been easy to manage. I have not seem any particular documentation from them about Moodle. There is now Autopilot mode for new clusters. Have not use it but seems less to manage, with some tradeoffs.
19:39
By the way, @Avimoto I came across SysBind's prometheus moodle exporter. It seems very useful.
AL
20:00
Avi Levy
In reply to this message
We use self managed postgresql cluster on k8s
20:07
In reply to this message
I really don't to answer your question right know, but it's depend on your sla requirments. We optimized our coast by using minimum nodes pool with persistent machin for statful services, and all the rest are preamtable machined ~33% of the coast from regular machins
20:09
In reply to this message
Thanks, feel free to PR, if you use it and make some changes
SG
20:09
Satish Gupta
Got it. Thanks Avi
SG
21:58
Satish Gupta
@Avimoto by any chance could you share the .yaml config file. Could be of great help for me to deploy resources
AL
22:04
Avi Levy
We don't have a one project that deploy just one moodle, We use it as Moodle SaaS service, its include a lot of FOSS projects, I'll try to make a sample of moodle on k8s deployment example and share it.
SG
22:04
Satish Gupta
That would be very helpful. thnks.
4 April 2021
HD
18:09
Hughes Dimka
@Avimoto ,that would be great,looking forward to it
SG
18:10
Satish Gupta
yes
18:11
can anyone provide me with steps of NFS creation for moodledata with configuration of same with webserver
18:12
I ll be using Google filestore
AL
19:05
Avi Levy
In reply to this message
You can use the nfs provision project, but for moodledata I'm not recommended using it (it's not HA)
https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner
SG
19:45
Satish Gupta
@Avimoto then how should i store moodledata
19:46
/filedir is of importance here I suppose? for cache (session and Application) I am thinking is using Redis
AL
20:23
Avi Levy
Yes, for shared cache and session we use Redis, all other right now store on nfs with filestore
SG
20:23
Satish Gupta
but you said you wouldn't recommend filestore
20:23
>
20:23
?
AL
20:25
Avi Levy
No, I say not recomended on nfs provisioner and nfs server for all moodledata dir
MŽ
20:25
Matej Žerovnik
In reply to this message
Can you post a link to the exporter?
SG
20:25
Satish Gupta
ok but for directories other than caches - should be fine right if I use Filestore?
AL
20:27
Avi Levy
Yes, Google provide filestore SLA, I recommended to read it before you use it
SG
20:27
Satish Gupta
yeah have read it. seems ok. and I suppose you are happy with it right?
AL
20:28
Avi Levy
Yes, do it work
SG
20:28
Satish Gupta
Great then it's done. Thank you so much for your kind advice @Avimoto
20:29
I intend to write a document on this when I have executed with decent performance
SG
20:30
Satish Gupta
will share in few weeks' time if everything goes well
6 April 2021
kdi invited kdi
Vojdan invited Vojdan
Nelson Moller invited Nelson Moller
7 April 2021
MB
10:09
Martin Božič
We're experiencing high error rate and response latency because of filled up max connections on MariaDB server. We don't think raising the limit would alleviate the problem. New Relic showed us that some large query on mdl_event table takes very long (about 40 seconds). Like we hit an invisible limit. Has anyone had such a problem?
TH
10:09
Tim Hunt
Have you got a recent Moodle version?
MB
10:10
Martin Božič
3.9.4
TH
10:10
Tim Hunt
Sadly, https://tracker.moodle.org/browse/MDL-68874 is 3.10
M
10:10
MoodleBot
MDL-68874 - New optional SQL debug mode which instruments SQL with the calling PHP code
Status: Closed - Fixed
Assignee: Brendan Heywood
Reporter: Brendan Heywood
Integrator: Jun Pataleta
Fix Versions: 3.10
Votes: 6
TH
10:11
Tim Hunt
Still, from the slow query log, can you see what the query is.
10:11
Normally, if you search the code from the right sub-string of that query, you can find it.
MB
10:15
Martin Božič
Well, here's the query: https://pastebin.com/5RzEC7aD
NK
10:16
Nadav Kavalerchik
There is also an option to turn ON slow SQL in the config.php . You can add logslow to the dboptions, see example:
$CFG->dboptions = array (
'dbpersist' => 0,
'dbport' => '',
'dbsocket' => '',
'logerrors' => '1', // mdl_log_queries
'logslow' => '5', // mdl_log_queries
//'logall' => '1', // mdl_log_queries
'dbcollation' => 'utf8mb4_unicode_ci'
);
10:17
All queries, longer the 5 seconds, will be logged into mdl_log_queries
TH
10:17
Tim Hunt
Nadav, most DB servers can do that for you.
NK
10:17
Nadav Kavalerchik
I use MariaDB
NK
10:22
Nadav Kavalerchik
In reply to this message
As far as I understand, these options are not related directly to a specific DB engine, and are calculated by Moodle
https://github.com/moodle/moodle/blob/master/lib/dml/moodle_database.php#L517
MŽ
10:24
Matej Žerovnik
Looking at the explain of costly queries on mdl_events table, we see different explain in prod and QA. It seems like prod is not using keys and is doing temporary tables
TH
10:24
Tim Hunt
No. Moodle seems to be duplicating what DBs can already do, for whatever reason.
MŽ
10:24
Matej Žerovnik
there is a bit different version of MariaDB (prod: 10.3.17 vs qa 10.3.27)
TH
10:24
Tim Hunt
Matej, Have you run Admin -> Development -> XMLDB -> Check indexes ?
MŽ
10:25
Matej Žerovnik
Tim: Martin did and indexes are OK. I also checked show create table to see if indexes are there and they are
TH
10:25
Tim Hunt
I would be very surprised in MariaDB version made a different (unless you happen to have hit a very specific bug, which would be very unlikely).
10:26
Does MariaDB has housekeeping that needs to be run to keep indexes in a good state (like VACUUM ALALYZE on Postgres)
MŽ
10:28
Matej Žerovnik
https://pastebin.com/qF2FTGPt
10:28
explain from prod and qa if it might be useful
MB
10:29
Martin Božič
Print mode is nicer: https://pastebin.com/print/qF2FTGPt
TH
10:32
Tim Hunt
What is the SQL for this?
MŽ
10:32
Matej Žerovnik
one moment
10:34
refresh for a little formating:)
TH
10:45
Tim Hunt
The only place in Moodle which has 'Min(ev.priority) AS priority' is in calendar/classes/local/event/strategies/raw_event_retrieval_strategy.php
10:46
in get_raw_events_legacy_implementation
10:46
I wonder why it is using 'legacy' code?
MŽ
10:47
Matej Žerovnik
yes
10:47
we are seeing these calls in request traces
TH
10:47
Tim Hunt
Oh! because get_raw_events jsut calls get_raw_events_legacy_implementation
MŽ
10:48
Matej Žerovnik
TH
10:48
Tim Hunt
All this code seems to come from MDL-58087
M
10:48
MoodleBot
MDL-58087 - Make sure event overrides (user, group) are obeyed on dashboard page
Status: Closed - Fixed
Assignee: cameron1729
Reporter: moodle.com
https://tracker.moodle.org/browse/MDL-58087
TH
10:49
Tim Hunt
That profiling info jsut confirms that all the slowness is this query - I see. You have confrimed which query it is.
10:49
Is your QA site a copy of the production one?
MŽ
10:51
Matej Žerovnik
Yes, we copy database every day during night
10:51
or maybe every other day
TH
10:51
Tim Hunt
The copy could well be rebuilding indexes? - which might help the query
10:52
I woudl say that you have enough iformation to open a tracker issue, and I don't think there is one currently open
MŽ
10:52
Matej Žerovnik
We are using mariabackup to copy the database, so there is no rebuild
10:52
but we will try runing optimize table during the night
TH
10:53
Tim Hunt
Can't hurt to try. Well be interesting to see if it makes a difference
MŽ
12:06
Matej Žerovnik
We will and will report back
MŽ
18:05
Matej Žerovnik
We ran optimize table, but the results are the same
18:05
we also downgraded QA mariadb version to the same as production, but the explain still returns different execution
18:05
and the difference in query execution time is almost 100x slower on production (and there is little load on prod right now)
18:07
Could it be an internal optimizer cache or something what will force this type of execution
TH
18:25
Tim Hunt
I don't know that much about MariaDB query optimizers, but in general sometimes query optimisers get it wrong. And, sometimes there are ways to re-write queries in a way that helps query optimisers get it right, but that is a bit of a magic art.
MŽ
18:25
Matej Žerovnik
true:)
18:33
We managed to simulate it in out QA env when we disabled derived_with_keys optimizer switch
18:34
but it is enabled on production. Turning it off and on doesn't help. Seems like we aren't the only ones: https://jira.mariadb.org/browse/MDEV-16487
18:35
Could be out table is broken, like the last post suggests.
8 April 2021
MŽ
00:35
Matej Žerovnik
It seems like we found the culprit. It was an index in mdl_event (mdl_even_modins_ix). This index was added here: https://github.com/NeillM/moodle/compare/77d1c415025f99f1c36dae67aac7af11b8a34b50...NeillM:MDL-66253. After removing the index, the performance skyrocketed (from 60qps to 6000qps). There might be a penalty when doing backups / deletions, based on commit message, but I think this is lesser evil. We will see during the night when backups start.
M
00:35
MoodleBot
MDL-66253 - Reduce the number of full table scans on the event table
Status: Closed - Fixed
Assignee: Neill Magill
Reporter: Neill Magill
Integrator: Eloy Lafuente (stronk7)
Fix Versions: 3.9
https://tracker.moodle.org/browse/MDL-66253
MŽ
00:41
Matej Žerovnik
Screenshot 2021-04-07 at 23.39.05.png
Not included, change data exporting settings to download.
65.4 KB
Before and after index was dropped
10 April 2021
Japhet Kamala invited Japhet Kamala
11 April 2021
🌿iherb🌿 invited 🌿iherb🌿
NK
12:34
Nadav Kavalerchik
Hello everyone 😊
I am pleased to announce that we are ready with our first webinar sharing experience of members in this group about running large scale Moodle systems.
Big thank you to everyone that accepted the invitation to present their experience with everyone else!

Here is the official agenda:
https://wiki.geant.org/display/NE/Managing+and+running+large+scale+Moodle+system%2C+for+IT
Please register to get the link to the online event!

There is still some room for one more speaker, so if you wish to share your experience... please contact me directly.
Nadav Kavalerchik (IUCC) & Gyöngyi Horváth GÉANT
12 April 2021
TH
12:25
Tim Hunt
What is an NREN? (I guess if I have to ask, I should not tick that box on the form.)
MŽ
12:25
Matej Žerovnik
national research and education network
A
16:58
Android
This seems very interesting.
However, 10AM is 4AM in EST Canada.
JC
18:58
Job Céspedes
In reply to this message
Yep. Very interesting. Thanks. I'll be watching the recording a few hours later.
NK
19:35
Nadav Kavalerchik
In reply to this message
We are going to record it, and publish it publicly
19:38
GÉANT is the network of networks (consortium) of all the European NRENs
13 April 2021
AS
01:02
Alistair Spark
In reply to this message
JISC
TH
10:09
Tim Hunt
Ah! I see. JISC is the UK's NREN. So, I answered that question wrong, because I did not understand the jargon. Oh well.
14 April 2021
FD
07:26
F Devine
looking forward to the recordings
Mahdi A. invited Mahdi A.
15 April 2021
larinyo invited larinyo
16 April 2021
Alvaro Patón invited Alvaro Patón
NK
14:57
Nadav Kavalerchik
Quick update about the online webinar coming up next week on th 20/4 ...
Two more presentation added to the agenda, and some fine tuning regarding the timing.
https://wiki.geant.org/display/NE/Managing+and+running+large+scale+Moodle+system%2C+for+IT
Looking forward to see you all 😊
גלית מרדכי ‎ invited גלית מרדכי ‎
19 April 2021
Rafa invited Rafa
Kiril Gold קיריל invited Kiril Gold קיריל
20 April 2021
Noam G. Nudelman invited Noam G. Nudelman
Eyal איל Ben-Hur invited Eyal איל Ben-Hur
Smz invited Smz
Juha Martikainen invited Juha Martikainen
pbdco invited pbdco
Roman invited Roman
21 April 2021
IC
23:32
Iban Cardona
In reply to this message
Hi Nadav. That was good! Will you upload the session and the presentations?
22 April 2021
NK
10:17
Nadav Kavalerchik
In reply to this message
Thank you Iban , I will probably finish editing the video recording on Friday, and upload it to the event wiki page, and post a link to it here, and also try to get the presentations from all the speakers too.
23 April 2021
Àngel invited Àngel
27 April 2021
Aom P. invited Aom P.
l
22:01
larinyo
In reply to this message
👍
28 April 2021
Miki Alliel מיקי invited Miki Alliel מיקי
29 April 2021
HD
08:56
Hughes Dimka
Hello Everyone anydate on the presentations
NK
09:15
Nadav Kavalerchik
In reply to this message
I am actually waiting for only one more to send an update
09:15
But you can all enjoy the recordings and presentations that are already ready
https://wiki.geant.org/display/NE/Managing+and+running+large+scale+Moodle+system%2C+for+IT
Majid Khoram invited Majid Khoram
MR
15:41
Michael Ruder Rapajic
In reply to this message
👍
2 May 2021
Ferhat invited Ferhat
3 May 2021
NK
15:17
Nadav Kavalerchik
https://pages.awscloud.com/EMEA-public-OE-AWS-on-Moodle-OCRE-2021-reg-event.html?
Might be interesting to anyone considering a move to AWS
TH
17:20
Tim Hunt
Be careful with that AWS webinar. I went to one like that. The most important part of the announcement to note is

"Who should attend?
University leaders, CIOs, IT LMS leaders, and academic LMS owners."
17:20
Expect management waffle, not interesting technical details.
YK
17:30
Yedidia Klein
I agree, went once .. a lot of PR and sales nothing techie...
NK
19:01
Nadav Kavalerchik
In reply to this message
I agree, I was also disappointed a few times with AWS webinars.
Let's hope they improve over time.
YK
19:02
Yedidia Klein
I don't think that they will approve... They are working on management level and not on hands on level...
6 May 2021
Siddharth Bhosale invited Siddharth Bhosale
Pedro M invited Pedro M
7 May 2021
Alisher Davlatov invited Alisher Davlatov
Milos Bajcetic invited Milos Bajcetic
10 May 2021
SB
08:17
Siddharth Bhosale
Moodle version: - 3.9.1+ Theme:- Fordson Theme version:- Moodle 3.9 Fordson v3.9 release 1.1 (2020061800)

I had a requirement to design a system for arranging missed courses/sessions for the absent students.

The requirement states that the absent student should send a request to the admin stating the reason for his absence for the course/session and then admin would go through the request and approve or deny the request based on the reason specified by the absent student in the request

How can I design such a system? Are there inbuilt features provided by Moodle for the absent students or are there any plugins or external features to be added to achieve this functionality? How do I approach to this problem?
BH
09:24
Brendan Heywood (Moodle)
Not really a question about large scale, but you probably want something like this https://moodle.org/plugins/enrol_apply
SB
10:03
Siddharth Bhosale
In reply to this message
Ok will go through it.thank you
11 May 2021
MŽ
13:06
Matej Žerovnik
Hey guys... We have a quiz that users can take multiple times. Lately, some users have a lot of attempts and it's killing our performance as queries take up to 2-3min to execute. It seems like MySQL is using the wrong index (if I compare working and non-working cases). If I force the usage of the right index, execution time goes to a few ms.
13:06
anyone came across this?
13:11
Problem is in `get_questions_from_categories_and_tags_with_usage_counts` function.
MK
13:12
Majid Khoram
I had kind of same problem, too many requests (too many users attempting the quiz) and the CPU usage of the server goes high.

I changed some PHP-FPM Parameters and increased CPU Cores and some RAM.

also tried to make a better timing for users to attempt the quiz in order to spread the load
MŽ
13:14
Matej Žerovnik
Yea, but this will be hard for us to scale-out as a few tried will kill our DB server as each query takes a few minutes to finish. Users are tired of waiting and are pressing F5, spawning more processes and killing the DB server
13:22
Will do some more debugging
MK
13:36
Majid Khoram
I ain't got any idea on how good or bad the DB and how the server resources are performing when some high load and pressure are on the server.

Anyone has ideas?
MŽ
13:45
Matej Žerovnik
zou need some kind of monitoring, such as datadog, new relic, prometheus or whatevery you use
13:45
we are using prometheus + new relic for APM
MK
13:50
Majid Khoram
I do have monitoring on the server, but I need to come up with a kind of calculation on how much RAM/CPU resource can handle how many users (or requests) per second
NK
13:54
Nadav Kavalerchik
We use Blackfire.io on production servers, that helps identify bottlenecks. RAM / CPU / NETWORK / SQL / CACHE
13:54
MŽ
13:55
Matej Žerovnik
In reply to this message
That is not easy as it really depends on do many variables... the easiest way is probably with some jmeter tests where you ramp up users and look at latencies
MK
14:00
Majid Khoram
In reply to this message
thanks
I'll give it a try
14:00
In reply to this message
sure
MŽ
14:32
Matej Žerovnik
Our issue: https://tracker.moodle.org/browse/MDL-70635. We added some code as a quick fix until we find a bettter solution.
M
14:32
MoodleBot
MDL-70635 - Retaking a quiz takes too long (db hangs)
Status: Open
Reporter: Matej Konobelj
TH
14:36
Tim Hunt
I assume you use MySQL. If you use a database that does not suck, surely the query optimiser would konw to use that index?
14:38
Anyway, the suggested fix is clearly a hack. Extremely unlikely to be accepted into Moodle core. But, thanks for sharing.
MŽ
14:40
Matej Žerovnik
Yes, we are using MySQL (MariaDB in this case). We are not expecting the fix to be accepted to Mooodle core as it's really just a workaround for us to get back on the horse, but we are searching futher what query optimized is picking wrong index.
TH
14:40
Tim Hunt
Very interesting to konw why MySQL query optiomiser cannot get that right.
MŽ
14:41
Matej Žerovnik
as are we, as we have been hit with this issue on 2 moodle instances now, once when searching mdl_events and once in quizes.
TH
14:44
Tim Hunt
we really need to see the SQL of the whole query, not just that fragement.
MŽ
14:45
Matej Žerovnik
The whole query is in the first post. Or do you need something more?
TH
14:46
Tim Hunt
(To be honese, I am multi-tasking while at the UK Moodle Moot) I should probably stop, I am not doing etiher well.
14:46
)
MŽ
14:46
Matej Žerovnik
🙂
TH
14:47
Tim Hunt
I think this is from question_bank::get_questions_from_categories_and_tags_with_usage_counts
MŽ
14:48
Matej Žerovnik
it is yes
TH
14:48
Tim Hunt
Oh, and the query is using a lot of subqueries. That ofen causes issues.
14:48
Normally, the 'good' fix is if you can work out how to turn it into a flat JOIN
14:51
There are some unit tests in test_get_questions_from_categories_with_usage_counts, which makes it somewhat save to try to change this.
Gavin Henrick invited Gavin Henrick
Martin Dougiamas invited Martin Dougiamas
Sam Binon invited Sam Binon
Chris Gibbons invited Chris Gibbons
Heiko Schach invited Heiko Schach
Sarah Cotton invited Sarah Cotton
Galin Vassilev invited Galin Vassilev
Shawn McGirr invited Shawn McGirr
Liz Carlin invited Liz Carlin
Roy Kerley invited Roy Kerley
Martin Greenaway invited Martin Greenaway
Mike Nodding invited Mike Nodding
12 May 2021
Simon L invited Simon L
14 May 2021
Rainbow invited Rainbow
R
13:23
Rainbow
Hello every one
Good day to all

I am new to this channel also to moodle and planning an installation for 15k+ users

Some one can advice me the best server configuration

Expecting 300+simultaneous users and lot of h5p interactive video contents


Thanks in advance
13:24
Planning to use a vps hosting with WHM and cpanel with maria DB
PM
13:44
Pedro M
Hi, you can get some good tips here, https://docs.moodle.org/310/en/Large_installations
MK
13:45
Majid Khoram
I think in your case:

Try not to use MariaDB as there are better Databases. (according to moodle documentations PostgreSQL has much better performance)

Try separating App and DB Servers, each with backups and replications.

you need a kind of load-balancer like another separatevery low-resourced VPS to handle the HAProxy.

DNS load balancing can also be an option (which I don't recommend)
AS
15:35
Alistair Spark
In reply to this message
I wouldn’t expect anyone to be using WHM and cPanel for an installation that size but maybe that’s just me
AL
17:00
Avi Levy
In reply to this message
One VPC WHM is not a production solution for high scale in several aspects like security, high availability and very hard to customize for special configuration, I also recommended to use postgresql database and not mariadb, in this link https://wiki.geant.org/display/NE/Managing+and+running+large+scale+Moodle+system%2C+for+IT you can find good alternatives solutions
MŽ
17:31
Matej Žerovnik
I can tell you about a case where mariadb shits the bed:) AFAIK Postgre works in the same case, but we are where we are
17:31
🙂
AS
18:15
Alistair Spark
In reply to this message
Yes watch the second talk 😉
15 May 2021
R
13:18
Rainbow
In reply to this message
Are you using whm and cpanel?😊
13:24
In reply to this message
What about the specs
At moment we have a capability of 8core cpu and 16gb ram and 500gb ssd storage

Is this ok?
MŽ
13:25
Matej Žerovnik
Best bet is to test it out
R
13:26
Rainbow
😊
MŽ
13:26
Matej Žerovnik
And there is a lot of optimization to do when you are setting it up
R
13:26
Rainbow
Can u pls share those ideas
MŽ
13:26
Matej Žerovnik
When we started we had 8 nodes, 40 cores each with 64gb memory and couldnt handle 200req/s
R
13:27
Rainbow
Oh really
MŽ
13:27
Matej Žerovnik
Now we handle 1000req/s with 1% load on phpfpm
R
13:27
Rainbow
Hows that
MŽ
13:28
Matej Žerovnik
And could run with single nod
R
13:28
Rainbow
Great
MŽ
13:28
Matej Žerovnik
Well it was a long way and cant be described in single post
13:28
But look at the links avi gave
13:28
There is a lot of best practices described
13:29
And also moodle article on large installations
13:29
Also: monitoring!!!
13:29
Without that you wont know whats happening
MK
13:29
Majid Khoram
In reply to this message
Try your bests😊😉
MŽ
13:30
Matej Žerovnik
In reallity its hard to say what works where. There are some stuff such as which db, nginx/apache with worker mpm, phpfpm with opcache
13:31
Redis (shared for shared resources, local for those who dont need shared resources)
13:32
I think we also fixed some upstream performace bugs in h5p and made PRs
13:32
But in the end it depends on the usage
13:32
How users are using the site, which modules and so on
R
13:34
Rainbow
If we keep the video files on another server, can we improve the performance?
MB
14:27
Martin Božič
In reply to this message
Videos are not so much a question about performance but of storage capacity. Bottlenecks mostly happen because of poorly tuned DB or caching.
BH
15:11
Brendan Heywood (Moodle)
large scale video will expose multiple bottle necks in your system such as file serving because each streamed chunk of video needs to bootstrap moodle and gain a session lock and other bits. It's better to offload video as much as you can, either not in moodle to start with or eg our objectfs file api plugin which offloads the video serving to s3 or cloudfront. But this is about step 27 along the journey and it sounds like you are at step 1 and would probably benefit from a moodle partner (eg Catalyst who I work for)
GH
15:58
Gavin Henrick
Vimeo is good for hosting the video on, or youtube , or stream.
R
16:35
Rafa
Hello
we have 7 web nodes and 7 db nodes ( phpfpm and Moodle 39)
lately and frequently, we have cpu load spikes due to video playback.
any ideas to improve this?
Thank you
MB
16:46
Martin Božič
In reply to this message
Why 7 db nodes??? MySQL or Postgres?
R
18:58
Rafa
MySQL
they are independent nodes
node1www <> node1db
node2www <> node2db
...
each pair of nodei (www and db) run several instances of Moodle
17 May 2021
NK
12:15
Nadav Kavalerchik
Better have video files on a dedicated video streaming solution, even small size video assignment files that are submitted by students add up to a very large course backup files, and they are not resized to stream well on small devices, and lockup Moodle web (NGINX) sessions. so better offload it from Moodle.
We use several different video solutions in various projects we run. All in all, we are very happy with each one of them in the context of their respective project.
(1) CE Kaltura NGINX module streamer + Wordpress frontend to manage videos on the file system and to help Moodle find and link videos into courses via wp-repository Moodle plugin
PROS: Free OS components
CONS: You have to manage it yourself
(2) Panopto
PROS: A very mature SAAS video streaming solution
CONS: Very expensive
(3) MEDIAL
PROS: A versatile entry level SAAS video solution that can start very very small and that can grow as you grow
CONS: basic video management backend, basic Moodle video permissions management.
12:23
In reply to this message
On one of the Moodle projects I am managing, which has double to triple the amount of concurrent user you are aiming for, we use only one Moodle web frontend (16MB RAM, 6 CPUs, NGINX+PHP-FPM) + one DB (MariaDB)
I do not think you need a cluster for this, as it one make your architecture more complicated.
MB
18:12
Martin Božič
In reply to this message
+1 I know of a similar site that's being handled with that kind of setup. Two beefed-up VMs and they've handled the lockdown pretty nicely.
18 May 2021
Deleted invited Deleted Account
MN
01:47
Mike Nodding
Same, to keep it simple, we have one app vm, one db vm. 20k students. 32cpu/64g, ssd, pgsql, varnish/nginx/apache. Handles 3k concurrent heavy use ok/ +- 1s avg page loads. Pg bouncer was a key factor in improving last year, plus optimising db generally. We do spin up a stripped down/ beefed up azure paas mdl env for "official" exams ie just quiz. Matomo is useful to see page loads and concurrency. Pg activity tool also good.
Roger invited Roger
CG
16:29
Chris Gibbons
In reply to this message
Have you had any success with option 1 - using Kaltura CE is certainly something we are considering. Would be helpful if we could find someone with experience in setting this up.
NK
17:20
Nadav Kavalerchik
In reply to this message
We are using it from 2016, and I hardly touched the code and the VOD server (touch wood!)
Here is a presentation about the setup: https://slides.com/nadavkavalerchik/deck-4-5#/1/2
You will need a Moodle WP VOD file picker repository plugin: https://github.com/nadavkav/moodle-repository_wpvod that search video files in the VOD server via WP restfull API calls, and a VOD video file player plugin:
https://github.com/nadavkav/moodle-mod_videofile
That includes some code modifications to play MPEG-DASH files that are streaming from the NGINX+ Kaltura streamer module
YK
18:31
Yedidia Klein
https://moodle.com/wp-content/events/mootglobal19/MoodleVideoDirectory.pdf

See this slideshow.. a video directory integrated inside moodle...
CG
18:32
Chris Gibbons
In reply to this message
Thanks that is really helpful
R
18:34
Rafa
In reply to this message
👍
19 May 2021
HJ
14:32
Hrishikesh J
MariaDB-Nginx | Performance Issues

Hello! Greetings!  

We are currently moving our Moodle Site to MariaDB and Nginx. We are using AWS infra, (C5 series). While performing tests we achieved the following results on a 4 CPU 8 GB Ram Server.  

Sampling Rate- 600/10 secs. (success) 

Following are the configs of MariaDB and Nginx

[mysqld] 

  

## Performance Specific ## 

########################## 

  

innodb_buffer_pool_size            = 12G  

innodb_file_per_table                   = 1 

innodb_flush_log_at_trx_commit  = 2 

innodb_io_capacity                        = 800 

read_buffer_size                              = 2M 

read_rnd_buffer_size                    = 8M 

sort_buffer_size                               = 8M 

table_open_cache                           = 4096 

thread_cache_size                          = 128 

thread_stack  = 256k 

tmp_table_size  = 64M 

innodb_flush_method                   = O_DIRECT 

innodb_log_file_size                      = 512M  

innodb_log_files_in_group  = 3 

innodb_read_io_threads                              = 8 

join_buffer_size                               = 2M 

max_heap_table_size                    = 64M 

query_cache_limit                           = 2M 

query_cache_min_res_unit         = 2048 

query_cache_size                            = 196M 

query_cache_type                          = 1 

table_open_cache_instances  = 8 

table-definition-cache                   = 4096 

key_buffer_size  = 16M 

back_log = 1500 

innodb_purge_threads                 = 2 

innodb_sort_buffer_size                              = 2M 

binlog_cache_size                           = 128K 

binlog_stmt_cache_size                = 1M 

max_allowed_packet                     = 32MB 

------------------------------------ 

Applied Nginx cnf changes 

worker_processes 4; 

worker_rlimit_nofile 100000; 

events{ 

    worker_connections 4000; 

    use epoll; 

    multi_accept on; 

} 

http{ 

open_file_cache max=200000 inactive=20s; 

    open_file_cache_valid 30s; 

    open_file_cache_min_uses 2; 

    open_file_cache_errors on; 

fastcgi_read_timeout 240; 

    reset_timedout_connection on; 

    client_body_timeout 10; 

    send_timeout 2;     

    keepalive_timeout 180;     

    keepalive_requests 100000; 

} 

 

 

 

 

 

PHP-FPM Changes :-  

/etc/php/7.4/fpm/pool.d/www.conf 

listen = /run/php/php7.4-fpm.sock  

------------------------------- 

/etc/nginx/sites-available/site.conf 

location ~ [^/]\.php(/|$) { 

        include snippets/fastcgi-php.conf; 

        fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; 

        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; 

        include fastcgi_params; 

    } 

} 

 -------------------------------------- 

Course Level activities :- 

predefined activities, 

skipped forum, 

interactive videos, 

interactive presentation,  

upload video,  

attempt video,  

scrom 

------------------------------------ 

Hard Disk Utilization- 2/30GB 

Size = 50GB 

Used = 30GB 

Avail = 20GB 

  

However, after this we moved onto increasing the hardware specs to 8 CPU and 16GB Ram, and tried testing for 600 users and 700 users, but it kept on failing. We tried changing settings but no improvement. Our max CPU usage for this server was 34% and ram was 3.2/16 GB.  

The instance kept on giving ‘Error-502 Bad Gateway’.  

What could be the possible issue in this case? 
https://moodle.org/mod/forum/discuss.php?d=422436
AS
16:29
Alistair Spark
Can you tell what was maxed out?
16:37
We always run htop on the actual VMs to understand the bottlenecks that monitoring cannot show
16:38
Seems like the logical things would be increase worker processes for nginx and disable query cache in mariadb
No name invited No name
MB
16:50
Martin Božič
Is php-fpm set to static or dynamic?
YK
16:51
Yedidia Klein
did you use mysqltuner.pl ?
Mosaab Haj Elsiddig invited Mosaab Haj Elsiddig
20 May 2021
HJ
07:17
Hrishikesh J
In reply to this message
No, while monitoring we didn't come across anything that maxed out as such. We would try increasing worker processes once to see effect.
07:17
In reply to this message
We have set it to dynamic
07:17
In reply to this message
No. We have used mysqltuner.
R
19:51
Rafa
In reply to this message
Better perfomance static
NK
20:03
Nadav Kavalerchik
In reply to this message
We also use static, and it is has better performance. You just have to monitor it to see that the numbers you use are still relevant, probably every few months.
GV
20:09
Galin Vassilev
just a thought. You could get the best of both worlds where you use dynamic, but start as many threads as you would under static but than allow yourself the room to grow and than monitor the logs for situations where additional threads were spawned. just my 2c
MB
20:36
Martin Božič
We tried dynamic, had similar problems, went back to static. It's just more predictable.
MŽ
22:20
Matej Žerovnik
We do have dedicated web nodes, so statically configuring the workers is easy and we can tune it to use all cores and not more than available
22:23
In reply to this message
I think you should search for the reason you are getting 502 bad gw. But I would check if there is anything in phofpm error logs. Or maybe there is too small max open files for phpfpm processes? Something is blocking nginx attempt to pass the request to the phpfpm
22:23
It could be all workers are used? You might have free cpu, but execution is slow due to iowait and you might exhaust phpfpm pool
22:24
Try to enable phpfpm monitoring to see pool usage. I dont know what you use for monitoring so I can recommend
21 May 2021
IC
11:12
Iban Cardona
Hi! Yesterday, we transformed agora.xtec.cat by changing the sessions of 2395 Moodle. We moved sessions from database to Redis. More info here: https://educaciodigital.cat/CHANGES.txt
24 May 2021
HJ
10:04
Hrishikesh J
In reply to this message
We tried static but since then during all our tests we got Database Overloaded Error.
25 May 2021
Deleted invited Deleted Account
2 June 2021
TH
18:52
Tim Hunt
Is there anyone who uses quizzes in big courses, who could peer reveiw https://tracker.moodle.org/browse/MDL-71838 ? Thanks.
M
18:52
MoodleBot
MDL-71838 - Quiz overview report runs out of memory with huge courses
Status: Waiting for peer review
Assignee: Tim Hunt
Reporter: Tim Hunt
3 June 2021
NK
14:58
Nadav Kavalerchik
Tried to create a public dashboard in the tracker for all issues that might be interesting and relevant for large scale Moodle systems, but... probably didn't have enough permissions for that (can anyone from Moodle HQ help?)
So for now, I am sharing a link to all open issues tagged with "performance" (which is a good start)
https://tracker.moodle.org/issues/?jql=project%20%3D%20MDL%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20Performance
Please see if any of them are interesting to follow, vote and even suggest a patch 😊
M
14:58
MoodleBot
MDL-71820 - Capability overview and comparison leads to timeout for large result set
Status: Open
Reporter: Joshua Ebarvia
BH
14:59
Brendan Heywood (Moodle)
that link is busted?
7 June 2021
Abhishek More invited Abhishek More
Moin Mulla invited Moin Mulla
8 June 2021
MJ
17:16
Mark Johnson
I've spent the last few days debugging an issue that might be of interest. We are using the newish Moodle feature to use a separate database connection for read-only queries, and also using AWS RDS proxy as a connection pool. We kept getting a lot of "SSL connection exception" on longer running scripts. Turns out that where the script was switching to one of the connections for several minutes (doing lots of writes) then switching back to the other (to do some reads), that was longer than the connection pool's idle connection timeout, and so it was dropping the read-only connection in between. Long story short: make sure your database idle connection timeouts are as long as your longest-running script!
9 June 2021
AS
00:21
Alistair Spark
You mean IdleClientTimeout
in
https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_DBProxy.html
?

The default is 30 minutes. What script do you have that takes that long? Something user facing?
00:26
Our main performance concern is currently assumed to be resolved by https://tracker.moodle.org/browse/MDL-55231
M
00:26
MoodleBot
MDL-55231 - Partial course cache rebuild
Status: Waiting for peer review
Assignee: Dongsheng Cai
Reporter: Mathieu Viossat
Votes: 10
AN
01:57
Andrew Nicols (Moodle)
In reply to this message
Some cron tasks may take longer than that
EL
01:58
Eloy Lafuente
(hopefully not executed over web, heh)
AN
01:59
Andrew Nicols (Moodle)
No, but the SSL connection exception was coming from the DB and the PG RDS proxy
02:00
i.e. open read + write connections, do some reads, do some writes for a long time (yay course backup?), finish course backup, go back to read. Read connection has now timed out and closed. proxy returns an SSL connection exception. Boom.
EL
02:01
Eloy Lafuente
whops, missread, you’re ok!
MJ
09:04
Mark Johnson
In reply to this message
We had it set lower than that, as it seemed sensible at the time. Now we know better!
SG
09:26
Satish Gupta
Hi guys, facing a problem with Assignment activity - Selected Activity option 'Student must submit this activity to complete it' . But even after submission the activity is not getting marked as complete
09:26
has anyone faced this issue before? and knows the fix?
GH
10:37
Gavin Henrick
is cron running?
SG
15:20
Satish Gupta
Photo
1280×131
Yes seems like so
10 June 2021
Julián Gómez invited Julián Gómez
JG
08:49
Julián Gómez
Hi there, i'm a novate developer, and i want migrate my site moodle to aws, any advice?, Do you know where I could start?. Thanks
NK
11:23
Nadav Kavalerchik
In reply to this message
Hello @Juliango15 and welcome 😊
We try to update all the insights we share in this channel into the official Moodle wiki page: https://docs.moodle.org/311/en/Large_installations
But, first please give us some details:
(1) How many users?
(2) How many online concurrent users?
(3) How many courses?
JG
15:46
Julián Gómez
Hello Nadav thanks for answering 😊 As I mentioned, the platform is already created, but to reduce costs I want to try AWS, (1) it currently has 70 registered users. (2) 10 users online concurrent 3) I have a total of 45 courses, again thanks for2 answering 🙌🏻😊
G
16:29
Gillian
At that scale you are unlikely to save money going to AWS.
NK
16:42
Nadav Kavalerchik
In reply to this message
Well... you have a very small system, and this channel is mainly dedicated to information running large scale Moodle systems. so I am not sure if the advice you can get here will help you.
Using AWS or any public cloud will not necessary reduce your costs as this scale. it might. but do not count on it.
AWS and other clouds can be attractive by there other aspects: large variety of services, managed services, high end server technologies, automation ...
16:44
You might like to try BitNami + AWS, as it can initiate a good instance very quickly. https://bitnami.com/stack/moodle/cloud
JG
16:51
Julián Gómez
ok I understand, thank you for your comment. I currently pay approximately 1500 dollars annually and I thought that maybe because of the small size of the site I could reduce this value, I will try with bitnami as you recommend. Thank you so much 👍🏻😊
GH
16:59
Gavin Henrick
are you hosting that by yourself? or having someone look after it for you ?
JG
17:02
Julián Gómez
I manage it, those 1500 is just server
MH
17:06
Mosaab Haj Elsiddig
In reply to this message
Have you checked this link ?

https://docs.moodle.org/311/en/Moodle_migration
AS
18:25
Alistair Spark
For that kind of load a DO droplet or Scaleway DEV-1S would do the job
11 June 2021
BH
05:46
Brendan Heywood (Moodle)
bitnami moodle has lots of issues I'd avoid it
JG
05:50
Julián Gómez
In reply to this message
What do you recommend me?
BH
05:54
Brendan Heywood (Moodle)
Umm honestly not sure, I work for a Moodle partner Catalyst IT and we do hosting and handle some very large sites, you can contact us with your requirements but we may not be the right fit for you - https://www.catalyst-au.net/contact-us . Just don't use bitnami's out of the box moodle as it has lots of known issues
14 June 2021
Deleted invited Deleted Account
15 June 2021
Adam Jenkins invited Adam Jenkins
Thom invited Thom
Marina invited Marina
NK
17:43
Nadav Kavalerchik
Case studies from Moodle Certified Premium Partner Catalyst IT
Offering fully managed e-learning solutions for higher education institutions and businesses, Moodle Certified Premium Partner Catalyst IT provide exceptional technical expertise and implementation of Moodle learning environments to organisations worldwide.

https://moodle.com/news/safeguarding-scaling-large-university-moodle-environments/
16 June 2021
MB
15:23
Martin Božič
I've discovered a big problem with my automated backups. About 25k of backups of courses seem to have been stuck with last successful backup before the 2nd lockdown (8 months ago). They seem to be stuck in "queued" state. Can I just manually set the timestamps in mdl_backup_courses to force a run before the schedule?
EL
18:55
Eloy Lafuente
+1 to ban idiots
HN
18:56
Huong Nguyen
I think @nadavkav is an admin
TH
18:56
Tim Hunt
I pinged him some time ago, but he seems tob e offline
EL
18:58
Eloy Lafuente
Maybe he can add some of us as admins so can delete/report/ban when he’s not around.
TH
18:59
Tim Hunt
In reply to this message
Sadly, Adnrew will be asleep now.
18:59
+1 for Eloy also being admin.
HN
18:59
Huong Nguyen
Not sure but I remember last time he plans to add Andrew as admin
EL
19:00
Eloy Lafuente
+1 for Tim here too :-)
TH
19:01
Tim Hunt
Oh! I wouldn't trust me 😉
EL
19:01
Eloy Lafuente
While you don’t spam us here about your Go parties, all ok! LOL
NK
21:01
Nadav Kavalerchik
Redundant messages...Cleaned 😊
21:03
Andrew is also admin, and if Tim likes... I would love to add him as admin too.
TH
22:21
Tim Hunt
OK. I don't mind.
22:21
Would be good to get someone from US Timezone too.
NK
22:28
Nadav Kavalerchik
Is anyone from a US timezone volunteers to be a co-admin?
17 June 2021
Niccolò invited Niccolò
P
14:17
Pablo Javier Etcheverry
I can help, Nadav. GMT-3
TH
17:44
Tim Hunt
Well, that gave me a chance to use my new admin powers!
GH
18:05
Gavin Henrick
go Tim :)
18 June 2021
BH
03:30
Brendan Heywood (Moodle)
fyi we have kicked off a project to build a ground up new sampling profiler admin plugin to replace tideways using excimer
AL
03:33
Avi Levy
Can you explain?
BH
03:37
Brendan Heywood (Moodle)
there are two types of profilers, continuous profilers which log 100% of the stacktrace of a given request and sampling profilers which just intermittently log a sample every N milliseconds
03:39
tideways is a continuous profiler and it really slows down a request. If you accidentally turn it on for all requests then you'll cripple your site. Our aim is to build a plugin which samples 100% of all pages with essentially no overhead, and then chooses to persist data around 'interesting' pages, basically slow one. So you just turn it on, and it will just sit and watch and gather data and over time it will give you a report of all the slow pages and what their bottle necks are with no intervention
03:40
its basically like a baby version of new relic or blackfire or similar but built into moodle which means it is aware of lots of moodle concepts so for instance it could report on web vs ajax vs webservice vs cli differently
AL
03:44
Avi Levy
Yes, I look on tideways.com right now. I'ts look like sentry and blackfire (this is the tools we use for that mission)
BH
03:45
Brendan Heywood (Moodle)
tideways means two things, it can mean their Saas offering which is like blackfire, or it can mean the open source profiling php library which moodle already uses. I was referring to the latter, but what I am building is more like the saas, but fully open source
AL
03:51
Avi Levy
Nice. I'll take a deep look on it later
AL
04:08
Avi Levy
@brendanheywood do You have new recommendations for moodledata filesystem? from MDL-70243. I'm still try to find the best way to map the dirs:
/cache
/filedir
/lang
/lock
/muc
/temp
/trashdir
M
04:08
MoodleBot
MDL-70243 - Add a file system performance summary into the footer and file IO debug mode
Status: Peer review in progress
Assignee: Brendan Heywood
Reporter: Brendan Heywood
Votes: 5
https://tracker.moodle.org/browse/MDL-70243
BH
04:09
Brendan Heywood (Moodle)
I put some thoughts into https://docs.moodle.org/311/en/Server_cluster
04:09
it is a little high level as it really depends on how big you need and what inf you have
04:10
if you have more specific questions I'll try and answer them in that wiki
AL
04:16
Avi Levy
We use GCP and Azure, on both we run Moodle on K8s. on GCP we use manged NFS and buckets for backups and on Azure We use the Azure Files but it make us a lot of problem withe temp files. So we build our own NFS server and use It for /tmp directories.
I'm still fill that this is not the best solutions.
last time we chat you told me that you use also S3 to /filedir and in the past it make the Moodle sites realy slow.
BH
04:17
Brendan Heywood (Moodle)
we use objectfs to manage filedir in s3 across our entire fleet
04:18
whats does 'really slow' mean and how did you set it up exactly?
AL
04:27
Avi Levy
It was before your plugin was supported in GCP, so We try to implemented the storage class. Do You use it on production site? I'll take a look on the https://docs.moodle.org/311/en/Server_cluster and try to implimant again the /filedir on objectfs.
BH
04:27
Brendan Heywood (Moodle)
we are all aws, but I know people in the wild are using GCP
04:28
the latency is slightly higher but its offset by shifting serving load to cloudfront and much cheaper costs. This one isn't a pure performance thing its balancing a few different dimensions, mostly price
04:29
I am not sure if there is an equivalent to the cloudfront serving in the GCP store
AL
04:29
Avi Levy
The NFS soultion is realy expensive vs the buckets.
04:29
cloudfront ?= CDN
BH
04:30
Brendan Heywood (Moodle)
yeah aws specific, big files like movies are not served by moodle, it just served a signed redirect to the cdn so it takes a ton of load off the frontends
AL
04:33
Avi Levy
In reply to this message
We use it for static files, but I never think to use it for Moodle
BH
04:35
Brendan Heywood (Moodle)
I have a feeling GCP attempts to be fully compatible with s3 but I'm not sure, so it may Just Work. If not this shouldn't be too hard, all the logic is there for cloudfront so it should be mostly a cut and paste / porting job, pull requests always welcome.
AL
04:36
Avi Levy
Thanks
MB
10:34
Martin Božič
Hi, a question on adhoc tasks: do the max concurrent tasks settings in GUI affect the adhoc "listener" tasks I set up as cronjobs on my servers? I have task_adhoc_concurrency_limit = 48, 1 VM for cronjobs and 8 adhoc task cronjobs running on each of 9 web nodes.
10:40
In reply to this message
Just found out in the adhoc log "concurrency limit reached".
10:40
In reply to this message
So it does affect the additional adhoc cronjobs.
AN
10:43
Andrew Nicols (Moodle)
Unless you tell it not to. As I recall it’s a cli argument
MB
10:44
Martin Božič
In reply to this message
🤦‍♂️ forgot about that one, thanks!
20 June 2021
Deleted invited Deleted Account
Kesara Weerathunga invited Kesara Weerathunga
21 June 2021
MB
14:47
Martin Božič
I'm playing with an idea to symlink $moodledata/temp/backup (on NFS) back to local disk. I'm trying to offload some of the IO off the NFS. AFAIK that shouldn't be a problem, since adhoc tasks run only locally and they don't actually need shared storage. Are there any potential problems with this solution?
MB
15:34
Martin Božič
In reply to this message
Well, now that I consulted a clean config-dist.php again, I see have been talking nonsense.
AN
15:35
Andrew Nicols (Moodle)
In reply to this message
Yeah. Don’t do that. Yup ya fine for backup but not restore
15:35
You can probably ( not 100% sure) get away with doing it on your cron servers
22 June 2021
JT
13:43
Jordan Tomkinson
we tried and this breaks Asynchronous course copy. backuptemp needs to be shared on cron + web servers.
TH
14:31
Tim Hunt
I folks. I just detected a pretty worrying problem with a file:

All the data in the database is consistent. There is a row in mdl_files saying this file exists, which relates to a particular quiz attempt which is also present and correct.

However, the file is missing from filedir in moodledata when you look up the conent hash.

Has anyone seen something like that before? This did happen at a time when we had a dodgy switch in our data centre. However, I would hope that Moodle only records the file in mdl_files after it has been saved to disc, The odd thing is that I don't see any errors logged from when the file was written.
BH
14:35
Brendan Heywood (Moodle)
its not uncommon, most migrations we do from other hosting has some base error rate of missing files, and I think there are very small race conditions that lead to this. Error rate of around 2 per million
TH
14:35
Tim Hunt
This was a user uploading a file.
BH
14:36
Brendan Heywood (Moodle)
I would assume that all the race conditions are at create time
TH
14:36
Tim Hunt
Yes, but not good that it fails silently (and hte file is a student's exam answers!)
BH
14:38
Brendan Heywood (Moodle)
we've internally talked about swapping the write order to disk and then db, but then you might end up with the reverse situation, it probably needs to be something like a file journal, eg write a temp record, write the file, then update the record.
14:44
the error rate is high enough for us to notice but low enough for us to have not chased it much yet
NK
15:05
Nadav Kavalerchik
First time a read about such a situation, and maybe it happened "silently" with no report from our users.
TH
15:21
Tim Hunt
Thanks Brenda. Perhaps write to DB with null (or other conventional) file hash. Then write file. Then write real hash. (Or add a new status column to make it really clear.) Then have a scheduled task to kick up a fuss if a file stays too long without the hash being supplied?
15:21
THough, I expect, like you, we probably won't get around to it.]
BH
15:23
Brendan Heywood (Moodle)
my working theory for at least some of them has been an internal timeout on large file uploads, or a very unluckily timed broken connection which killed the script. That would depend on apache vs nginx and caching layers
TH
15:40
Tim Hunt
OK, well I have done minimal effort, which is to create MDL-71990
M
15:40
MoodleBot
MDL-71990 - File API seems to be capable of losing files (very rarely)
Status: Open
Reporter: Tim Hunt
https://tracker.moodle.org/browse/MDL-71990
JT
15:41
Jordan Tomkinson
likewise we see missing files on almost all big sites
TH
15:42
Tim Hunt
Oh great! and I am the first person to raise it in the tracker. (Or, I suck and finding duplicates, which is also very possible.)
15:42
But, thanks for confirming.
JT
15:42
Jordan Tomkinson
i also see orphaned files in filedir that dont have a db record
15:42
its a crazy world
TH
15:43
Tim Hunt
Yes, but that way is less worrying!
BH
15:43
Brendan Heywood (Moodle)
its just a gray area, as it could be just inf failures as scale and not strictly a code bug
JT
15:43
Jordan Tomkinson
as brendan said, we see them often during importing new customers data, so its hard to say where the error occured
TH
15:44
Tim Hunt
Indeed. I think the thing that worries me most is that the problem happens when the user uploads the file, and Moodle things it has saved it - to error is thrown - even thoguh it has not.
JT
15:45
Jordan Tomkinson
if filedir is backed by something like NFS, gluster etc - those services could return the OK to moodle and then have an error comitting to disk themselves which moodle could never account for
TH
15:45
Tim Hunt
For me, the most important part of making the situation better would be to get it to throw an exception at that point, so the user sees an error, and knows to try agian.
15:45
Yes, I am pretty sure it was somethign like that Jordan.
15:45
In which case, I guess there is almost no answer.
JT
15:46
Jordan Tomkinson
it would be nice to have some sort of "scan for and remove missing files" cron job.. we have CLI's to do it
TH
15:46
Tim Hunt
Generically (outside the crappy world of PHP) is there some mechanism for doing error handling in that scenario?
BH
15:46
Brendan Heywood (Moodle)
we've toyed with exactly that, also also trawling the filedir and recalcing the hashes to see if they still match, but it can be super slow
JT
15:48
Jordan Tomkinson
one area in particular is user profile pictures is the most common i see
15:48
i have to reset them back to the default
TH
15:57
Tim Hunt
The code in lib/filestorage/file_system_filedir.php tries very hard to get this right - from a quick read.

E.g. it writes the file with a temp name, then renames it.

However, and Jordan says, not much you can do if NFS tells you it has done it, then does not - with exceptions for most things.

HOwever, the return value fo the rename is not checked. Throwing an exception if that returns false would be a good improvement, I guess.
BH
15:58
Brendan Heywood (Moodle)
+1
אS
16:00
אסף @ Sysbind
/connect_group@IFTTT
I
16:00
IFTTT
In reply to this message
This group was successfully connected to your IFTTT account. You can now use it with Telegram Applets on IFTTT.
AN
16:01
Andrew Nicols (Moodle)
In reply to this message
It really really really does. But in some file systems they will return before the file is actually written
TH
16:16
Tim Hunt
OK. MDL-71991 done, if anyone wants to peer review.
M
16:16
MoodleBot
MDL-71991 - file_system_filedir should check the return value of rename when writing files
Status: Waiting for peer review
Assignee: Tim Hunt
Reporter: Tim Hunt
https://tracker.moodle.org/browse/MDL-71991
BH
16:17
Brendan Heywood (Moodle)
swap you one? almost as small: https://tracker.moodle.org/browse/MDL-69498
M
16:17
MoodleBot
MDL-69498 - Break down the Manage site administrators by email domain and sort
Status: Development in progress
Assignee: Brendan Heywood
Reporter: Brendan Heywood
TH
16:17
Tim Hunt
I resisted the temptation to try to refactor all the dupilcation between add_file_from_path and add_file_from_string.
16:17
That one is not ready for PR
16:17
And, you konw what I shoudl be PRing 😉
BH
16:18
Brendan Heywood (Moodle)
yes :)
TH
16:28
Tim Hunt
By the way, I assume it is hopelessly naïve to think that this problem with go away we we move our infrastructure to AWS, right?
BH
16:29
Brendan Heywood (Moodle)
devil in the detail, if you believe the spin: Designed for 99.999999999% (11 9’s) of Durability
16:30
thats for s3
TH
16:31
Tim Hunt
But, Moodledata is EFS, or something (backed by objectfs)
BH
16:32
Brendan Heywood (Moodle)
ours is home grown gluster in aws, we use efs for some bits from time to time. They all seem to be imperfect in different ways
TH
16:34
Tim Hunt
Not surprising really. I remember years ago when Marin Langhoff introduced me to this concept: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/
BH
16:34
Brendan Heywood (Moodle)
yeah I miss Joel's blog, pure gold
TH
16:35
Tim Hunt
Right. The time has come: Showing 56 files with 3,779 additions and 115 deletions.
BH
16:35
Brendan Heywood (Moodle)
good luck
TH
16:35
Tim Hunt
And, that's only the first issue in the epic.
NK
17:22
Nadav Kavalerchik
MOOSH can help with files: https://moosh-online.com/commands/#file-check
23 June 2021
AN
01:19
Andrew Nicols (Moodle)
In reply to this message
It doesn’t have to be EFS. It’s very easy to make moodledata S3. And you will probably stop losing files in that way
G
02:32
Gillian
Are you referring to the Catalyst S3 filesystem plugin? I think with that you still need a small "rump" EFS or other shared volume for things that haven't been transferred to S3 yet?
BH
02:33
Brendan Heywood (Moodle)
yeah ours is a hybrid, but others have set it up so its pure s3. We'll eventually add a synchronous mode where it writes directly to s3
AN
03:39
Andrew Nicols (Moodle)
In reply to this message
I was referring to the one I wrote ;)
03:40
Mine wrote directly to S3 with a per request directory
03:40
I also wrote an FTP one for testing. It was comically slow
G
04:12
Gillian
Ah, which one is that? I'm only familiar with https://moodle.org/plugins/tool_objectfs
AN
04:16
Andrew Nicols (Moodle)
It’s used internally. I wrote it when I wrote the file system API
IC
11:04
Iban Cardona
In reply to this message
Great idea. Any plugin?
TH
11:10
Tim Hunt
Still not a good look for HQ to have written a useful plugin like Andrew's S3 one, and not shared it. (In my opinion.)
AN
11:11
Andrew Nicols (Moodle)
We have lots of things which are not public. Most things are, but some things just aren't appropriate to be.
TH
11:12
Tim Hunt
Sure, same with the OU. However, this S3 one sounds generally useful.
11:13
Still, we went down the Catalyst objectfs route, and don't really have capacity to change now.
G
14:37
Gillian
I'd also be interested in seeing this S3 plugin, if it does become possible to make it public. The Catalyst one is intriguing but it doesn't totally replace having a shared moodledata at all, so if there's an alternative that does - I'd be interested in it.
BH
14:37
Brendan Heywood (Moodle)
nothing can replace moodledata
AN
14:38
Andrew Nicols (Moodle)
You can never get away with having no+ shared files. You always need to share temp
14:38
always
14:38
So our S3 plugin does not need moodledata but it does need temp
BH
14:38
Brendan Heywood (Moodle)
snap :)
AN
14:38
Andrew Nicols (Moodle)
Which is, by default, in moodledata
14:39
The trick is to reduce the dependency upon temp.
14:40
We've been slowly getting rid of things from there but the main offenders are backup, and some things related to assign, and I think some external software (Typo3?)
14:40
We've just removed typo3 on master so that one is no more.
14:41
Backup is hard to remove without re-architecting it
14:41
It's complex
14:41
And it needs to store state between steps
14:42
IN theory it should be possible to do a CLI backup or restore on a single host like a cron server, so I'm surprised that @jtomkinson encountered issues when trying that
BH
14:43
Brendan Heywood (Moodle)
the way I see it objectfs writing to filedir and then later shuffling things off to the real object store is a feature not a liability, adding code to make it write immediate would be quite minor but we've never actually wanted it to. When you first upload a file all sorts of things typically happen to the files and its good to keep it warm a bit closer just for a little bit. Especially really small files where the tradeoffs aren't as good. Having lots of levers to tune is a good thing
AN
14:43
Andrew Nicols (Moodle)
^^
14:44
I agree with that. My implementation used a per-request directory so that most of those tasks would be completed. The file is only removed when the request finishes. For a frontend server that's usually sufficient time because you aren't performing those next tasks on the frontend.
G
14:47
Gillian
Yeah, I understand why objectfs is written the way it is.
BH
14:47
Brendan Heywood (Moodle)
the moodle fork whose name shall not be mentioned does something quite different, it treats object storage as canonical and selectively pulls down files the each front end similar to localcache where they can stay for a while
G
14:47
Gillian
What's the best way to see if the filedir got hit on a request? I don't think that's actually in the caches info in the footer?
BH
14:48
Brendan Heywood (Moodle)
https://tracker.moodle.org/browse/MDL-70243
M
14:48
MoodleBot
MDL-70243 - Add a file system performance summary into the footer and file IO debug mode
Status: Peer review in progress
Assignee: Brendan Heywood
Reporter: Brendan Heywood
Votes: 6
BH
14:49
Brendan Heywood (Moodle)
this is working, I just got stuck on some small unit test bugs which I couldn't track down, but its definitely workable enough to cherry pick into a non prod env and use it to track things down
AN
14:49
Andrew Nicols (Moodle)
In reply to this message
Moodle fork?
G
14:49
Gillian
Oh, perfect.
BH
14:49
Brendan Heywood (Moodle)
totara
AN
14:49
Andrew Nicols (Moodle)
Ah
BH
14:50
Brendan Heywood (Moodle)
objectfs is no longer compatible with their latest
14:52
I would really love some eyeballs on that tracker test bug and to help get it landed
G
14:56
Gillian
Interesting, I hadn't heard of $cfg->alternate_component_cache before. Is that new-ish, or different from localtempdir or a MUC category?
AN
14:57
Andrew Nicols (Moodle)
It's a replacement cache configurator for MUC
14:57
It's been around for a couple of years, but is not widely used or known about outside of Catalyst.
14:58
I assume oyu mean alternative_cache_factory_class
14:58
ba9fc5ed572 - MDL-41492
M
14:58
MoodleBot
MDL-41492 - Allow alternate MUC cache config class (eg allow setup in pure $CFG / config.php)
Status: Closed - Fixed
Assignee: Peter Burnett
Reporter: Jai Gupta
Integrator: Jun Pataleta
Fix Versions: 3.10
Votes: 14
https://tracker.moodle.org/browse/MDL-41492
G
14:59
Gillian
From the testing instructions on 70243:

20) In the footer you should see that datadir is red and this is due to the component cache, lets move that to somewhere local:

$CFG->alternative_component_cache = '/tmp/component';
BH
14:59
Brendan Heywood (Moodle)
```
lib/classes/component.php:238: if (!empty($CFG->alternative_component_cache)) {
lib/classes/component.php-239- // Hack for heavily clustered sites that want to manage component cache invalidation manually.
lib/classes/component.php:240: $cachefile = $CFG->alternative_component_cache;
AN
15:00
Andrew Nicols (Moodle)
d7245e34002 MDL-40475
M
15:00
MoodleBot
MDL-40475 - alternative component cache location and other cleanup
Status: Closed - Fixed
Assignee: Petr Skoda
Reporter: Petr Skoda
Integrator: Damyon Wiese
Fix Versions: 2.6
https://tracker.moodle.org/browse/MDL-40475
AN
15:00
Andrew Nicols (Moodle)
8 years.
BH
15:00
Brendan Heywood (Moodle)
we do use this feature, it is tangential to everything above and it has a real impact
AN
15:01
Andrew Nicols (Moodle)
I forget about that one. It's very set and forget
SL
15:01
Simon L
We use that too, and defo has a real (positive) impact 👍
G
15:02
Gillian
Good to know. I wonder if it should be added to the cluster docs somewhere.. not sure where exactly
BH
15:03
Brendan Heywood (Moodle)
adding it now
G
15:03
Gillian
It sounds like it could safely use localcachedir? Okay to set it to that?
15:03
Ah, thanks
BH
15:09
Brendan Heywood (Moodle)
mostly a cut and paste job but this is where I'd expect it to be
TH
15:10
Tim Hunt
Brendan, the issue you mention seems to be 'In peer review' but with no per revier's name given. that is a bit confusing.
BH
15:11
Brendan Heywood (Moodle)
ah that was Peter, fixed. I wish the tracker auto added your name when you click the start peer review button
AN
15:11
Andrew Nicols (Moodle)
Also means it will not show In the peer review lists
BH
15:11
Brendan Heywood (Moodle)
where do I put tracker trackers?
AN
15:12
Andrew Nicols (Moodle)
MDLSITE
G
15:15
Gillian
Thanks for the link - I'd missed that before. I wonder if it would be good to even have all the weird cluster stuff in config.dist? Maybe in a "you don't need this unless you're running a cluster" section?
BH
15:17
Brendan Heywood (Moodle)
so much of setting up a cluster is much bigger than config.php though, but that particular one should be there and no idea why it isn't
G
16:16
Gillian
Also, that doc says to use a transparent caching proxy, but I get 100% cache misses on pages - is that outdated advice or is there something else that needs to be done for Cloudflare type caching to be useful? Moodle caching has improved since that doc was written, as well..
Karen Holland invited Karen Holland
KW
16:27
Kesara Weerathunga
Hi all
Anybody done large scale moodle deployments on k8s?
Just that we are exploring the possibility of integrating video conferencing as well
NK
16:31
Nadav Kavalerchik
In reply to this message
For large scale video conferencing (that is not Zoom 😉) you better try BBB with https://gitlab.com/bbbatscale/bbbatscale
More info: https://bigbluebutton.org/event-page/
BH
17:06
Brendan Heywood (Moodle)
In reply to this message
this should work for pure anon files, things like public css and images
G
17:12
Gillian
In reply to this message
Seems there are almost no files like this, though? I'm seeing 100% cache misses
BH
17:14
Brendan Heywood (Moodle)
$ curl -Is https://moodle.org/theme/image.php/moodleorg/theme_moodleorg/1623157737/moodle_logo_small | grep cache
cache-control: public, max-age=7776000, no-transform, immutable
cf-cache-status: HIT
17:14
first header shows what moodle sends out, second header shows that cloudflare cached it
G
17:30
Gillian
Ah I got them confused again, but still.. CloudFront shouldn't see a 100% cache miss rate either, right?
BH
17:31
Brendan Heywood (Moodle)
no, the cdn / cache / varnish takes a fairly decent fraction of traffic away from your origin server
17:31
double digit %
YK
17:31
Yedidia Klein
Hi, just reading all messages here... is it possible to use $CFG->alternative_component_cache = '/tmp/component'; mapped to a local directory on each node ? What is the impact on performance if most of my cache is on shared redis ?
BH
17:32
Brendan Heywood (Moodle)
In reply to this message
yes, and this has no relation to redis / muc
YK
17:32
Yedidia Klein
In reply to this message
great news...
BH
17:33
Brendan Heywood (Moodle)
but you need to read that wiki and make sure you update this file on upgrade on each front end
YK
17:36
Yedidia Klein
In reply to this message
got it... this rebuild... php admin/cli/alternative_component_cache.php --rebuild
17:39
anyone with experience on performance of s3 on aws or blob on azure VS nfs ? (efs/netapp) (I understand that from 3.9 catalyst s3fs do not need any patch .. right ?)
BH
17:41
Brendan Heywood (Moodle)
correct no patches for objectfs in 3.9
KW
18:53
Kesara Weerathunga
In reply to this message
Thanks will try and see. We are just trying to reach large set of students as a CSR project
If anybody done large scale deployement on kubernetes, just give me your 2 cents
NK
19:09
Nadav Kavalerchik
In reply to this message
@pablo_javier & @Avimoto have experience with K8s and large scale, and maybe others too.
Just wait a little bit to see if they pickup your comment
MŽ
19:18
Matej Žerovnik
In reply to this message
We are seeing 15% cache hit.
19:19
In reply to this message
Actually, its between 18-25 when looking at larger intervals. But looking at graphs, avg is around 20
AL
19:21
Avi Levy
In reply to this message
We run Moodle on K8S as SaaS on also as PaaS(bring your own moodle code) and each one has different implementation on K8s
24 June 2021
Kevin invited Kevin
25 June 2021
Omo Oaiya invited Omo Oaiya
NK
14:28
Nadav Kavalerchik
In reply to this message
Are you referring to this: https://github.com/catalyst/moodle-tool_excimer ?
BH
16:54
Brendan Heywood (Moodle)
yes, very early days
16:54
K
17:34
Kevin
hello guys to receive the emails of users who have signed up what should be activated?
EL
18:03
Eloy Lafuente
Sorry Kevin, these chats (I’ve seen you’ve posted on various) are not the place for support.

Please take a look to the community forums, search and look there.

https://moodle.org/course/ is a good start .

Ciao 🙂
NK
23:10
Nadav Kavalerchik
In reply to this message
Super cool 😊