Aggregate ZIM traffic statistics

[Popolechien]

It is currently very difficult to know how much a given ZIM file has been downloaded. Our only tool is Matomo, and it will distinguish between flavours and dates so that wikipedia_en_all_maxi_2026-02' is different from wikipedia_en_all_maxi_2026-04' and `wikipedia_en_all_nopic_2026-02' and all three need to be search separately.

This prevents us from sharing useful data with partners who gave us bespoke permission to share their content.

I suspect this would be better integrated with the CMS, but putting it here for the time being.

[kelson42]

We will agregate ZIM download statistics by title for the publishers (visible in the CMS).

Maybe it would be good as well to get this number per collection.

We need to define more precisaly how and what exactly we should do.

I will transfer the issue accordingly.

[benoit74]

I propose that we store in the CMS number of ZIM downloads per title flavour and day.

This will allow to filter and aggregate by these three dimensions and their own aggregates like title (ignoring the flavour), collection, week, month, year, ...

This means we will loose details about which exact files users have downloaded but I think it is not relevant.

In a first version, we can store number of raw ZIM file download and number of torrent file download. Maybe more interesting things could be stored as well.

Then comes the question of what and how we display this information in the UI. And honestly I'm quite clueless here. Would it make sense to have a "Stats" page in the CMS where one can adjust what stats he wanna see (which aggregation, which filters)?

[Popolechien]

Yeah pretty much a time-based stats graph, so we know how many downloads for a given time period.

[kelson42]

I propose that we store in the CMS number of ZIM downloads per title flavour and day.

That sounds good, but I have a few concerns I want to express about:

• The scalability of the whole setup (might be quite a bit of data over time)
• we need here to be careful not reinventing the wheel regarding data manipulation AND visualisation

At this stage it seems we will need:

• Simple number, for example to just tell in the catalog how popular is a book
• Minimal visualisation/chart in the CMS for the publishers

I suspect that if we are moving in the right direction (fulfilling a real need), this set of features will expand over time and I would recommend to be prepared to that.

@benoit74 What are your thoughs about the points above?

[benoit74]

The scalability of the whole setup (might be quite a bit of data over time)

ATM we have 3735 ; let's say we are very popular and end-up with 50k title flavours at some point (more than x10). Let's say that we keep 5k days of data (about 13.6 years). This means 250M integers. We should use a 4 bytes integer so this is not more than 1GB of data per measurement. Today we expect 2 measurements (ZIM download + torrent download) so 2GB. Not negligible, but not a problem either.

The core problem could be the 250M rows in a single table if we have a naive table schema. Which would also impose additional storage overhead. I suggest we rather use title_id, flavour, year, zim_downloads, torrent_downloads as table schema, where xxx_downloads is an integer[], i.e. a list of 365 (or 366) values. This reduces the table size to less than 1M rows which is very practical. It allows to easily purge old data if we want to. Downside is that every day we need to update the whole table again. And we need to be cautious to pad the list with zeros should miss some days (i.e. do not expect we can simply append data at the end of the array. Probably worth to store the whole list of 365 (or 366) values every day, starting 1st January.

we need here to be careful not reinventing the wheel regarding data manipulation AND visualisation

I don't know what this means. Since needs are very modest for now, I don't see where adding an off-the-shelf tool will help. It will need consequent work to integrate and maintain, for a very limited set of real needs. Do you have something specific in mind that we should use / avoid?

What we wanna store is indeed time series, but I'm not sure moving to a TimescaleDB or equivalent approach is really useful at this stage.

Simple number, for example to just tell in the catalog how popular is a book

Which number? Very open to suggestions if you have a precise, but at this stage I'm afraid this requirement would open the door to many more questions (and delays): over which period of time we compute the number? how we handle titles which have been just very recently added to the catalog? etc ... I feel like being able to store and request "raw" data is a first step forward, and it opens the door to extracting that data and trying to build some strategy about how we should compute that number and exercise it on real data.

Minimal visualisation/chart in the CMS for the publishers

I suggest that we start with:

• ability to filter by date range, by collection, title name and flavour
• display a graph with total number of ZIM downloads and torrent downloads on selected filters
• display 2 tables of sorted results: one without aggregation (sorted list of most popular title + flavour) and one aggregated at title level (merge all flavours, so a sorted list of most popular titles) ; allow to sort tables by ZIM download, torrent download, or some of both criteria.