Improve performance #9794

Merged

thatonecalculator merged 2 commits from supakaity/hajkey:hk/delete-fixes into develop

supakaity commented

delete !== undefined

So during the rome refactor (6b00abf05c), some delete object[key] statements were changed to object[key] = undefined.

Unfortunately setting to undefined means that the key is still iterable and this causes some issues.

In the case of the chart updates, upon chart.commit(...), it leaves the entry there and over time it causes the worker to keep creating empty entries in the chart database for every event that's ever happened since the process first started.

This means that after several days of running, the server may be updating many thousands of hashtags every 20 minutes, regardless of whether the hashtag had any use during that collection interval.

This also means that the database tables are growing continually at an ever increasing rate the longer the process has run without being restarted.

To fix this, some of the objects in question have been converted from plain {} objects to Map<K,V>s so we can use the accessor methods as recommended in the Rome documentation. In the case of the chart core's buffer, I had to put in a rome-ignore directive as I couldn't work out an elegant way to do it "properly" within the Commit<T> type.

Parallel execution

The second issue is that when performing large numbers of chart updates, the Promise.all allows unrestrained parallel execution of the chart update queries, which can adversely affect the operation of regular website queries and cause excess CPU consumption.

To fix this I have put a promiseLimiter in to restrict the number of concurrent queries per chart update.

# delete !== undefined So during the rome refactor (6b00abf05c0825244c66a20627b29805e2e14a4b), some `delete object[key]` statements were changed to `object[key] = undefined`. Unfortunately setting to undefined means that the key is still iterable and this causes some issues. In the case of the chart updates, upon `chart.commit(...)`, it leaves the entry there and over time it causes the worker to keep creating empty entries in the chart database for every event that's ever happened since the process first started. This means that after several days of running, the server may be updating many thousands of hashtags every 20 minutes, regardless of whether the hashtag had any use during that collection interval. This also means that the database tables are growing continually at an ever increasing rate the longer the process has run without being restarted. To fix this, some of the objects in question have been converted from plain `{}` objects to `Map<K,V>`s so we can use the accessor methods as recommended in the Rome documentation. In the case of the chart core's buffer, I had to put in a `rome-ignore` directive as I couldn't work out an elegant way to do it "properly" within the `Commit<T>` type. # Parallel execution The second issue is that when performing large numbers of chart updates, the `Promise.all` allows unrestrained parallel execution of the chart update queries, which can adversely affect the operation of regular website queries and cause excess CPU consumption. To fix this I have put a `promiseLimiter` in to restrict the number of concurrent queries per chart update.

Labels Milestones

Improve performance #9794

delete !== undefined

Parallel execution

Step 1:

Step 2: