I like the implementation, but it looks like nobody has pointed out that Python already has a built-in string template system [1]. Granted, it's not quite as simple to use, but I have been using it for a while.
The best form of encryption is a one-time pad. It is mathematically proven to be unbreakable, unlike every other form of encryption. If you can exchange a 100MiB pad file with each contact, you should be set for life if all you exchange is text messages.
IMHO, with this type of issue is often more likely blowing through the multixact cache or the query planner reverting to SEQSCAN due to the number of locks or mxact id exaustion etc.. It is most likely not a WAL flush problem that commit_delay would help with.
From the above link:[1]
> I found that performing extremely frequent vacuum analyze (every 30 minutes) helps a small amount but this is not that helpful so problems are still very apparent.
> The queue table itself fits in RAM (with 2M hugepages) and during the wait, all the performance counters drop to almost 0 - no disk read or write (semi-expected due to the table fitting in memory) with 100% buffer hit rate in pg_top and row read around 100/s which is much smaller than expected.
Bullet points 2 and 3 from here [2] are what first came to mind, due to the 100% buffer hit rate.
Note that vacuuming every 30min provided "minor improvements" but the worst case of:
25000 tps * 60sec *30min * 250rows == 11,250,000,000 ID's (assuming worst case every client locking conflicting rows)
Even:
25000tps 60sec 30min
Is only two orders of magnitude away from blowing through the 32bit transaction ID's.
45,000,000
4,294,967,296
But XID exhaustion is not as hidden as the MXID exhaustion and will block all writes, while the harder to see MXID exhaustion will only block some writes.
IMHO, if I was writing this, and knowing that you are writing an orchestration platform, getting rid of the long term transactions with just a status column would be better, row level locks are writing to the row anyways, actually twice.
Long lived transactions are always problematic for scaling, and that status column would allow for more recovery options etc...
But to be honest, popping off the left of a red black tree like the linux scheduler does is probably so much better than fighting this IMHO.
This opinion is assuming I am reading this right from the linked to issue [1]
> SELECT FOR UPDATE SKIP LOCKED executes and the select processes wait for multiple minutes (10-20 minutes) before completing
There is a undocumented command pg_get_multixact_members() [3] that can help troubleshoot as many people are using hosted Postgres, the tools too look into the above problems can be limited.
It does appear that Amazon documents a bit about the above here [4].
That sounds like double/triple glazing which is has been around for years in Europe. The most common type uses Argon as an insulator, but there are also vacuum versions.
My business is fiber broadband, and we are laying as much as we can! Fortunately the legislative environment in the UK is conducive to this goal. For now we provide XGS PON (10Gbs minus overhead), with 50G PON becoming available in some locations later this year. Unlike regular point-to-point, it is point-to-multipoint through the use of optical splitters with one head-end port capable of servicing up to 256 clients. With the UK being so far behind most other developed countries, it means it can take advances in all of the R&D from the last twenty years. It has also led to the strange situation where there are some properties that have a choice between 4 or even more fibre network providers. Some fiber networks are tied to a single ISP whereas others are open to wholesale. I am not sure if there are any ISP's that support multiple connections to the same property via different fiber networks - but I think you would generally want to use a different ISP for each network anyway. IPv6 is supposed to make that easier, though in practice there are still no protocols (a la MEF 17 Service OAM) for the ONT to signal to the router that there is a fault condition. This is one circumstance where PPPoE is useful. As the UK has been slow to deploy fiber there are many ISPs still using it. I found this out the hard way when I made the switch. My trusty old Ubiquiti Edgerouter Lite (Cavium silicon) is not capable of hardware offload for ipv6+pppoe+vlan at the same time, so anything particularly demanding on IPv6 (e.g. Steam) essentially locks up the router. I will be trying out an Alta Labs Route10 this week to see if it provides any improvement.
Perhaps there is an app that makes using one-time pads simple. It is really the only way to be reasonably certain that your communication is protected. With storage so abundant it is feasible to share a 100MiB pad for each contact which should be enough for a lifetime of messages.
It has been said that a system is not the sum of its parts, but the product of the interactions between its components. I hope someone can remind me who said this.
I am lucky I have access to a company AWS account, otherwise I would never have started to learn their ecosystem. I cannot afford the personal risk of a massive unexpected bill.
[1] https://docs.python.org/3/library/string.html#template-strin...
edit: this was mentioned by milesrout in https://news.ycombinator.com/item?id=43649607
reply