Forum OpenACS Q&A: Re: Issue with reachable method in 5.10.1 dynamic clustering setup

Collapse
14: Re: Issue with reachable method in 5.10.1 dynamic clustering setup (response to 1)
Posted by Gustaf Neumann on 12/05/23 03:46 PM
The recent change was just about the display on the cluster admin page. The detection of not-reachable servers is there since a while.
Here is a screenshot with the canonical server and an active node

... and a screenshot with the canonical server and an inactive node

It looks like it would require manual cleanup using the status page as a guide for the level of inactivity?
What makes you think this?
If you experience an error, please submit a bug report.

-g

Collapse
15: Re: Issue with reachable method in 5.10.1 dynamic clustering setup (response to 14)
Posted by Jonathan Kelley on 12/05/23 03:58 PM
In my testing nodes have been inactive for an hour or more and not removed from this page or the DynamicClusterPeers parameter.

I haven't pulled down the latest but I didn't see any code change in your commits or the source I have that does this cleanup.

If there is code to do the automatic cleanup can you send along a pointer so I can review.

Thanks.

Collapse
16: Re: Issue with reachable method in 5.10.1 dynamic clustering setup (response to 14)
Posted by Gustaf Neumann on 12/05/23 04:20 PM
In my testing nodes have been inactive for an hour or more and not removed from this page or the DynamicClusterPeers parameter.

It is intentional to show on the cluster-admin page inactive cluster nodes to show their state, the last contact, etc. If these nodes would be deleted from the admin page, the administration would be much harder.

It would be possible to provide a display mode to omit these nodes from the listing. Such a mode might be useful, when there is a high number of cluster nodes (more than one page), but I am not sure if this is somewhere needed.

It would also be possible to distinguish between short network separations (short times of lost reachability) and final deletion (when one scales a cluster up and shrinks it down later, without any intention to re-include after a network separation these nodes). Maybe this is what you are looking for?

Collapse
17: Re: Issue with reachable method in 5.10.1 dynamic clustering setup (response to 16)
Posted by Jonathan Kelley on 12/05/23 07:55 PM
Yes, we will have scaling events that bring up dynamic nodes with new ips and then bring them down when load is decreased. This will lead to a large set of inactive hosts over time if the load increases/decreases multiple times a day or over a week.

We have some logic to clean up similar parameters ClusterPeerIP and ClusterAuthorizedIP for this type of situation but thought maybe the DynamicClusterPeers would be self-controlled within the core functionality.