Nyzo techRelease notesNyzo 475: mesh reopening 1

Nyzo 475: mesh reopening 1

Nyzo version 475 (commit on GitHub) is the first of two updates to allow the mesh to be reopened for new verifiers.

Shortly after the launch of the Nyzo blockchain, interest in the project resulted in rapid growth of the list of verifiers waiting to join. The design of the Nyzo verifier software at this time caused the computational demands of in-cycle verifiers to grow proportionally to the size of the waiting-verifiers list, and the magnitude of that growth was such that it was causing an unacceptable burden on in-cycle verifiers.

While the decentralized nature of Nyzo prevented us from truly closing the mesh to new verifiers, we were able to hide new verifiers from the nyzo.co website, and we were able to instruct the official Nyzo verifiers to ignore messages from verifiers that joined after a cutoff date. We were neither happy about nor proud of this solution, and some users figured out how to work around it, adding one or more lines to the trusted_entry_points file to contact verifiers that the Nyzo team did not control.

"Closing" the mesh did, however, reduce the rate at which new verifiers were started. This reduced rate gave us time to implement design improvements that effectively decoupled computational load of in-cycle verifiers from the size of the pool of verifiers waiting to join the cycle. This version, along with version 476, implement those improvements.

In BlockFileConsolidator, consolidation is no longer performed as soon as the last block in a 1000-block range is frozen. Instead, consolidation is performed when that block falls behind the retention edge. This was done to improve the efficiency of the initialization process when the verifier is restarted. Loading of blocks from individual files is more efficient than loading of blocks from consolidated files.

RN_475 code 0

Deletion of the individual file for the Genesis block was eliminated. This block will always be required by the BlockManager.

RN_475 code 1

In BlockManager, the verifierInCurrentCycle set has been replaced with currentCycleList and currentCycleSet. The list is used when an ordered collection is needed, and the set is used when ordering is unimportant.

The currentCycleEndHeight field stores the block height of the last element in currentCycleList. The cycleComplete field tracks whether the BlockManager has been tracking the blockchain long enough to have full knowledge of the current cycle.

RN_475 code 2

A main() method was added for testing the initialization process.

RN_475 code 3

In the getTrailingEdgeHeight() accessor, recalculation of the trailing edge height is now performed if the current value is negative. This ensures that a valid value is returned as soon as sufficient block information is available.

RN_475 code 4

In the getRetentionEdgeHeight() accessor, a value of -1 is now returned when the trailing edge height is -1. This does not affect behavior, but it makes more sense than returning a value of -25 when the value cannot be calculated. Also, when the the trailing edge is close to 0, the retention edge is limited to no less than 0. This does not affect behavior, either, and is only to improve display.

RN_475 code 5

In frozenBlockForHeight(), the HistoricalBlockManagerMap fallback was eliminated. Delivering blocks behind the retention edge was an unnecessary burden on verifiers.

RN_475 code 6

Two TODO comments were removed after testing. Until commit a24f170884c05679e4634d7cc7bbb7e760273f65, the Nyzo verifier did not have a BalanceListManager, instead storing balance lists as properties on blocks. Introduction of the BalanceListManager was necessary to control memory usage, but it required extensive testing to ensure that balance lists were available when they were needed for processing.

RN_475 code 7

The one-argument overload of BlockManager.freezeBlock() retrieves the previous block and passes information about it to the other overload of the method. This other overload now accepts another argument, the list of cycle verifiers, and the one-argument method provides a null value for this.

RN_475 code 8

The three-argument overload of BlockManager.freezeBlock() was modified to accept four arguments. The new argument, cycleVerifiers, is used in some cases as a lightweight alternative to allow the verifier to know the membership the current cycle when it would otherwise be unable due to a short time tracking the blockchain.

Two notifications were removed from this method, also.

RN_475 code 9

In BlockManager.loadBlockFromFile(), blocks are no longer loaded from consolidated files. Loading blocks from individual files is much more efficient than loading blocks from consolidated files. When a single block from a file is needed, many more from the same file are typically needed. So, when a single block is needed from a consolidated file, the entire file is extracted, and the block is loaded from the individual file.

RN_475 code 10

In BlockManager.initialize(), a check was added to prevent entry into the initialization code if initialization has already completed. This causes the diff to show substantial changes due to indentation differences.

RN_475 code 11

Further into the method, there are more indentation changes, and loading of blocks from consolidated files has been eliminated.

RN_475 code 12

Blocks are loaded from individual files as before. To attempt to have cycle information available as often as possible, blocks are loaded behind the frozen edge until the cycle information for the frozen edge can be calculated or until a missing block is encountered.

RN_475 code 13

Loading of blocks between the trailing edge and frozen edge has been eliminated, replaced with the loading of blocks starting at the frozen edge and stepping backward, described above. The remaining changes in the method are indentation differences due to the !initialized condition now wrapping the entire method.

RN_475 code 14

The BlockManager.loadBalanceListFromFileForHeight() method now extracts consolidated files if their contents are needed. The updated consolidation logic should prevent blocks from ever needing to be used after consolidation, but this extraction pattern for blocks and balance lists assures elimination of performance issues due to loading data from consolidated files.

RN_475 code 15

The BlockManager.extractConsolidatedFile() method produces individual block files from consolidated block files. This is roughly the inverse of the function of the BlockFileConsolidator.

RN_475 code 16

As blocks are no longer read directly from consolidated files, the BlockManager.findHighestConsolidatedFileStartHeight() method is no longer needed.

RN_475 code 17

BlockManager.setFrozenEdge() now accepts a list of cycle verifiers as its second argument. This is a backup for situations when the list of verifiers cannot be calculated from available blocks. The list is passed to the updateVerifiersInCurrentCycle() method for the calculation.

Also, the trailingEdgeHeight is now set to -1L whenever the frozen edge's cycle information is not available.

RN_475 code 18

The currentCycleList replaces the currentCycleLength field as the source of the value for the currentCycleLength() method.

RN_475 code 19

The BlockManager.verifiersInCurrentCycleList() method provides the identifiers of the verifiers in the current cycle in an ordered list. The BlockManager.verifiersInCurrentCycleSet() method, a renaming of BlockManager.verifiersInCurrentCycle(), returns the same information as an unordered set.

RN_475 code 20

The BlockManager.verifierInCurrentCycle() method uses the set that was renamed from verifiersInCurrentCycle to currentCycleSet.

RN_475 code 21

The BlockManager.updateVerifiersInCurrentCycle() method now accepts the supplemental bootstrapCycleVerifiers argument. This list of cycle verifiers, provided by BootstrapResponseV2, can be used by a verifier in the absence of sufficient block history to know what verifiers are in the current cycle.

The first part of this method attempts to determine the verifiers in the current cycle through examination of blocks.

RN_475 code 22

If the standard (block-based) calculation of cycle verifiers was unsuccessful, an alternate calculation using either the BlockManager.currentCycleList or bootstrapCycleVerifiers is performed instead.

If either of the calculations succeeds, the values are stored in the class fields.

RN_475 code 23

In BlockVoteManager, a minimumVoteInterval was added to limit the rate at which votes are flipped. A receipt timestamp is stored on votes to ensure that enforcement is based on local timestamps, not on timestamps set by the sender. If the interval between receipt timestamps of votes is less than the specified minimum, the vote is discarded.

RN_475 code 24

When retrieving the verifiersInCurrentCycle from BlockManager, the updated method name is used.

RN_475 code 25

The ChainInitializationManager no longer uses the BootstrapVoteTally class, so it was deleted.

RN_475 code 26

In ChainInitializationManager, the hashVotes map, which stored BootstrapVoteTally objects keyed on height, has been replaced with the bootstrapResponses map, which stores BootstrapResponseV2 objects keyed on verifier identifier.

The ChainInitializationManager.processBootstrapResponseMessage() method previously retrieved and modified BootstrapVoteTally objects based on the bootstrap response. Now, it stores the BootstrapResponseV2 in a map, using the verifier's identifier as the key.

RN_475 code 27

The ChainInitializationManager.frozenEdgeHeight() method was replaced with ChainInitializationManager.winningResponse(). The old method provided a hash and height derived from the BootstrapVoteTally objects in the hashVotes map. The new method performs a count of BootstrapResponseV2 objects in the bootstrapResponses map. The BootstrapResponseV2 object contains both the hash and height of the frozen edge, like the previous result. It also provides the list of cycle verifiers used by the BlockManager in the alternate cycle calculation.

RN_475 code 28

The ChainInitializationManager.fetchChainSection() method was replaced with ChainInitializationManager.fetchBlock(). The new method only needs to fetch a single block and balance list. This, combined with the list of cycle verifiers from the BootstrapResponseV2, allows a verifier to start tracking the blockchain. The request is performed in a loop that continues until a valid block and balance list are available.

RN_475 code 29

When the BlockResponse is received in the fetchBlock() method, its block is checked against the winning bootstrap response to ensure the correct block was sent.

RN_475 code 30

The method waits up to 5 seconds for the response to the block request.

RN_475 code 31

If a good response was received, the block is frozen. The list of cycle verifiers from the bootstrap response is also provided to the freezeBlock() method.

RN_475 code 32

In MeshListener.start(), an error is now printed and UpdateUtil.terminate() is called if an exception is thrown.

RN_475 code 33

In MeshListener.response(), the response to the BootstrapRequest1 message was removed. Responding to this message was a considerable burden and a large part of why temporary closure of the mesh was necessary.

RN_475 code 34

Later in MeshListener.response(), the call to NodeManager.updateNode() was removed from the process of responding to MessageType.NewBlock9 messages. This call was initially added to help keep information about the cycle current. It is being removed to prepare for introduction of the sentinel.

RN_475 code 35

A response to BootstrapRequestV2_35 was added to replace BootstrapRequest1.

RN_475 code 36

A response to the private NewVerifierTallyStatusRequest414 message was added. The response provides the verifier's tally of votes from the NewVerifierVoteManager.

RN_475 code 37

The Message.broadcast() method previously sent messages to all nodes. This design allowed out-of-cycle nodes to follow the consensus process and track the blockchain just as in-cycle nodes could. However, this was too much of a burden on in-cycle nodes as the list of out-of-cycle nodes grew. While the cycle grows at a strictly controlled rate, out-of-cycle nodes can be added as quickly as operators choose to start them.

The broadcast() method was modified to only send messages to in-cycle nodes and a small number of out-of-cycle verifiers at the top of the voting list.

RN_475 code 38

To further reduce the burden on in-cycle verifiers, if a BlockVote19 message is received from an out-of-cycle verifier, a response is no longer provided.

RN_475 code 39

In Message.processContent(), conditions for BootstrapRequest1 and BootstrapResponse2 were removed. Conditions for BootstrapRequestV2_35, BootstrapResponseV2_36, and NewVerifierTallyStatusResponse415 were added.

RN_475 code 40

In MessageQueue, three static fields were added. The shouldPrintZeroOnRemoval field is used to avoid showing that the message queue emptied unless its size was significantly large. The inBadState field marks when the MessageQueue is suspected to be stalled. The lastMessageStatus field is used in conjunction with inBadState to try to help diagnose problems with the queue.

RN_475 code 41

The MessageQueue.blockThisThreadUntilClear() method sleeps on the calling thread until the queue has cleared. If the MessageQueue is filled with messages faster than those messages can be removed, message-processing time increases and the system stops functioning properly. This method is called before sending new messages to avoid clogging the queue.

RN_475 code 42

In MessageQueue.add(), print statements were added to periodically show as the queue grows in size. If the queue passes a certain size, the shouldPrintZeroOnRemoval field is set to true so another message will be shown when the queue empties.

RN_475 code 43

In MessageQueue.next(), print statements were added to show status as the queue shrinks in size.

RN_475 code 44

In MessageQueue.start(), statements were added to provide insight into queue stalls. The lastMessageStatus field is set in this method to indicate the last processing state the queue tried to perform.

RN_475 code 45

In the MessageType enumeration, values were added for the bootstrap version 2 and new-verifier tally messages. The values for the version-1 bootstrap messages were removed.

RN_475 code 46

In NewVerifierQueueManager.updateVote(), a condition was added to only update the vote if this verifier is in cycle.

RN_475 code 47

A method call in NewVerifierQueueManager.calculateVote() was updated for the renamed BlockManager.verifiersInCurrentCycleSet() method.

RN_475 code 48

A method call in NewVerifierVoteManager.removeOldVotes() was also updated for the renamed BlockManager.verifiersInCurrentCycleSet() method.

RN_475 code 49

Creation of the vote-count map was encapsulated in the NewVerifierVoteManager.voteTotals() method. This allows the logic, which was previously inline in the NewVerifierVoteManager.topVerifiers() method, to be reused by the NewVerifierTallyStatusResponse.

RN_475 code 50

The NewVerifierVoteManager.topVerifiers() method has been modified to use the NewVerifierVoteManager.voteTotals() method and the renamed BlockManager.verifiersInCurrentCycleSet() method.

RN_475 code 51

At the end of NewVerifierVoteManager.topVerifiers(), setting of the top new field on the status response was eliminated. The NewVerifierTallyStatusResponse provides a detailed look into the vote tally, so this field is no longer needed.

RN_475 code 52

In NodeManager, the nodeJoinRequestTimestamps map was added to limit the frequency at which node-join messages are sent to other nodes.

RN_475 code 53

In NodeManager.updateNode(), the types of messages registered were reduced in preparation for the sentinel.

RN_475 code 54

Using the nodeJoinRequestTimestamps map, a minimum interval of 60 seconds between messages to each node is now imposed in NodeManager.sendNodeJoinMessage(). Most of the differences in this method are due to indentation changes.

RN_475 code 55

In UnfrozenBlockManager, the disconnectedBlocks map was added to allow a verifier to store blocks past the frozen edge when they cannot yet be registered normally. The lastBlockVoteTimestamp is used to ensure that this verifier does not change its vote more frequently than is allowed by the minimum vote interval. The attemptToRegisterDisconnectedBlocks() method tries to register blocks from the disconnectedBlocks map into the primary map.

RN_475 code 56

In UnfrozenBlockManager.registerBlock(), blocks more than one past the frozen edge for which a balance list cannot be derived are added to the disconnectedBlocks map.

RN_475 code 57

The BlockVoteManager.minimumVoteInterval is enforced in UnfrozenBlockManager.updateVote().

RN_475 code 58

In UnfrozenBlockManager.castVote(), the lastBlockVoteTimestamp is set to help enforce the minimum vote interval. Votes are now only broadcast from in-cycle verifiers and during the Genesis cycle, when all verifiers are considered in-cycle.

RN_475 code 59

In UnfrozenBlockManager.attemptToFreezeBlock(), the lastBlockVoteTimestamp is reset to 0 when a block is frozen.

RN_475 code 60

In UnfrozenBlockManager.fetchMissingBlock(), a message was added to indicate receipt of a block response.

RN_475 code 61

To reduce network traffic, in UnfrozenBlockManager.requestMissingBlocks(), requests are now limited to only the height one past the local frozen edge.

RN_475 code 62

In the static block of Verifier, printing of the verifier's identifier was added after loading of the private seed. This allows the operator to ensure that the seed was read properly.

RN_475 code 63

To eliminate any possible concerns of odd behavior due to multiple concurrent accesses, the Verifier.loadPrivateSeed() method is now synchronized.

RN_475 code 64

In Verifier.start(), starting of the MeshListener was moved earlier in the process to allow the verifier to start receiving messages sooner. Several print statements were added to allow an operator to observe initialization progress.

RN_475 code 65

Later in the Verifier.start() method, sending of the old bootstrap messages was removed, and a condition was added to bypass the bootstrap process completely after restarts with obviously short downtime.

RN_475 code 66

The new bootstrap process uses the new BootstrapRequestV2_35 message. It also uses the MessageQueue.blockThisThreadUntilClear() method to avoid backing up the queue, and the processBootstrapResponseMessage() method is now encapsulated in ChainInitializationManager.

RN_475 code 67

The ChainInitializationManager.winningResponse() provides the consensus BootstrapResponseV2 object, which encapsulates the height, hash, and cycle list of the consensus frozen edge.

RN_475 code 68

If a consensus bootstrap response is determined, the verifier considers its own current state and its distance from this consensus response to determine whether the frozen edge should be reinitialized or the local state should be kept.

RN_475 code 69

Premature termination of the verifier was eliminated. The new bootstrap process will not reach this point without satisfactory initialization. This code diff appears more substantial than it is due to indentation changes.

RN_475 code 70

In Verifier.loadGenesisBlock(), a null argument was added to the BlockManager.freezeBlock() method call. This argument is for the new list of cycle verifiers provided by the bootstrap response, and it is irrelevant for the Genesis block.

RN_475 code 71

The Verifier.processBootstrapResponseMessage() method, which processed the original bootstrap responses, was removed. An equivalent method was added to ChainInitializationManager.

RN_475 code 72

In Verifier.verifierMain(), a call to MessageQueue.blockThisThreadUntilClear() was added at the beginning of the loop to avoid backing up the MessageQueue. A dead block of code related to reinitialization after an IP address change was removed. Before attempting to freeze a block, an attempt is now made to register disconnected blocks.

RN_475 code 73

In the BlockResponse constructor, responses are now limited to 10 blocks or fewer, but the maximum size of the response was raised from 50,000 bytes to 1,000,000 bytes.

RN_475 code 74

In BlockVote, the receiptTimestamp field was added to ensure a minimum time interval between block votes from a verifier.

RN_475 code 75

The entire BootstrapResponse class was deleted. The image below only shows the beginning of the class, but the entire class was removed.

RN_475 code 76

The BootstrapResponseV2 class replaces the BootstrapResponse class. It encapsulates a height for the frozen edge, a hash for the frozen edge, and a list of identifiers of verifiers in the current cycle.

The no-argument constructor populates the object with values representing the current state of the verifier. It is used for responding to requests. The three-argument constructor populates the object with the argument values. It is used for reconstructing responses from other verifiers.

RN_475 code 77

Accessors are provided for all fields of BootstrapResponseV2.

RN_475 code 78

The BootstrapResponseV2.getByteSize() and BootstrapResponseV2.getBytes() methods are implemented as would be expected for this class. All fields are serialized.

RN_475 code 79

The BootstrapResponseV2.fromByteBuffer() method is similarly predictable. It reads all the fields from a ByteBuffer and returns a BootstrapResponseV2 object containing the values.

RN_475 code 80

The BootstrapResponseV2.toString() method displays the frozen-edge height, frozen-edge hash, and the size of the list of cycle verifiers.

RN_475 code 81

The NewVerifierTallyStatusResponse provides information about the state of new-verifier voting. It is a debug (private) response, so it is only produced in response to self-signed requests. To simplify the code, at some expense to messaging efficiency, it is implemented as a MultilineTextResponse. The constructor that takes a Message argument builds a response based on the current state of this verifier.

RN_475 code 82

The NewVerifierTallyStatusResponse constructor that takes a List<String> argument is used to reconstruct a response from a different verifier.

RN_475 code 83

The getLines() accessor fulfills the MultilineTextResponse interface.

RN_475 code 84

The NewVerifierTallyStatusResponse.getByteSize() and NewVerifierTallyStatusResponse.getBytes() methods are used for serialization. A single value denotes the number of String objects in the list, and each String object is serialized with a length value followed by the bytes of the String.

RN_475 code 85

NewVerifierTallyStatusResponse.fromByteBuffer() deserializes NewVerifierTallyStatusResponse objects. The line count is read to determine how many String objects to read, and each String is added to a List as it is read.

RN_475 code 86

NewVerifierTallyStatusResponse.toString() displays the number of lines in the response list.

RN_475 code 87