NDB Cluster APIs: The version of Node.js used by NDB has been upgraded to 12.22.1. (Bug #32640847)

NDB Cluster APIs: Added the NdbScanFilter::setSqlCmpSemantics() method to the NDB API. Previously, NdbScanFilter has always treated NULL as equal to itself, so that NULL == NULL evaluates as Boolean TRUE; this is not in accordance with the SQL standard, which requires that NULL == NULL returns NULL. The new method makes it possible to override the traditional behavior, and enforce SQL-compliant NULL comparisons instead, for the lifetime of a given NdbScanFilter instance.

For more information, see NdbScanFilter::setSqlCmpSemantics(), in the MySQL NDB Cluster API Developer Guide. (WL #14476)

ndb_restore now supports conversion between NULL and NOT NULL columns, as follows:

For more information, see the descriptions of the indicated ndb_restore options. (Bug #32702637)

Added the PreferIPVersion configuration parameter, which controls the addressing preference of the DNS resolver for IPv4 (4) or IPv6 (6), with 4 being the default. This parameter must be the same for all TCP connections; for this reason, you should set it only in the [tcp default] section of the cluster global configuration file. (Bug #32420615)

Packaging: The ndb-common man page was removed, and the information it contained moved to other man pages. (Bug #32799519)

Packaging: The mysqlbinlog utility was not included in the NDB Cluster Docker image. (Bug #32795044)

NDB Replication: Use of NDB_STORED_USER on an SQL node participating in NDB replication sometimes led to the repeated revocation and restoration of grants. We fix this by ensuring that log_slave_updates and the originating server_id are handled correctly when performing ACL operations in such cases. (Bug #32832676)

NDB Replication: When starting a cluster with 255 SQL nodes, some mysqld processes did not properly initialize schema distribution. This caused binary log setup to fail, with the result that the SQL node never became operational. This error occurred when subscribing to schema changes; setting up the NdbEventOperation for the mysql.ndb_schema table failed. In addition, a MySQL Server also needs to set up a subscription for the mysql.ndb_schema_result table so each MySQL Server requires two resources.

To fix this problem, the effective default for the MaxNoOfSubscriptions configuration parameter is now treated as 2 * MaxNoOfTables + 2 * [number of API nodes], rather than as 2 * MaxNoOfTables. (Bug #32380993)

NDB Cluster APIs: The Node.js adapter did not always handle character set and collation numbers correctly. (Bug #32742521)

NDB Cluster APIs: Added the NDB_LE_EventBufferStatus3 log event type to Ndb_logevent_type in the MGM API. This is an extension of the NDB_LE_EventBufferStatus type which handles total, maximum, and allocated bytes as 64-bit values.

As part of this fix, the maximum value of the ndb_eventbuffer_max_alloc server system variable is increased to 9223372036854775807 (2⁶³ - 1).

For more information, see The Ndb_logevent_type Type. (Bug #32381666)

Conditions which were pushable to the NDBCLUSTER engine were not pushed down to the table if it was referred to as part of a view or a table subquery. (Bug #32924533)

RPM builds of NDB for Docker which used dynamic linking did not complete due to the inclusion of the ndbclient library by ndbxfrm. Now ndbxfrm uses the internal ndbgeneral and ndbportlib libraries instead.

As part of this fix, ndb_restore also now links against ndbgeneral and ndbportlib. (Bug #32886430)

NDB now uses std::min() and std::max() in place of its own internal macros for determining the minimum and maximum of two numbers. (Bug #32854692)

Some error messages printed by ndb_restore tried to access transactions that were already closed for error information, resulting in an unplanned exit. (Bug #32815725)

The error messages for NDB errors 418 (Out of transaction buffers in LQH...), 419 (Out of signal memory...), and 805 (Out of attrinfo records in tuple manager...) all referred to increasing LongSignalMemory, although there is no configuration parameter by that name. All three of these error messages have been corrected to refer to the LongMessageBuffer parameter instead. (Bug #32797240)

An unsuccessful CREATE TABLE of an NDB table returns a generic error message (ERROR HY000: Can't create table 'tbl'), with additional, more specific error messages often pushed as warnings. To assist users who may not be aware of this and see only the generic message, we have added reminder text regarding the SHOW WARNINGS statement to the generic error message, to prompt the user to obtain additional information that might help resolve a given issue more quickly. (Bug #32788231)

An NDB error which is not mapped to a MySQL handler error code is typically presented to a MySQL user as error 1296 or 1297, with a message indicating the underlying NDB error code; one exception to this behavior is a COMMIT error (originating from ndbcluster_commit()), for which the usual NDB error is 4350 Transaction already aborted. MySQL eventually passed this to strerror() in the C library, where it was prefixed with Unknown error or similar, but the precise format of this prefix varied with platform-specific differences with the version of libc being used.

We fix this by creating both a new handler error HA_ERR_TX_FAILED, and a new client error ER_TRANSACTION_FAILED, associated with SQL State 25000 Invalid Transaction State. (Bug #32763179)

References: See also: Bug #30264401.

When started with the --print-full-config option, ndb_mgmd exited with the error Address already in use. This is fixed by skipping free port validation when this option is specified. (Bug #32759903)

Removed unneeded printouts that were generated in the cluster log when executing queries against the ndbinfo.cluster_locks table. (Bug #32747486)

The DbUtil class did not call mysql_library_end() when a thread using the MySQL client library had finished doing so, and did not release the thread's local resources by calling mysql_thread_end(). (Bug #32730214)

A memory leak took place in DbUtil when running a query for the second time using same DbUtil instance; the connection check did not detect the existing MYSQL instance, and replaced it without releasing it. (Bug #32730047)

Returning an error while determining the number of partitions used by a NDB table caused the MySQL server to write Incorrect information in table.frm file to its error log, despite the fact that the indicated file did not exist. This also led to problems with flooding of the error log when users attempted to open NDB tables while the MySQL server was not actually connected to NDB.

We fix this by changing the function that determines the number of partitions to use the value loaded from the MySQL data dictionary without fetching it from NDB, which also saves one round trip when opening a table. For the special case in which the table is opened for upgrade, we fall back to fetching the value from NDB in the upgrade code path. (Bug #32713166)

Using duplicate node IDs with CREATE NODEGROUP (for example, CREATE NODEGROUP 11, 11) could lead to an unplanned shutdown of the cluster. Now when this command includes duplicate node IDs, it raises an error. (Bug #32701583)

Improved the performance of queries against the ndbinfo.cluster_locks table, which could in some cases run quite slowly. (Bug #32655988)

Fixed a number of issues found in ndb_print_backup_file relating to argument parsing, error reporting, and opening of encrypted files using classes from ndbxfrm. (Bug #32583227)

The directory unittest/ndb was generated by the build process even though it is not used. This directory is no longer created when building NDB. (Bug #32553339)

To ensure that the log records kept for the redo log in main memory are written to redo log file within one second, a time supervisor in DBLQH acquires a lock on the redo log part prior to the write. A fix for a previous issue caused a continueB signal (introduced as part of that fix) to be sent when the redo log file was not yet opened and ready for the write, then to return without releasing the lock. Now such cases we release the acquired lock before waiting for the redo log file to be open and ready for the write. (Bug #32539348)

References: This issue is a regression of: Bug #31585833.

Updating the Ndb object used for receiving events from NDB in the binary log injector thread with the value for ndb_eventbuffer_max_alloc was performed both at the start of each epoch and after having handled one event, when it is sufficient to update the value once per epoch.

We fix this by not updating from the global value during processing of each event, which reduces the amount of work required during each event processing loop. (Bug #32494904)

Failure to find all blob parts for a blob column while reading from the event stream was not handled properly, which caused the data in the caller's copy-out buffer to be incomplete, with no error returned to the caller.

When a user of the event API has been notified that data has been received for a table with blob column, it creates a buffer large enough to hold the entire blob and then calls the function to read the blob column from the event stream. Most blob types are stored as several small parts in NDB; to read the blob data for a blob column from the event stream, the buffered event data must be traversed to find the blob parts and to copy each part into the provided buffer. Each piece of buffered event data associated with the blob column is examined to see whether it contains the data for the blob part desired. When a blob part is found, it is copied into the buffer at the original offset provided by the caller.

The function which finds the blob parts can copy out one or more blob parts at a time. This function is normally called several times while putting the blob parts together—first to find the first blob part, then all the parts in the middle (several at a time), and then the remainder in the last part. When the function does not find all requested blob parts in the buffered event data, this results in an inconsistency which may occur due to any of several different cases—all parts may not have been sent, the received parts may have been stored in the wrong place, there is a problem in the logic putting the blob parts together, or possibly some other issue. The inconsistency is detected by comparing how many blob parts have been found with how many were requested to be copied out this time.

This problem was noticed while investigating problem with an unplanned SQL node shutdown that could occur while executing some ALTER TABLE operations, where a debug-compiled mysqld asserted after having printed information about missing blob parts; manual code inspection shows that a release-compiled binary would just return the incomplete buffer to the caller. This problem was also noticed in addressing some previous similar issues.

We fix this problem by returning an error from NdbEventOperationImpl::readBlobParts() whenever requested blob parts cannot be found. Since this is a serious inconsistency, we also extend the printout provided when this problem is detected. A sample of the extended printout is shown here:

= print_blob_part_bufs =============================
part_start: 0, part_count: 15
table: { id: 13, version: 2, name: 't1' }
column: { attrid: 1, name: 'b' }
blob parts table: { id: 14, version: 2, name: 'NDB$BLOB_13_1' }
available buffers: {
[0]*:  part_number: 1, size: 2000, offset: 2000
[1]*:  part_number: 14, size: 2000, offset: 28000
[2]*:  part_number: 7, size: 2000, offset: 14000
[3]*:  part_number: 5, size: 2000, offset: 10000
[4]*:  part_number: 3, size: 2000, offset: 6000
[5]*:  part_number: 0, size: 2000, offset: 0
[6] :  part_number: 15, size: 2000, offset: 30000
[7]*:  part_number: 13, size: 2000, offset: 26000
[8]*:  part_number: 12, size: 2000, offset: 24000
[9]*:  part_number: 11, size: 2000, offset: 22000
[10]*:  part_number: 10, size: 2000, offset: 20000
[11]*:  part_number: 9, size: 2000, offset: 18000
[12]*:  part_number: 8, size: 2000, offset: 16000
[13]*:  part_number: 6, size: 2000, offset: 12000
[14]*:  part_number: 4, size: 2000, offset: 8000
[15]*:  part_number: 2, size: 2000, offset: 4000
} 

(Bug #32469090)

References: See also: Bug #32405937, Bug #30509284.

A node was permitted during a restart to participate in a backup before it had completed recovery, instead of being made to wait until its recovery was finished. (Bug #32381165)

Removed NDB_WIN32 from the NDB Cluster sources. This define was once intended to demarcate code to be conditionally compiled only for Windows, but had long since been superseded for this purpose by _WIN32. (Bug #32380725)

Running out of disk space while performing an NDB backup could lead to an unplanned shutdown of the cluster. (Bug #32367250)

The index statistics thread relies on the binary log injector thread to inform it about initial system restarts. The index statistics thread then (asynchronously) recycles its Ndb object and creates its system tables. Depending on timing, it was possible for the index statistics thread not to be ready to serve requests for a period of time during which NDB tables were writable. This also led to issues during the setup of stored grants when the data node parameter IndexStatAutoCreate was set to 1.

We fix this in two ways:

(Bug #32355045)

It was possible to start ndb_mgmd with NoOfReplicas set equal to 1 and with more than 72 data nodes defined in the config.ini file. Now the management server checks for this condition, and refuses to start if it is found. (Bug #32258207)

It was possible to start ndb_mgmd with an invalid value set in config.ini for the NodeGroup parameter; subsequently, data node processes using that value were unable to start. Now in such cases, the management server refuses to start, and provides an appropriate error message. (Bug #32210176)

A statement such ALTER TABLE t1 ALGORITHM=INPLACE, RENAME COLUMN B to b that performed an in-place rename of a column changing only the lettercase of its name was successful, but the change was not reflected in the NDB dictionary (as shown, for example, in the output of ndb_desc). We fix this issue by ensuring that the NDB dictionary always matches the lettercase specified in the SQL statement, and that this matches the name as stored in the MySQL data dictionary. (Bug #31958327)

Event buffer congestion could lead to unplanned shutdown of SQL nodes which were performing binary logging. We fix this by updating the binary logging handler to use Ndb::pollEvents2() (rather than the deprecated pollEvents() method) to catch and handle such errors properly, instead. (Bug #31926584)

The --resume option for ndb_import did not work correctly unless the --stats option was also specified. (Bug #31107058)

Reverted a previous change in the scope of the flags used by INSERT IGNORE and other similar SQL statements to inform the handler that duplicate key errors during an insert or update do not stop an ongoing transaction. Now these flags are cleared after every write row event, as before. (Bug #27538524)

References: See also: Bug #22603412. This issue is a regression of: Bug #20017428.

NDBCLUSTER uses bitmaps of type MY_BITMAP for keeping track of which columns are to be used in various contexts. When used in short-lived performance-critical code, these are initialized with a bit buffer whose (fixed) size is defined at compile time. The size of these buffers was calculated in multiple ways, which could lead to copy-paste errors, uncertainty whether the buffer is large enough, and possible allocation of excess space.

We fix this by implementing an internal Ndb_bitmap_buf class that takes the number of bits the buffer should hold as a template argument, and changing all occurrences of static bitmap buffers to instances of Ndb_bitmap_buf. This also saves several bytes in the condition pushdown code in which the buffers were too large. (Bug #27150799)

The NDB binlog injector thread now detects errors while handling data change events received from the storage engine. If an error is detected, the thread logs error messages and restarts itself, and as part of the restart an exceptional, incident, or LOST_EVENTS entry is written to the binary log. This special entry indicates to a replication applier that the binary log is incomplete. (Bug #18280115, Bug #27150740)

A DELETE statement whose WHERE clause referred to a BLOB column was not executed correctly. (Bug #13881465)

Analysis of data node and management node logs was sometimes hampered by the fact that not all log messages included timestamps. This is fixed by replacing a number of different logging functions (printf, fprintf, ndbout, ndbout_c, << overloading, and so on) with and standardizing on the existing EventLogger mechanism which begins each log message with a timestamp in YYYY-MM-DD HH:MM:SS format.

For more information about NDB Cluster event logs and the log message format, see Event Reports Generated in NDB Cluster. (WL #14311)

References: See also: Bug #21441915, Bug #30455830.