MySQL Shell 8.0.24 for MySQL Server 8.0 and 5.7 has been released

[Nawaz Nazeer ahamed]

Dear MySQL users,

MySQL Shell 8.0.24 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under Oracle's
dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and 5.7.
Please upgrade to MySQL Shell 8.0.24.

MySQL Shell is an interactive JavaScript, Python and SQL console interface,
supporting development and administration for the MySQL Server. It provides
APIs implemented in JavaScript and Python that enable you to work with MySQL
InnoDB Cluster and use MySQL as a document store.

The AdminAPI enables you to work with MySQL InnoDB Cluster and InnoDB
ReplicaSet, providing integrated solutions for high availability and
scalability using InnoDB based MySQL databases, without requiring advanced
MySQL expertise.  For more information about how to configure and work with
MySQL InnoDB Cluster and MySQL InnoDB ReplicaSet see

https://dev.mysql.com/doc/mysql-shell/8.0/en/admin-api-userguide.html

The X DevAPI enables you to create "schema-less" JSON document collections and
perform Create, Update, Read, Delete (CRUD) operations on those collections
from your favorite scripting language.  For more information about how to use
MySQL Shell and the MySQL Document Store support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI you can
also use the latest MySQL Connectors for your language of choice. For more
information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell's SQL mode you can communicate with servers using the legacy
MySQL protocol. Additionally, MySQL Shell provides partial compatibility with
the mysql client by supporting many of the same command line options.

For full documentation on MySQL Server, MySQL Shell and related topics, see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.24, see the "General
Availability (GA) Releases" tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy!

Changes in MySQL Shell 8.0.24 (2021-04-20, General Availability)


     * AdminAPI Added or Changed Functionality

     * AdminAPI Bugs Fixed

     * Functionality Added or Changed

     * Bugs Fixed

AdminAPI Added or Changed Functionality


     * The API command-line integration, used for scripting
       MySQL Shell, has been improved to use the function and
       option metadata to properly parse arguments and associate
       them to the corresponding parameter in the API call.
       Previously, the command-line integration was processing
       all the arguments as strings, causing errors using
       operations such as:
> mysqlsh user@hostname:3306 -- cluster setOption "autoRejoinTries"  5
       The command-line integration now interprets the data
       being passed in based on what the API functions expect to
       receive. This enhancement also introduces support for
       lists in command-line integration calls.
       Additionally, you can now access the MySQL Shell online
       help from the command-line integration. For example, to
       get help on the shell.options functions, issue:
$ mysqlsh -- shell options --help

       For more information, see API Command Line Integration
(https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-command-line-integration.html).
       In addition, when registering a new MySQL Shell extension
       function, the new boolean cli option is supported by the
       shell.addExtensionObjectMember() operation. When an
       operation is registered with the cli option set to true,
       the object and the functions are made available for the
       command-line integration. This enables you to extend the
       scripting possibilities of MySQL Shell. (Bug #31186637)

AdminAPI Bugs Fixed


     * Most of the AdminAPI operations contain metadata
       preconditions to determine if it is valid or not to
       execute them. When this check was being done, it was
       adding two entries to the log, one to indicate the check
       was about to be done, and another to indicate the
       metadata state. This meant that operations such as
       monitoring a cluster, which implies executing regular
       status requests, resulted in a large number of log
       entries. Now, these two log messages have been merged
       into a single entry that gets logged at the info level
       when the metadata state is not correct, and as debug info
       when the state is correct. In other words, if the
       metadata state is correct, the message is only logged
       when log_level=debug. (Bug #32582745)

     * The memberRole is now included in the default output of
       Cluster.status(). Previously, this information was only
       included when the extended option had a value of 1 or
       higher. This makes it easier to know what an instance's
       role is in the cluster, regardless of whether its
       operating mode is R/W or R/O. (Bug #32381513)

     * dba.checkInstanceConfiguration() was performing an
       incorrect validation regarding the required privileges.
       This resulted in an endless loop where the operation
       detected missing privileges, you would give the grants as
       specified by the interactive help, but the operation
       would fail, indicating that the grants were missing. Now,
       the verification checks the correct list. Additionally,
       the internal list of required grants included SUPER,
       which is deprecated in 8.0. The fix replaces the SUPER
       grant with the fine-grained grants. (Bug #32287986)

     * The memberSslMode option did not support the VERIFY_CA
       and VERIFY_IDENTITY modes for the following operations:

          + dba.createCluster()

          + Cluster.addInstance()

          + Cluster.rejoinInstance()
       Now, the memberSslMode option supports these modes, and
       when they are used there is a validation to ensure that
       the CA certificates are supplied. If you choose to use
       the VERIFY_CA or VERIFY_IDENTITY mode, on each cluster
       instance you must manually supply the CA certificates
       using the ssl_ca and/or ssl_capath option. For more
       information, see Securing your Cluster
(https://dev.mysql.com/doc/mysql-shell/8.0/en/configuring-innodb-cluster.html#mysql-innodb-cluster-securing).
       Thanks to Daniël van Eeden for the contribution. (Bug
       #32247631, Bug #32241000)

     * In version 8.0.23, a check was added to verify if the
       server_id of all cluster instances is registered in the
       metadata as an instance attribute, and if not then the
       metadata is updated accordingly. This check is executed
       on add, rejoin and rescan operations. However, when
       upgrading a cluster from a version earlier than 8.0.23 to
       version 8.0.23 and higher, the server_id was not
       registered in the metadata unless you performed a manual
       rejoin. This was being silently ignored because it was
       not included in any diagnostic messages. Now, a new
       verification checks if the server_id of cluster instances
       is missing from the metadata and includes a note message
       in the instanceErrors attribute in the output of
       Cluster.status() indicating to use Cluster.rescan() to
       fix it. (Bug #32226871)

     * From MySQL Shell version 8.0.17, AdminAPI stores the
       replication or recovery accounts used for each added
       instance in the metadata schema, in the instances table.
       However, the specific transaction could fail or that
       entry might have been manually removed from the metadata
       schema, resulting in failures when trying to add other
       instances to the cluster, and there was no way to resolve
       this using AdminAPI. In such a situation, if you tried to
       add an instance the operation failed with an error. Now,
       AdminAPI tries to detect problems in the metadata related
       to these accounts for an InnoDB Cluster or InnoDB
       ReplicaSet. The status() operation prints a message if
       the required account is missing in the metadata schema,
       and also if it is not the one actually used. In such
       situations, the MySQL Shell help instructs you to either
       re-add the instance or run rescan() based on the detected
       problem. The addInstance() operation also prints a hint
       to call rescan() if any missing recovery users are found
       in the metadata. (Bug #32157182)

     * Cluster.addInstance() was permitting usage of the
       expelTimeout and consistency options when it should not.
       These options are cluster level settings that can only be
       set using dba.createCluster() and Cluster.setOption().
       (Bug #29779995)

     * The dba.checkInstanceConfiguration() operation detects if
       the instance has any tables that do not have a primary
       key. Group Replication requires every table that is to be
       replicated by the group to have a defined primary key.
       However, this does not mean that having a table without a
       primary key causes Group Replication to block or fail.
       Rather, the outcome is that changes to that table are not
       replicated but the group continues operating. Previously,
       if the dba.checkInstanceConfiguration() operation
       detected a table without a primary key, the operation
       returned with a status of ok and only mentioned tables
       with a missing primary key and unsupported engines. Now,
       if the operation detects such a table, it returns with a
       status of error. As part of this work, the
       dba.createCluster() operation has been changed to fail if
       it finds such tables. (Bug #29771457)

     * As part of the fix for Bug#28701263, AdminAPI started
       setting and persisting a default value of READ_ONLY for
       the group_replication_exit_state_action system variable.
       The default used by Group Replication was ABORT_SERVER.
       However, in MySQL Server 8.0.16 the default value of
       group_replication_exit_state_action became READ_ONLY so
       AdminAPI should not change and persist it. Now, on
       instances running 8.0.16 and later, the value of
       group_replication_exit_state_action is not modified. (Bug
       #29037274)

     * The exception information listed in the online help had
       become outdated and unwieldy for the interactive MySQL
       Shell, so it has been removed. (Bug #28542904)
       References: See also: Bug #29853828, Bug #32426083, Bug
       #32157120, Bug #28825389.

     * Using the allowRootFrom option with the
       dba.deploySandboxInstance() operation was creating a
       different remote root account depending on whether MySQL
       Shell was running in interactive mode or not. Now, the
       default value of allowRootFrom is consistent between both
       modes, and the account is created as root@% in both
       interactive and non-interactive mode. (Bug #27369121)

     * When you issue dba.createCluster() and
       dba.createReplica(), tables are created to store the
       metadata. If the default storage engine was not InnoDB,
       these operations could fail. Now, metadata creation
       operations always use the InnoDB storage engine. (Bug
       #101446, Bug #32110085)

Functionality Added or Changed


     * From MySQL 8.0.24, SQL statements that you issue in MySQL
       Shell's SQL mode can be sent to the operating system's
       system logging facility (syslog on Unix, or the Windows
       Event Log). You can select this option by specifying the
       --syslog command-line option when starting MySQL Shell,
       or by setting the history.sql.syslog MySQL Shell
       configuration option. SQL statements that would be
       excluded from the MySQL Shell code history are also
       excluded from the system logging facility. (Bug
       #31995742, Bug #31514599)

     * MySQL Shell's instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() can now check for tables that
       do not contain primary keys. The check is carried out
       when the ocimds option is enabled for checking
       compatibility with MySQL Database Service, and an error
       is reported for every table included in the dump that
       does not have a primary key. The compatibility option,
       which implements appropriate measures for compatibility,
       has two new modification choices to notify MySQL Shell's
       dump loading utility to create primary keys in invisible
       columns for tables that do not have them, or to ignore
       the missing primary keys. Primary keys are required for
       MySQL Database Service High Availability, which uses
       Group Replication.

Bugs Fixed


     * Previously, MySQL Shell retried requests to Oracle Cloud
       Infrastructure Object Storage a maximum of 5 times, with
       a 30 second wait in between retries, and a maximum
       overall wait of 5 minutes. The retry strategy has now
       been changed to increase the wait window and reduce the
       possibility of a dump or load operation failing. MySQL
       Shell now retries a maximum of 10 times, with a 1 minute
       wait in between retries, and a maximum overall wait of 10
       minutes. (Bug #32592962)

     * MySQL Shell's instance dump utility util.dumpInstance()
       stopped with an error if the last schema to be dumped was
       a schema that contained no tables. The issue has now been
       fixed. (Bug #32540460)

     * MySQL Shell's instance dump utility util.dumpInstance()
       has been optimized so that it can still be used
       successfully if there are limitations on the server's
       resources such as disk space or the thread stack. To
       handle such situations, the queries from the utility can
       be repeated to retrieve smaller chunks of data if
       required, and file sorting is avoided. (Bug #32528186)

     * MySQL Shell's instance dump utility util.dumpInstance()
       incorrectly removed grants of all privileges to users.
       The utility now expands GRANT ALL statements in the dump
       to list all privileges granted on all schemas and tables
       (*.*), and to list allowed privileges for system schemas.
       The dump loading utility util.loadDump() now extracts the
       lists of allowed and revoked global privileges during
       loading, and strips these from GRANT statements relating
       to system schemas and to all schemas and tables. (Bug
       #32526567)

     * MySQL Shell's dump loading utility util.loadDump() now
       grants privileges after all the data is loaded.
       Previously, an error could occur if the utility tried to
       grant a privilege on a routine that did not yet exist.
       (Bug #32526496)

     * MySQL Shell's instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() could not complete the dump if
       the gtid_executed system variable or the Information
       Schema's COLUMN_STATISTICS table was unavailable. The
       utilities now display a warning message and log a
       detailed error message in this situation. These items are
       not required for a successful dump. (Bug #32515696)

     * MySQL Shell's handling and formatting has been improved
       for the help text that you provide for dictionary
       parameters and their options when you register a Python
       plugin. (Bug #32509309)

     * MySQL Shell's instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() no longer write a FLUSH TABLES
       statement to the binary log, as this can interfere with
       replication. (Bug #32490714)

     * From MySQL 8.0.23, MySQL Server supports replication from
       a source server that does not have GTIDs enabled and does
       not use GTID-based replication, to a replica that has
       GTIDs enabled, using the
       ASSIGN_GTIDS_TO_ANONYMOUS_TRANSACTIONS option of the
       CHANGE REPLICATION SOURCE TO statement. MySQL Shell's
       instance dump utility util.dumpInstance(), schema dump
       utility util.dumpSchemas(), and table dump utility
       util.dumpTables() now support this functionality by
       storing the binary log file name and position in the dump
       metadata, in addition to the gtid_executed GTID set. The
       additional privilege REPLICATION CLIENT is required in
       order for the utilities to be able to collect this
       information, although if the user ID does not have that
       privilege, the dump continues but without the binary log
       information.
       The binary log information can be used after loading the
       dumped data into the replica server to set up replication
       with a non-GTID source server. MySQL Shell's dump loading
       utility util.loadDump() prints the binary log and GTID
       set information from the dump metadata (in YAML format)
       when you specify the new option showMetadata: true. (Bug
       #32430402)

     * MySQL Shell did not correctly handle an empty array that
       was added to a collection. The result set normally
       returned from the server is now skipped in this
       situation. (Bug #32377134)

     * MySQL Shell's instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() were unable to chunk table data
       from a MySQL 5.6 server instance, due to differences in
       the output of the EXPLAIN SELECT statement for that
       version. The utilities now account for the difference,
       and also cache the server version information for faster
       access. (Bug #32376447)

     * MySQL Shell's parallel table import utility
       util.importTable() set a zero exit code if a non-critical
       error occurred that did not interrupt the import, such as
       a directory or file not being found. The utility now sets
       a non-zero error code instead when the first non-critical
       error is observed. (Bug #32286186)

     * MySQL Shell's upgrade checker utility
       util.checkForServerUpgrade() now checks for spatial data
       columns that were originally created in MySQL 5.6. The
       underlying data type for such columns in MySQL 5.6 does
       not match their underlying data type in MySQL 8.0, so
       upgrade of the table is prohibited, and it must be
       recreated. (Bug #32257211, Bug #101944)

     * When MySQL Shell casts a string to a boolean value, the
       operation is now case insensitive. Previously, the
       results could differ between platforms. (Bug #32217910)

     * When MySQL Shell's \warnings command was used to show
       warnings after each statement, warnings were not
       displayed for a classic MySQL protocol connection. (Bug
       #32151137)

     * MySQL Shell's parallel table import utility
       util.importTable() now checks whether an uploaded object
       is a directory, and excludes these from wildcard matching
       that was specified for files. (Bug #31991122)

     * MySQL Shell's dump loading utility util.loadDump() can
       split oversized chunks of data into smaller chunks for
       upload. Previously, if loading was stopped then resumed
       partway through this stage, the rows in the smaller
       chunks that were already loaded were not taken into
       account and skipped, which could lead to deadlocks. The
       utility's progress file now records the smaller chunks
       individually so that they can be skipped if the load is
       stopped and resumed. (Bug #31961688)

     * An event that contained a sequence of two semi-colons
       caused MySQL Shell's instance dump utility
       util.dumpInstance(), schema dump utility
       util.dumpSchemas(), and table dump utility
       util.dumpTables() to enter an infinite loop looking for
       delimiters. (Bug #31820571)

     * The decodeColumns option for MySQL Shell's parallel table
       import utility util.importTable() could be specified
       without an accompanying columns option, resulting in the
       import stopping with an error. (Bug #31407058)

     * If a script that was run interactively in MySQL Shell's
       Python mode did not have a newline character at the end,
       and the script ended with a multiline command, MySQL
       Shell waited for input instead of processing the command.
       The user had to press Enter to finish running the script,
       and the last line of the script was incorrectly saved in
       MySQL Shell's code history. MySQL Shell now adds an empty
       line after processing a script input stream, to ensure
       that this situation does not occur. (Bug #30765725)

     * MySQL Shell used a different character set for collations
       depending on whether X Protocol or classic MySQL protocol
       was used to connect to the MySQL server instance, leading
       to inconsistency and in some situations, errors. For
       MySQL 5.7 instances, MySQL Shell now uses a SET NAMES
       statement to set all the relevant session system
       variables to the utf8mb4 character set. For MySQL 8.0
       instances, MySQL Shell now sets the collation_connection
       system variable to the utf8mb4_0900_ai_ci character set.
       (Bug #30516645)

Enjoy and thanks for the support!

On Behalf of the MySQL Engineering Team,
Nawaz Nazeer Ahamed