MySQL Shell 8.0.18 for MySQL Server 8.0 and 5.7 has been released

[Balasubramanian Kandasamy]

Dear MySQL users,

MySQL Shell 8.0.18 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under
Oracle's dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and
5.7. Please upgrade to MySQL Shell 8.0.18.

MySQL Shell is an interactive JavaScript, Python and SQL console
interface, supporting development and administration for the MySQL
Server. It provides APIs implemented in JavaScript and Python that
enable you to work with MySQL InnoDB cluster and use MySQL as a document
store.

The AdminAPI enables you to work with MySQL InnoDB cluster, providing an
integrated solution for high availability and scalability using InnoDB
based MySQL databases, without requiring advanced MySQL expertise. For
more information about how to configure and work with MySQL InnoDB
cluster see

https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create "schema-less" JSON document
collections and perform Create, Update, Read, Delete (CRUD) operations
on those collections from your favorite scripting language.  For more
information about how to use MySQL Shell and the MySQL Document Store
support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI
you can also use the latest MySQL Connectors for your language of
choice. For more information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell's SQL mode you can communicate with servers using the
legacy MySQL protocol. Additionally, MySQL Shell provides partial
compatibility with the mysql client by supporting many of the same
command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.18, see the
"General Availability (GA) Releases" tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy and thanks for the support!


Changes in MySQL Shell 8.0.18 (2019-10-14, General Availability)

     * InnoDB Cluster Added or Changed Functionality

     * InnoDB Cluster Bugs Fixed

     * Functionality Added or Changed

     * Bugs Fixed

InnoDB Cluster Added or Changed Functionality


     * MySQL Shell can now optionally log SQL statements that
       are executed by AdminAPI operations, and output them to
       the console if the --verbose option is set. The
       dba.logSql MySQL Shell configuration option or
       --dba-log-sql command line option activates logging for
       these statements. Statements executed by sandbox
       operations are excluded. Viewing the statements lets you
       observe the progress of the AdminAPI operations in terms
       of SQL execution, which can help with problem diagnosis
       for any errors.

     * AdminAPI now supports IPv6 addresses if the target MySQL
       Server version is higher than 8.0.13. When using MySQL
       Shell 8.0.18 or higher, if all cluster instances are
       running 8.0.14 or higher then you can use an IPv6 or
       hostname that resolves to an IPv6 address for instance
       connection strings and with options such as localAddress,
       groupSeeds and ipWhitelist. For more information on using
       IPv6 see Support For IPv6 And For Mixed IPv6 And IPv4
       Groups
(https://dev.mysql.com/doc/refman/8.0/en/group-replication-ipv6.html).
       References: See also: Bug #29557250, Bug #30111022, Bug
       #28982989.

     * You can now reset the passwords for the internal recovery
       accounts created by InnoDB cluster, for example to follow
       a custom password lifetime policy. Use the
       Cluster.resetRecoveryAccountsPassword() operation to
       reset the passwords for all internal recovery accounts
       used by the cluster. The operation sets a new random
       password for the internal recovery account on each
       instance which is ONLINE. If an instance cannot be
       reached, the operation fails. You can use the force
       option to ignore such instances, but this is not
       recommended, and it is safer to bring the instance back
       online before using this operation. This operation only
       applies to the passwords created by InnoDB cluster and
       cannot be used to update manually created passwords.
       Note
       The user which executes this operation must have all the
       required clusterAdmin privileges, in particular CREATE
       USER, in order to ensure that the password of recovery
       accounts can be changed regardless of the password
       verification-required policy. In other words, independent
       of whether the password_require_current system variable
       is enabled or not.

     * MySQL Shell now supports specifying TLS version 1.3 and
       TLS cipher suites for classic MySQL protocol connections.
       You can use:

          + the --tls-version command option to specify TLS
            version 1.3.

          + the --tls-ciphersuites command option to specify
            cipher suites.

          + the tls-versions and tls-ciphersuites connection
            parameters as part of a URI-type connection string.
       Note
       tls-versions (plural) does not have a key-value
       equivalent, it is only supported in URI-type connection
       strings. Use tls-version to specify TLSv1.3 in a
       key-value connection string.
       To use TLS version 1.3, both MySQL Shell and MySQL server
       must have been compiled with OpenSSL 1.1.1 or higher. For
       more information see Using Encrypted Connections
(https://dev.mysql.com/doc/refman/8.0/en/encrypted-connections.html).


InnoDB Cluster Bugs Fixed


     * The Cluster.rejoinInstance() operation was not setting
       the auto increment values defined for InnoDB cluster,
       leading to the use of the default Group Replication
       behavior if the instance configuration was not properly
       persisted, for example on 5.7 servers. The fix ensures
       that the Cluster.rejoinInstance() operation updates the
       auto increment settings of the target instance. (Bug
       #30174191)

     * The output of Cluster.status() now includes the
       replicationLag field. The value is displayed in HH:MM:SS
       format and shows the time difference between the last
       transaction commit timestamp and the last transaction
       applied timestamp. This enables you to monitor the amount
       of time between the most recent transaction being
       committed and being applied on an instance. (Bug
       #30003034)

     * Cluster.addInstance() did not ensure that the MySQL Clone
       plugin was installed or loaded on all cluster instances,
       when available and not disabled. This meant that whenever
       a cluster was created using an older MySQL Shell version,
       on a target MySQL instance supporting clone, the instance
       would not have the clone plugin installed. The result was
       that any Cluster.addInstance() call that used clone would
       fail. The same issue happened if an instance was added to
       a cluster consisting of one instance using the
       incremental recovery type and afterwards the seed
       instance was removed. This resulted in all cluster
       instances not having the clone plugin installed and
       therefore any instance added using the clone recovery
       method would fail. The fix ensures that the clone plugin
       is installed on all cluster members (if available and not
       disabled) at cluster creation time and also whenever an
       instance is added to a cluster. (Bug #29954085)

     * The Cluster.rejoinInstance() operation was not checking
       the GTID consistency of an instance being rejoined to a
       cluster, which could result in data diverging. Now, the
       GTID consistency checks conducted as part of the
       Cluster.rejoinInstance() operation have been improved to
       check for irrecoverable or diverged data-sets and also
       for empty GTID sets. If an instance is found to not be
       consistent with the cluster, it is not rejoined and the
       operation fails with a descriptive error. You are also
       shown the list of errant transactions, possible outcomes
       and solutions. (Bug #29953812)

     * Cluster.describe() was retrieving information about the
       cluster's topology and the MySQL version installed on
       instances directly from the current session. Now, the
       information is retrieved from the Metadata schema, and
       the MySQL version is not included in the information
       output by Cluster.describe(). (Bug #29648806)

     * Using a password containing the ' character caused
       dba.deploySandbox() to fail. Now, all sensitive data is
       correctly wrapped to avoid such issues. (Bug #29637581)

     * The Cluster.addInstance() operation creates internal
       recovery users which are required by the Group
       Replication recovery process. If the
       Cluster.addInstance() operation failed, for example
       because Group Replication could not start, the created
       recovery users were not removed. Now, in the event of a
       failure any internal users are removed. (Bug #25503159)

     * When a cluster had lost quorum and the majority of the
       cluster instances were offline except the primary, after
       reestablishing quorum and adding a new instance to the
       cluster, it was not possible to remove and add the
       previous primary instance to the cluster. This was
       because the operation failed when trying to contact
       offline instances, which was because the feature to
       verify if a Group Replication protocol upgrade is
       required was not considering the possibility of some
       cluster instances being offline (not reachable). The fix
       improves the Group Replication protocol upgrade handling
       for the Cluster.removeInstance() operation, which now
       attempts to connect to other cluster instances and use
       the first reachable instance for this purpose. (Bug
       #25267603)

     * The dba.configureInstance() operation was not setting the
       binlog_checksum option with the required value (NONE) in
       the option file for instances that did not support SET
       PERSIST (for example instances running MySQL 5.7), when
       the option file path was not provided as an input
       parameter but instead specified though the operation
       wizard in interactive mode. (Bug #96489, Bug #30171090)

Functionality Added or Changed


     * MySQL Shell's upgrade checker utility (the
       util.checkForServerUpgrade() operation) includes the
       following new and extended checks:

          + The utility now checks for tablespace names
            containing the string "FTS", which can be
            incorrectly identified as tablespaces of full-text
            index tables, preventing upgrade. The issue has been
            fixed in MySQL 8.0.18, but affects upgrades to
            earlier MySQL 8.0 releases.

          + The check for database objects with names that
            conflict with reserved keywords now covers the
            additional keywords ARRAY, MEMBER, and LATERAL.

          + - The checks for obsolete sql_mode flags now check
            the global sql_mode setting.
       Running the upgrade checker utility no longer alters the
       gtid_executed value, meaning that the utility can be used
       on Group Replication group members without affecting
       their synchronization with the group. The upgrade checker
       also now works correctly with the ANSI_QUOTES SQL mode.
       (Bug #30002732, Bug #30103683, Bug #96351)
       References: See also: Bug #29992589.

     * MySQL Shell has two new built-in reports, which provide
       information drawn from various sources including MySQL's
       Performance Schema:

          + threads lists the current threads in the connected
            MySQL server which belong to the user account that
            is used to run the report. Using the report-specific
            options, you can choose to show foreground threads,
            background threads, or all threads. You can report a
            default set of information for each thread, or
            select specific information to include in the report
            from a larger number of available choices. You can
            filter, sort, and limit the output.

          + thread provides detailed information about a
            specific thread in the connected MySQL server. By
            default, the report shows information on the thread
            used by the current connection, or you can identify
            a thread by its ID or by the connection ID. You can
            select one or more categories of information, or
            view all of the available information about the
            thread.
       You can run the new reports using MySQL Shell's \show and
       \watch commands. The reports work with servers running
       all supported MySQL 5.7 and MySQL 8.0 versions. If any
       item of information is not available in the MySQL Server
       version of the target server, the reports leave it out.

     * MySQL Shell has two new control commands:

          + The \edit (\e) command opens a command in the
            default system editor for editing. If you specify an
            argument to the command, this text is placed in the
            editor, and if you do not, the last command in the
            MySQL Shell history is placed in the editor. When
            you have finished editing, MySQL Shell presents your
            edited text ready for you to execute or cancel. The
            command can also be invoked using the short form \e
            or the key combination Ctrl-X Ctrl-E.

          + The \system (\!) command runs the operating system
            command that you specify as an argument to the
            command, then displays the output from the command
            in MySQL Shell. MySQL Shell returns an error if it
            was unable to execute the command.

     * MySQL Shell now uses Python 3. For platforms that include
       a system supported installation of Python 3, MySQL Shell
       uses the most recent version available, with a minimum
       supported version of Python 3.4.3. For platforms where
       Python 3 is not included, MySQL Shell bundles Python
       3.7.4. MySQL Shell maintains code compatibility with
       Python 2.6 and Python 2.7, so if you require one of these
       older versions, you can build MySQL Shell from source
       using the appropriate Python version.

Bugs Fixed


     * In debug mode, MySQL Shell raised an assertion when
       handling a character contained in SQL strings. (Bug
       #30286680)

     * If a Python lambda was added as a member of a MySQL Shell
       extension object, the Python object was not released
       correctly when MySQL Shell shut down, causing a
       segmentation fault. (Bug #30156304)

     * A memory leak could occur when Python code was executed
       in interactive mode. (Bug #30138755)

     * Help information for a MySQL Shell report could not be
       displayed unless there was an active session. MySQL Shell
       now checks for an open session only before actually
       running the report. (Bug #30083371)

     * If a default schema was set for the MySQL Shell
       connection, and a different default schema was set after
       the connection was made, MySQL Shell's \reconnect command
       attempted to use the default schema from the original
       connection. The user's current default schema is now used
       for the reconnection attempt. (Bug #30059354)

     * Due to a bug introduced by a change in MySQL Shell
       8.0.16, the MSI file that is used by Windows Installer to
       install MySQL Shell overwrote the Windows PATH
       environment variable with the path to the application
       binary (mysqlsh), removing any other paths present. The
       issue has now been fixed. (Bug #29972020, Bug #95432)

     * When the \reconnect command is used to attempt
       reconnection to a server, if the last active schema set
       by the user appears to be no longer available, MySQL
       Shell now attempts to connect with no schema set. (Bug
       #29954572)

     * In interactive mode, MySQL Shell now handles multiline
       comments beginning with a slash and asterisk (/*) and
       ending with an asterisk and slash (*/). (Bug #29938424)

     * The MySQL Shell \source command was not handled correctly
       when used in combination with SQL statements. (Bug
       #29889926)

     * With MySQL Shell in SQL mode, if multiple SQL statements
       including a USE statement were issued on a single line
       with delimiters, the USE statement was not handled
       correctly. (Bug #29881615)

     * If MySQL Shell's JSON import utility was used to send a
       large number of JSON documents to a server with
       insufficient processing capacity, the utility could fill
       up the write queue with batches of prepared documents,
       causing the connection to time out and the import to
       fail. The utility now waits to read the response from the
       server before sending the next batch of prepared
       documents to the server. (Bug #29878964)

     * When MySQL Shell was built from source with a bundled
       OpenSSL package, the required linker flags were not set.
       The issue has now been fixed. (Bug #29862189)

     * If a new query was executed in MySQL Shell while a result
       was still active, resulting in rows being cached, not all
       rows were returned by the old query. (Bug #29818714)


On Behalf of Oracle/MySQL Release Engineering Team,
Balasubramanian Kandasamy