After setting it up in the Tools\Options\Applications window as shown, it works perfectly for me.

Even after running the latest CCleaner, including the the registry "Issues", it still works fine.
We could check that the setting is registering in Firefox configuration.
Type "about:config" (no quotes) in the address bar, and then enter "mailto" (no quotes) in the filter search window. In the list of items displayed, make sure of the following:
"gecko.handlerService.schemes.mailto.1.name" has "Gmail" in the value column.
"network.protocol-handler.external.mailto" is set to "true".
"gecko.handlerService.schemes.mailto.1.uriTemplate" contains "https://mail.google.com/mail/?extsrc=mailto&url=%s"
In other words, looks like this: