= System Wide Change: NSS Default File Format SQL = https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql
Change owner(s): * Kai Engert kaie@redhat.com
Change the NSS library default to use the sqlite based data storage, when applications don't specify their preferred storage file format.
== Detailed Description ==
Applications that use the NSS library often use a database for storage of keys, certificates and trust. NSS supports two different file formats, one called DBM (based on berkeley DB files) and another one called SQL (based on sqlite DB files).
Today's default file format used by NSS, used when applications omit the type parameter, is the older DBM file format, which forbids parallel access to the storage. The suggestion is to change the default file format to SQL, which allows parallel access to the storage.
Applications, or users using the NSS command line utilities, often provide the database storage location using a simple directory path parameter. Some might not be aware, or forget, that the parameter can be prefixed with a type modifier, either "dbm:" or "sql:".
As a result, when not providing this parameter, the file format used will be the fragile DBM file format. This is particuarly problematic, if a user attempts to modify the NSS storage using command line tools, while another process, such as a daemon, is running concurrently, which also accesses the same database in the DBM file format. This often results in corrupted database storage, which cannot be recovered.
By changing the default, all applications that currently use the DBM file format, will automatically be migrated to the SQL file format. NSS has the ability to discover if a storage location (a directory) contains the DBM file format. If configured to use the modern SQL format, NSS will automatically perform a one-time conversion from the DBM to the SQL format.
The same applies to the NSS command line utilities. If the NSS library default is changed to SQL, the NSS tools will also trigger the one-time conversion, or access the already converted files.
== Scope ==
* Proposal owners:
A small downstream patch needs to be applied to the NSS library package, which changes the library default.
* Other developers:
It's up to developers of NSS applications, if they accept the new default and an automatic conversion, or if they prefer to continue to use the classic DBM storage format. Although not recommended, developers can easily do so, by adding a "dbm:" prefix to the storage parameter they provide to NSS at NSS library initialization time.
* Release engineering: [1]
No help should be necessary. No mass rebuild necessary.
* Policies and guidelines: N/A
* Trademark approval: N/A
[1] https://pagure.io/releng/issue/6883
Thanks, Jaroslav
TL;DR: The change originally planned for Fedora 27 will now be done for Rawhide Fedora 28, probably tomorrow.
We had previously announced to change the NSS crypto library to use the new sql file format by default. Please see the attached message for the technical details.
Because of blocker bugs, it wasn't done for Fedora 27. During the past months the known bugs in NSS were fixed, and the Firefox/Mozilla code was improved. I think now we're ready to make the change for Rawhide.
If I understand correctly, at the current time of the development phase of Rawhide Fedora 28, it isn't necessary to go through a formal process to make this change.
The new tracking bug is: https://bugzilla.redhat.com/show_bug.cgi?id=1496560
The following is noteworthy, when using NSS with the sql file format:
- Performance is slightly reduced. The old dbm storage didn't use any locking mechanism, and therefore it was easy to get corrupted storage when accessing the files in parallel by more than one application. The new sqlite storage can safely be used by multiple applications in parallel, but this has a performance cost.
- NSS databases (old and new) can be protected with a password. Previously, some modification operations could be performed without unlocking the database. When using the new sql storage, NSS more strictly requires the user to log in to (unlock) the database prior to performing modifications. For example, when modifying the trust settings of a CA certificate with certutil, it will be necessary to provide the database password.
- With sql databases, NSS is more strict with half-initialized databases. In the early years of the Netscape/Mozilla era, in order to support some application/browser level functionality, it had been necessary to distinguish between "half initialized state: no password ever set on the database, not even an empty password" and "fully initialized state: password set on the database, even if it's just the empty string". We believe this state is no longer required, certutil and modutil no longer create half-initialized databases. Until recently, NSS databases created by Firefox/Thunderbird etc. were still in this half-initialized state. There might be other applications that did this. When using a database in the half-initialized state, some database operations could fail, with similar symptons as not having logged in to the database, even after databases are (automatically) migrated to the sql format (the half-initialized state is kept). When experiencing such issues, it will be necessary to explicitly set a password on the database, even if it's just the empty string.
In order to adjust for these properties, Firefox has been changed to always initialize NSS databases with an empty password.
In addition, Firefox has been changed to prompt the user for the NSS database password (Firefox calls it the master password) if necessary, prior to e.g. trust database modificiations.
These recent changes will be contained in NSS 3.34, Firefox 58 and Thunderbird 59, which aren't released yet.
In order to allow us testing the new sql database file format with Fedora Rawhide as soon as possible, the changes have been backported to the versions currently used in Rawhide: NSS 3.33, Firefox 57 beta, Thunderbird 52.4. The packages were built yesterday.
If other applications based on Mozilla Gecko, e.g. like Iceowl or Icecat, would like to avoid that users run into these potential failures, they could pick up the same backported patches. Please look at the commits for Firefox and Thunderbird to find them, or see the dependency list of bug 1496560.
Please let us know if you experience problems. (Ideally, please file a bug against the "nss" component and CC kengert@rh, hkario@rh, rrelyea@rh, dueno@rh.)
Thanks Kai
On 25 October 2017 at 10:45, Kai Engert kaie@kuix.de wrote:
TL;DR: The change originally planned for Fedora 27 will now be done for Rawhide Fedora 28, probably tomorrow.
We had previously announced to change the NSS crypto library to use the new sql file format by default. Please see the attached message for the technical details.
Because of blocker bugs, it wasn't done for Fedora 27. During the past months the known bugs in NSS were fixed, and the Firefox/Mozilla code was improved. I think now we're ready to make the change for Rawhide.
If I understand correctly, at the current time of the development phase of Rawhide Fedora 28, it isn't necessary to go through a formal process to make this change.
There's always process if something is high enough level to be considered a "Change"
Please follow the appropriate process to have this included as a system level change for F28.
On 25.10.2017 15:22, James Hogarth wrote:
There's always process if something is high enough level to be considered a "Change"
Please follow the appropriate process to have this included as a system level change for F28.
Ok. The change was previously accepted for Fedora 27.
I've edited and updated the previous trackers: - https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql#Current_statu... - https://bugzilla.redhat.com/show_bug.cgi?id=1474771 - https://pagure.io/releng/issue/6883
Will this be sufficient to restart the change acceptance process for Fedora 28?
Thanks, Kai
On 25 October 2017 at 16:16, Kai Engert kaie@kuix.de wrote:
On 25.10.2017 15:22, James Hogarth wrote:
There's always process if something is high enough level to be considered a "Change"
Please follow the appropriate process to have this included as a system level change for F28.
Ok. The change was previously accepted for Fedora 27.
I've edited and updated the previous trackers:
https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql#Current_statu...
Will this be sufficient to restart the change acceptance process for Fedora 28?
Although I'm not on FESCo my advice here is to treat it as a fresh change and for the large part ignore the missed target of F27.
You're fortunate that Change is already written up so you're spared redoing that effort and also that the contingency is known, and in this case tested.
If steps are skipped then it risks other Changes in future adopting the "we tried it for N-1 and it was approved in N-1 so we're good" causing problems... and besides risks to be aware of will change from release to release.
If we follow process each time, whether it was a missed target previously or not, we ensure that there are no slip ups with missing documentation or knock on effects that have not been considered.
In general if you have to ask the question "Will this be sufficient to restart the change acceptance process for Fedora Next?" then realistically the only suitable response is "check with FESCo, which you'd have to do with any Change anyway".
On 25.10.2017 17:16, Kai Engert wrote:
https://fedoraproject.org/wiki/Changes/NSSDefaultFileFormatSql#Current_statu...
FYI, this had been re-approved on Nov 6 in bug 1474771.
The change was applied to Rawhide on Nov 7. https://bugzilla.redhat.com/show_bug.cgi?id=1496560
So far I haven't seen any feedback, neither negative nor positive.
Kai
-
James Hogarth -
Jaroslav Reznik -
Kai Engert