Hello everyone,
we are aiming to remove libdb-java package from Fedora-rawhide, as we are
currently preparing for jdk update from jdk-1.8 to jdk-11 in Fedora
rawhide. The problem is that we are unable to rebuild this package with
jdk-11. It is still possible to "hack" it and rebuild it with jdk-1.8, but
that can cause unexpected runtime behaviour according to JVM-11, which will
soon be default in Fedora-rawhide.
There seems to be no packages, which depend directly to libdb-java and
upstream does not support version 5.3.28 anymore.
If anyone has any reasons why this should not be made, or someone is
currently active user of this JDBC connector, please leave a comment with
your opinion in the tracker [1] mentioned below.
There is also an existing tracker for deprecating libdb in Fedora [2], so
this can be understand as a first step.
Additional info about jdk-11 here [3].
Best regards,
Ondrej Dubaj
Associate Software Engineer
Red Hat
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1846398
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1834842
[3]
https://fedoraproject.org/wiki/Changes/Java11#Intermediate_step_build_with_…
Voting in the Fedora 32 elections is now open. Go to the Elections app
to cast[1] your vote. Voting closes at 23:59 UTC on Thursday 11 June.
Don't forget to claim your "I Voted" badge when you cast your ballot.
Links to candidate interviews are in the Elections app and on the
Community Blog[2].
[1] https://elections.fedoraproject.org/
[2] https://communityblog.fedoraproject.org/fedora-32-elections-voting-now-open/
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
https://fedoraproject.org/wiki/Changes/CompilerPolicy
== Summary ==
Fedora has historically forced packages to build with GCC unless the
upstream project for the package only supported Clang/LLVM. This
change proposal replaces that policy with one where compiler selection
for Fedora follows the package's upstream preferences.
== Owner ==
* Name: Jeff Law
* Email: law(a)redhat.com
== Detailed Description ==
The main goal here is to make selection of the compiler to build a
package flow from upstream in an effort to preserve our development
resources. In cases where there is no upstream the Fedora package
maintainer should be allowed to make the compiler choice for the
package. For packages where upstream does not have a strong compiler
preference, we should (for now) stick with the status quo to avoid
unnecessary churn.
== Benefit to Fedora ==
This change allows packages to be built with whatever compiler the
upstream project recommends/supports (so long as that compiler is in
Fedora). Thus, Fedora package owners no longer need to spend time
making a package work with GCC if the upstream project is using
Clang/LLVM.
An obvious example is Firefox. Upstream, the Firefox project builds
primarily with Clang/LLVM. Yet we force the Fedora package owner to
find and fix issues building with GCC then either carry those custom
fixes forward in Fedora or negotiate with upstream to get those
changes upstreamed. While this process can be helpful in finding
non-portable code, this is ultimately a poor use of the packager's
time.
Additionally Fedora loses the benefit of the testing provided by other
distributions where Firefox is compiled in the same way as the
upstream project -- when issues arise the Fedora team must consider
the possibility that the problem is due to using GCC instead of
Clang/LLVM or the patches to make that possible. Again, this is a
poor use of Fedora developer's time.
In the immediate term this change in policy only affects a few
packages (Firefox, Chromium and perhaps a few others). The benefit
will likely expand over time.
== Scope ==
* Proposal owners:
Update the Fedora Packaging Guidelines to reflect the policy change.
* Other developers:
Developers working with packages where upstream builds with
Clang/LLVM, but Fedora policy has forced the package to build with GCC
should update their packages to build with Clang/LLVM, dropping all
Fedora specific changes that were necessary to make the package build
with GCC.
Firefox and Chromium are known to be impacted, there may be other
packages. While there is no specific timeframe where this would need
to be accomplished as the existing packages are already building with
GCC, getting the conversion done earlier rather than later seems
beneficial. Failure to identify all the impacted packages is not a
catastrophic failure, we just fail to reap the benefits in the
immediate term.
* Release engineering: [https://pagure.io/releng/issue/9503] (a check
of an impact with Release Engineering is needed)
I do not believe this change requires any coordination with release
engineering. No mass rebuild is required.
* Policies and guidelines:
Yes, the packaging guidelines certainly need to be updated for this
feature. That can happen as soon as the exact text is agreed upon.
Development work on packages such as Firefox or Chrome can happen as
soon FESCO agrees to the change while the final packaging guideline
text is hammered out.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
This should not require any configuration changes or data migration,
nor should it change existing functionality.
== How To Test ==
For packages where the compiler should change, the package owner will
need to update the spec file and build the package with the new
compiler. Once done, the package's testsuite should be run (if it's
not part of the standard build process).
In general, I would think the standard Fedora QE work should be
sufficient here, perhaps with a bit of additional attention to the
affected packages. The graphical nature of some of the affected
packages like Firefox and Chrome will make testing difficult on some
of Fedora's architectures.
== User Experience ==
Users should not notice any change.
== Dependencies ==
One the policy change is made a set of packages will need to be
updated. Firefox and Chrome are known to be affected. There may be
others. It seems like the Fedora package owners are in the best
position to know if their package is affected by the policy change.
It is useful to remember that the packages as they exist today still
work. Therefore if a package which should change is not identified
now, it will continue to work. Similarly if the package owner does
not have the time to implement the change right now, the existing
package will continue to work.
== Contingency Plan ==
The backup plan is trivial. We can keep the current policy in place.
If the policy change is approved, but a particular package has not
switched to the upstream preferred compiler, the package can continue
to build with GCC until the package maintainer has the time to make
the change for their particular package.
Failure to convert any particular package should not create any
downstream or distribution wide delays.
* Contingency mechanism:
It seems like we could institute the policy change anytime we choose.
But it also seems like once the policy change is in place, packages
that are going to convert should do so before beta freeze.
* Contingency deadline: Fedora can ship with this feature in an
incomplete state.
* Blocks release? No
* Blocks product? N/A
== Documentation ==
Several years ago Red Hat's tools team championed for Fedora policy to strongly
discourage the use of LLVM/Clang for package building. Exceptions were made for
packages that could only be built with Clang/LLVM:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#compiler
At that point in history Red Hat had no Clang/LLVM engineers or expertise. In
fact, the LLVM packages were actually maintained by an engineer on the desktop
team (they had a hard requirement for llvm-pipe, so they got to own the
Clang/LLVM bits). The policy essentially was a risk management strategy for
Fedora.
Times have changed and as a result we should revisit that Fedora policy.
The Red Hat tools team believes that LLVM/Clang and GCC should be
considered equals from
a Fedora policy standpoint. Selection of one toolchain over the
should should be
driven by the upstream project's preferences not by Fedora specific policy.
What that means in practice is that if the project upstream prefers Clang/LLVM,
then Fedora should in turn be using Clang/LLVM to build those packages. As a
concrete example, let's consider Chromium.
Chromium upstream has been building with Clang/LLVM for several years.
Yet policy
has forced Fedora package owners to shoulder a significant burden to make it
build with GCC. Furthermore, Fedora does not get as much benefit at it could by
forcing Chromium to be built with GCC since most other instances are built with
Clang/LLVM.
By changing policy Fedora package maintainers no longer have to waste
time trying
to make Chromium build/work with GCC and Fedora gains additional "many eyes" and
"many users" benefits by relying on the same tools to build Chrome as the
upstream developers and other distributions.
Additionally, if an upstream project currently uses GCC, but switches to
Clang/LLVM (or vice-versa), then the package in Fedora should switch
in a similar
manner. The only policy restriction should be that the compiler must
exist in Fedora.
In some ways this means there is no "default" compiler for Fedora. The default
is whatever the upstream project supports/recommends. However, there are
probably many packages with upstreams that are ambivalent about their compiler
choice. For those packages I would recommend we keep the status quo at the
current time. For a package with a dead upstream, the Fedora packager should be
able to select the compiler they want to use for the package.
== Release Notes ==
This change should not have any end user impacts nor does it strictly
require a release note. However, a short release note could be
written if FESCo or the development community thinks it would be
useful.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
https://fedoraproject.org/wiki/Changes/SwapOnZRAM
== Summary ==
Swap is useful, except when it's slow. zram is a RAM drive that uses
compression. Create a swap-on-zram during start-up. And no longer use
swap partitions by default.
== Owner ==
* Name: [[User:chrismurphy| Chris Murphy]]
* Email: chrismurphy(a)fedoraproject.org
== Detailed Description ==
==== zram Basic function ====
The zram† device, typically <span style=color:brown>/dev/zram0</span>,
has a size set at create time during early boot, by zram-generator†
per its configuration file. The memory used is not preallocated. It's
dynamically allocated and deallocated, on demand. Due to compression,
a full <span style=color:brown>/dev/zram0</span> uses half as much
memory as its size.
The <span style=color:brown>/dev/zram0</span> behaves like any other
block device. It can be formatted with a file system, or mkswap, which
is the intention with this change proposal.
The system will use RAM normally up until it's full, and then start
paging out to swap-on-zram, same as a conventional swap-on-drive. The
zram driver starts to allocate memory at roughly 1/2 the rate of page
outs, due to compression. But, there is no free lunch. This means
swap-on-zram is not as effective at page eviction as swap-on-drive,
the eviction rate is ~50% instead of 100%. But it is at least an order
of magnitude faster than drive based swap.
zram has about 0.1% overhead or ~1MiB/1GiB. If the workload never
touches swap, this overhead is the sole cost. In practice when not
used at all, feature owner has experienced ~0.04% overhead.
Example: A system has 16 GiB RAM. The proposed defaults suggest the
<span style=color:brown>/dev/zram0</span> device will be 4 GiB. If the
workload completely fills up swap with 4 GiB of anonymous pages,
what's happened? The <span style=color:red>zramctl</span> command will
display the true compression ratio. If 2:1 is really obtained, it
means 4GiB swap data is compressed to 2GiB. Therefore 2GiB is the
actual RAM usage, and is also the net effective eviction. i.e. 4 GiB
anonymous pages are evicted, but are then compressed and pinned into 2
GiB RAM, for a net memory savings of 2 GiB.
†</br >
[https://www.kernel.org/doc/Documentation/blockdev/zram.txtkernel.org zram.txt]
[https://github.com/systemd/zram-generator Github zram-generator project]
==== Overview of the Feature ====
Using swap is a good idea†, but no one likes it when it's slow.
Anaconda and Fedora IoT have been using swap-on-zram by default for
years. This builds on their prior effort.
There are three components to the change:
# Install systemd rust-zram-generator† package. This does not enable
swap-on-zram, it only makes the generator available.</br >
# Install a default zram-generator configuration. When present,
swap-on-zram is set-up during startup.</br >
# Do not create swap partition/LV with default installations.
This proposal aims to apply all three, for all Fedora editions and
spins, by default.
It further aims to apply the first two, for upgrades and custom installations.
It might be useful to only make the generator available (1), should an
edition/spin wish to opt out, or as a fallback if applying the feature
to upgrades fails to withstand scrutiny.
†</br >
There is a tl;dr section at the top. Highly recommend reading the
whole article. [https://chrisdown.name/2018/01/02/in-defence-of-swap.html
In defence of swap: common misconceptions]
==== Default zram device configuration: ====
During startup, create a zram device <span
style=color:brown>/dev/zram0</span>, with a size equal to 50% RAM, but
capped† to 4 GiB, and with a higher than typical swap priority†.
These values seem reasonably conservative, and are based on prior work
in Fedora. Anaconda sets swap-on-drive sized to 50% RAM in the no
hibernation case, common outside x86. Fedora IoT's implementation also
sets swap-on-zram size to 50% RAM.
†</br >
[https://github.com/systemd/zram-generator/issues/10 RFE: should be
able to set a cap on zram device size #10]
[https://github.com/systemd/zram-generator/issues/8 RFE: should set priority #8]
==== Default installer behavior ====
The installer is currently responsible for creating a swap-on-drive
device. This will be dropped. The zram-generator + configuration file
will trigger the setup and activation of swap-on-zram. This means
hibernation isn't possible, even on systems that could support it.
Please see [https://pagure.io/fedora-workstation/blob/master/f/hibernationstatus.md
Supporting hibernation in Workstation edition] for much more detailed
information, including why it's increasingly likely hibernation isn't
possible anyway, and a path to improving hibernation support.
==== Custom/Advance partitioning installer behavior ====
The user can add swap using Custom partitioning at install time. This
is swap-on-drive. And the installer will also include the <span
style=color:red> resume=UUID </span> kernel parameter for this swap
device. No change in behavior here.
Since swap-on-zram is still enabled by default, there will be two
swaps: swap-on-zram, and swap-on-drive. The swap-on-zram will have
higher priority, thus being favored over drive based swap. The kernel
is smart enough to know it can't hibernate to a zram device, and will
instead use drive based swap.
==== How can it be disabled? ====
Immediately:</br >
<span style=color:red>swapoff /dev/zram0</span>
Permanently:</br >
<span style=color:red>rm /etc/systemd/zram-generator.conf</span>
== Feedback ==
==== You're enabling it on upgrades? ====
That's the current plan. As a technical matter, feature owner is
confident this feature will improve the experience of all users
regardless of configuration. As a non-technical matter, it's
recognized that (a) ''hey pal, you're messing with my customizations,
not cool!'' and (b) ''swap always stinks, I don't care if it has a 'Z'
in the name!'' may need more convincing.
There are possible risks.
* Workloads that expect full use of memory, and depend on 100% page
eviction. These may run slower if they really need full use of memory,
but some memory is used for the zram device instead. Such workloads
might favor zswap.
* Workloads with low compressible pages. In the worst case, this means
unnecessary work merely moving pages around.
* Workloads with memory full, and hibernation. Hibernation is already
stressful to memory-management subsystem and prone to bailing out in
such cases. The swap-on-zram will be favored for evictions in the
attempt to free memory to create the hibernation image. It could
increase instances of hibernation entry failure. This isn't a crash,
it just means the attempt doesn't succeed, and the system resumes
operation instead of hibernating.
While possible, it's difficult to estimate their probability. But this
is a significant consideration in the conservative default zram size.
Users can easily increase zram size as needed for their use case,
simply by editing <span
style=color:red>/etc/systemd/zram-generator.conf</span> and the change
takes effect at next boot.
==== Why systemd zram-generator? ====
It's the most upstream implementation to date, is fast and
lightweight. The zram-generator uses existing systemd infrastructure
to setup the zram block device, format it as swap, and swapon - all
during early boot. It's very similar in behavior to fstab-generator,
gpt-auto-generator, and cryptsetup-generator†.
Converging on one implementation avoids user confusion. And while the
alternatives are nice and work fine, a systemd generator is
particularly well suited for this use case compared to a systemd
service unit.†
Also, it's an reference implementation of a system generator written in Rust.
†</br >
[https://www.freedesktop.org/software/systemd/man/systemd.generator.htmlfreedesktop.org About systemd generators.]</br >
[https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org…
devel@ ''Re: swap-on-zram by default'' Zbigniew Jędrzejewski-Szmek,
systemd zram-generator author/maintainer]
==== Why not a bigger zram device? ====
The main idea of being conservative is to address concerns about
upgrades. It's possible some workloads will have less compressible
data. Hence, not going with <span style=color:brown>/dev/zram0</span>
sized to 100% of RAM at this time. Even a <span
style=color:brown>/dev/zram0</span> of 200% RAM is not unreasonable
*if* the compression ratio is at least 2:1. However, it's possible a
system can get "stuck" in a kind of swap thrashing similar to
conventional swap-on-drive, except it's CPU and memory bound, rather
than IO bound. Feature owner thinks it's better to just oom, instead
of getting overly aggressive with the zram device size.
Conversely it's possible to be too conservative with the size, and
result in more instances of OOM kill. If applying the feature to
upgrades is rejected, it's probably reasonable to increase the cap to
~8GiB. Of course more feedback and testing is needed, and it will be
taken into consideration.
Note that the kernel zram doc says an excessively sized zram device
does come with overhead. Users's can increase the size easily
post-install, a capability they don't easily have with swap-on-drive.
The goal for Fedora 33 is a default that's useful and safe for the
vast majority of use cases.
==== Why not zswap? ====
Zswap† is a similar idea, but with a totally different implementation.
It is swap specific, uses a RAM cache, and requires a conventional
swap partition existing already. It might be true certain workloads
are better suited for using zswap. But swap-on-zram depends only on
volatile storage. This is simpler and it's more secure. Whereas zswap
"spills over" into swap-on-drive and will leak user data if that swap
device isn't encrypted. Some workloads may do better with zswap, and
it's a valid future feature for a new generator, or possibly extend
zram-generator to support it via the configuration file. Maybe the
generator could favor zswap when swap-on-drive already exists; and
fallback to swap-on-zram?
†</br >
[https://www.kernel.org/doc/Documentation/vm/zswap.txtkernel.org zswap.txt]
== Benefit to Fedora ==
* significantly improves system responsiveness, especially when swap
is under pressure;
* more secure, user data leaks into swap are on volatile media;
* without swap-on-drive, there's better utilization of a limited
resource: benefit of swap without the drive space consumption;
* complements on-going resource control work, including earlyoom;
* further reduces the time to out-of-memory kill, when workloads exceed limits;
* improves performance for both "no swap" and "existing swap" setups;
== Scope ==
* Proposal owners:
** add zram-generator package to comps and kickstarts as appropriate
** obsolete zram package (used by Fedora IoT)
** means of per edition/spin configurations, if needed
** coordinate a test day
* Other developers:
**Anaconda are agreeable to deprecating their built-in implementation
in favor of swap-on-zram
**RFE's for zram-generator: users are not worse off if they don't
happen. Open request for help, to make it possible. It's much
appreciated.</br >
[https://github.com/systemd/zram-generator/issues/10 RFE: should be
able to set a cap on zram device size #10]</br >
[https://github.com/systemd/zram-generator/issues/8 RFE: should set priority #8]
* Release engineering: [https://pagure.io/releng/issues #9495]
* Policies and guidelines: N/A
* Trademark approval: N/A
== Upgrade/compatibility impact ==
Add Supplements:fedora-release-common to zram-generator to pull it in
on upgrades.
Existing systems without swap will have swap-on-zram enabled.
Existing systems with swap-on-drive, will also have swap-on-zram
enabled (two swap devices), with higher priority for the zram device.
Existing swap-on-drive will not be removed.
'zram' package contains zram-swap.service and associated bash scripts,
and is currently used by Fedora IoT and ARM spins. It will be
obsoleted to avoid conflicting/duplicative swap-on-zram
implementations.
== How To Test ==
Any hardware. Any version of Fedora.
# dnf install zram-generator
# cp /usr/share/doc/zram-generator/zram-generator.conf.example
/etc/systemd/zram-generator.conf
# Edit the configuration
# Reboot
# Check that swap is on a zram device: zramctl, swapon
# Detailed check: journalctl -b -o short-monotonic | grep 'swap\|zram'
# Check that priority is higher than existing swap if two or more are
listed. ## (Enhancement is needed for this.)
Suggested configuration file values:</br >
<span style=color:red>[zram0]</span></br >
<span style=color:red>memory-limit = none</span></br >
<span style=color:red>zram-fraction = 0.5</span></br >
Feel free to run your usual workloads more aggressively or in
parallel. Suspend-to-RAM and suspend-to-drive are expected to continue
to work too (or at least hit all the same bugs as without zram being
used).
Also, you can see the actual compression ratio achieved with the
following command:</br >
<span style=color:red> zramctl </span>
==== Test Day ====
[https://pagure.io/fedora-qa/issue/632 QA: SwapOnzram Test Day] to
discover edge cases, and tweak the default configuration if necessary
to establish a good one-size-fits all approach.
== User Experience ==
The user won't notice anything displeasing. If their usual workload
causes them to dread swap thrashing, they'll be surprised that
thrashing doesn't happen. The user might get curious if they don't
find a swap entry in /etc/fstab. Or if they 'swapon' and see swap
pointing to <span style=color:brown>/dev/zram0</span> instead of a
drive partition or LV.
== Dependencies ==
N/A
== Contingency Plan ==
* Contingency mechanism: Don't ship the generator = big hammer, but
easy. Preferable to ship the generator, but only selectively ship
configuration files = scalpel, pretty easy.
* Contingency deadline: Beta freeze
* Blocks release? No.
* Blocks product? No.
== Documentation ==
Consider adding a hint in an /etc/fstab comment? There is no man page
for this, and the documentation is also minimal, besides what's in
this feature proposal. It's an open question how the user should get
more information on how to configure and tweak it. But then, they
don't have that for swap today either. There's just institutional
knowledge.
Hence, a strong test day, with a lot of people and press coverage of
the feature, might help spread the word for institutional knowledge
changes coming.
Ideas welcome.
== Release Notes ==
Pending feedback and test day.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
Greetings.
As previously announced, fedoraproject is moving many of it's servers
from one datacenter (phx2 near phoenix, arizona, usa) to another (iad2:
near arlington, virginia, usa).
As we move from the old datacenter to the new, we will have a temporary
reduction in capacity. The new datacenter has a smaller, less-redundant,
lower-capacity version of our infrastructure. Over the next two weeks,
we will migrate services to it so that we can finish moving out of the
old datacenter.
After everything is moved from the old datacenter, many of the servers
there will be shipped to the new datacenter and then re-added to bring
us back to full redundency and capacity.
Out detailed checklist for these migrations is available at
https://hackmd.io/@fedorainfra2020/rJpsA4FLL
To summarize what we are moving when:
2020-06-03 wed: The fedoraproject master mirrors will move to IAD2. A
very small outage may be noticed as dns changes. There may be some
mirroring slowdowns as we work out bugs.
2020-06-04 thu: Our internal ansible control host and the fedoraproject
wiki will move. The wiki will be down for a few hours.
2020-06-05 fri: Our meeting minutes archive
(https://meetbot.fedoraproject.org) and our freenode irc bot (zodbot).
These two services will see a hour outage or less.
2020-06-07 sun: We will pause for the next week adding new packages and
unretiring packages to avoid problems.
2020-06-08 mon: Our fedora-messaging bus and gateways to it
(github2fedmsg, bugzilla2fedmsg), mirrormanager, product definition
center (pdc), and our identity and authentication systems. Messages over our
message bus may be slow or missing and users may be unable to login at
various times as we migrate services over.
Additionally, we will be stopping services that will not be back until
later in the month.
These include:
* Fedocal
* Badges
* Nuancier
* koschei
* simple-koji-ci
* All staging services (*.stg.fedoraproject.org)
2020-06-09 tue: The build and packaging ecosystem. This includes koji,
src.fedoraproject.org, osbs, odcs, container registries, bodhi (updates
system). During this day maintainers should avoid builds/updates if at
all possible as they may or may not work at various times.
2020-06-10 wed: Various small apps (mdapi, anitya, waiverdb, greenwave,
etc), mailman/lists.fedoraproject.org, and our datagrepper/datanommer
services. Mailing lists will be down for several hours as data is
migrated. Datagrepper will be down for most of the day as it's database
is moved. Other services will be down for short amounts of time while
they are moved.
2020-06-11 thu: Various small site building apps (docs building, fedora
websites building, reviewstats, blockerbugs) and elections will be
moved. elections will be up until the currently running elections
complete. (GO VOTE! https://elections.fedoraproject.org)
2020-06-12 fri: Catch up and fix issues day, along with re-enabling
package unretirements/new packages, and other 'paused' items.
The week after this servers will be shipped and the week after that we
expect to start setting them up and getting them re-added. During this
time, we may have to make further changes to what services are available
in order to deal with load changes.
If you have any questions or concerns, please file an infrastructure
ticket ( https://pagure.io/fedora-infrastructure) or come talk to us in
#fedora-admin on irc.freenode.net.
Finally, I'd like to ask everyone to be patient as we do this move. I
know that it's painful when you are unable to contibute something when
you have time to do so, but rest assured that we are trying to migrate
things as quickly and smoothly as we can.
Thanks.
kevin