bigdata June 2013

bigdata@lists.fedoraproject.org

4 participants
11 discussions

Re: Bringing Hadoop to Fedora
by Tim St. Clair 18 Jun '13

18 Jun '13

Hi Arun - It's been a week so I figured I would follow up on the status regarding the patch-set listed below. Cheers, Tim ----- Original Message ----- > From: "Arun C Murthy" <acm(a)hortonworks.com> > To: yarn-dev(a)hadoop.apache.org > Cc: common-dev(a)hadoop.apache.org, mapreduce-dev(a)hadoop.apache.org, hdfs-dev(a)hadoop.apache.org, dev(a)bigtop.apache.org > Sent: Tuesday, June 11, 2013 12:54:11 PM > Subject: Re: Bringing Hadoop to Fedora > > Tim, > > This is great! I'll take a look at some of these patches, welcome! > > Arun > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote: > > > Greetings Hadoop Development Community, > > > > As some of you are aware, we've been looking to bring the upstream 2.X > > series into the Fedora channels. We believe that there are several > > benefits that this can bring to the community: > > > > - Updated & streamlined rpm packaging > > - Deeper level of system integration > > - System managed dependencies (security + defect tracking) > > - Proving ground for OpenJDK7 > > - Greater exposure for the community (both adoption and defect tracking) > > ... > > > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), > > we've come across some build issues as they relate to Fedora packaging > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these > > issues (listed below) are very minor dependency changes, and we were > > wondering if there are folks who would be willing to review. > > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594 > > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610 > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611 > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613 > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 > > (uber deprecated) > > > > There are other patches we have in the queue but they still need to be > > flushed out, and in an effort to follow the KISS principles we figure this > > would be a good start. > > > > Cheers, > > Tim > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > >

1 0

Fwd: Bringing Hadoop to Fedora
by Tim St. Clair 17 Jun '13

17 Jun '13

Folks, As a point of note, all tracking details have been updated on the wiki: https://fedoraproject.org/wiki/Features/Hadoop#Upstream_Patch_Tracking I plan on following up with upstream this week re: the message below. Cheers, Tim ----- Forwarded Message ----- > From: "Arun C Murthy" <acm(a)hortonworks.com> > To: yarn-dev(a)hadoop.apache.org > Cc: common-dev(a)hadoop.apache.org, mapreduce-dev(a)hadoop.apache.org, hdfs-dev(a)hadoop.apache.org, dev(a)bigtop.apache.org > Sent: Tuesday, June 11, 2013 12:54:11 PM > Subject: Re: Bringing Hadoop to Fedora > > Tim, > > This is great! I'll take a look at some of these patches, welcome! > > Arun > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote: > > > Greetings Hadoop Development Community, > > > > As some of you are aware, we've been looking to bring the upstream 2.X > > series into the Fedora channels. We believe that there are several > > benefits that this can bring to the community: > > > > - Updated & streamlined rpm packaging > > - Deeper level of system integration > > - System managed dependencies (security + defect tracking) > > - Proving ground for OpenJDK7 > > - Greater exposure for the community (both adoption and defect tracking) > > ... > > > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), > > we've come across some build issues as they relate to Fedora packaging > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these > > issues (listed below) are very minor dependency changes, and we were > > wondering if there are folks who would be willing to review. > > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594 > > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610 > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611 > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613 > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 > > (uber deprecated) > > > > There are other patches we have in the queue but they still need to be > > flushed out, and in an effort to follow the KISS principles we figure this > > would be a good start. > > > > Cheers, > > Tim > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > >

1 0

Re: Package Review Requests for Hadoop dependencies
by gil 12 Jun '13

12 Jun '13

Il 12/06/2013 19:04, Björn Esser ha scritto: > Am Mittwoch, den 12.06.2013, 12:54 -0400 schrieb Tim St Clair: >> Many thanks Björn! >> >> It looks like we are green on dependencies! >> "for now" ;-) >> >> Cheers, >> Tim > Hi Tim! > > You're welcome! If there are any further reviews or pkgs needed during > intergration of hadoop, just feel free to ask me for "fixing" this. > > Cheers, > Björn thanks very thanks gil

1 0

Re: Package Review Requests for Hadoop dependencies
by Tim St. Clair 12 Jun '13

12 Jun '13

Many thanks Björn! It looks like we are green on dependencies! "for now" ;-) Cheers, Tim ----- Original Message ----- > From: "Björn Esser" <bjoern.esser(a)gmail.com> > To: "gil" <puntogil(a)libero.it> > Cc: devel(a)lists.fedoraproject.org > Sent: Tuesday, June 11, 2013 1:35:50 PM > Subject: Re: Package Review Requests for Hadoop dependencies > > Am Dienstag, den 11.06.2013, 20:16 +0200 schrieb gil: > > hi > > yes , many thanks Björn > > have some trouble with boost with bookkeeper > > > > data.cpp:249:3: error: 'lock_guard' is not a member of 'boost' > > boost::lock_guard<boost::mutex> lock(mutex); > > ^ > > data.cpp:249:33: error: expected primary-expression before '>' token > > boost::lock_guard<boost::mutex> lock(mutex); > > ^ > > data.cpp:249:45: error: 'lock' was not declared in this scope > > boost::lock_guard<boost::mutex> lock(mutex); > > > > channel.cpp:705:43: error: 'shared_dynamic_cast' is not a member of 'boost' > > boost::shared_dynamic_cast<AsioSSLDuplexChannel>(shared_from_this()), > > ^ > > channel.cpp:705:90: error: expected primary-expression before '>' token > > boost::shared_dynamic_cast<AsioSSLDuplexChannel>(shared_from_this()), > > ^ > > channel.cpp: In member function 'void > > Hedwig::AsioSSLDuplexChannel::sslShutdown()': > > channel.cpp:752:42: error: 'shared_dynamic_cast' is not a member of 'boost' > > boost::shared_dynamic_cast<AsioSSLDuplexChannel>(shared_from_this()), > > ^ > > channel.cpp:752:89: error: expected primary-expression before '>' token > > boost::shared_dynamic_cast<AsioSSLDuplexChannel> > > (shared_from_this()), > > > > boost problems > > any ideas? > > thanks > > Which boost version resp. which fedora-release? I think it would be the > best to join #fedora-devel irc and and have our discussion there, > wouldn't it? > > -- > devel mailing list > devel(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel

1 0

Fwd: gmaven
by Peter MacKinnon 12 Jun '13

12 Jun '13

With respect to JSP compilation in Hadoop, I think we should stay the course through Fedora 19 (https://koji.fedoraproject.org/koji/buildinfo?buildID=396972) but this is something we'll definitely want to track. We'll have two options when we need to act (all things being equal): 1. try using the groovy+ant workaround as you suggested at https://fedoraproject.org/wiki/Features/Hadoop#Packager_tips 2. pivot away from the original codehaus jspc plugin to the jetty maven plugin which should be available with Jetty 9. This will then have to be presented to Apache upstream as a viable alternative, but before that we need to resume development on a stable and released F19 first. http://www.eclipse.org/jetty/documentation/current/jetty-jspc-maven-plugin.… https://koji.fedoraproject.org/koji/rpminfo?rpmID=4037794 \Pete -------- Original Message -------- Subject: gmaven Date: Wed, 12 Jun 2013 00:40:02 +0200 From: gil <puntogil(a)libero.it> To: Pete MacKinnon <pmackinn(a)redhat.com> hi gmaven seem at the ens of life see https://jira.codehaus.org/browse/GMAVEN-109 and http://docs.codehaus.org/display/GROOVY/Groovy-Eclipse+compiler+plugin+for+… http://docs.codehaus.org/display/GROOVY/Groovy-Eclipse+compiler+plugin+for+… and https://github.com/groovy/groovy-eclipse/blob/master/extras/groovy-eclipse-… https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries regards gil

1 0

hadoop.spec && systemd && cgroups.
by Tim St. Clair 12 Jun '13

12 Jun '13

Rob - We will "eventually" need to provide integration points inside of the systemd scripts, and hadoops cgroup config, so out of the gate it plays nicely. Systemd config for cgroups: http://0pointer.de/blog/projects/resources.html ^^ There is a full list of knobs here: http://0pointer.de/public/systemd-man/systemd.directives.html However, it may require translation as it appears they have recreated names for existing cgroup params: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux… We will then need to match up the config settings ref'd here: https://issues.apache.org/jira/browse/YARN-160 & here (search for cgroup): http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-yarn/hadoop-yarn-common/y… Cheers, Tim

1 0

Re: Packaging Hadoop in the Fedora community
by Tim St. Clair 12 Jun '13

12 Jun '13

Bruno - Feel free to join the dev-lists and IRC channels(fedora-devel, fedora-bigdata), as there are numerous items we are still hashing out here (https://fedoraproject.org/wiki/Features/Hadoop), and we would be happy to coordinate. For example, getting parity across the tests is a stubborn foe. Once Hadoop we've tackled the main package we can eval and coordinate other ecosystem packages. Cheers, Tim ----- Original Message ----- > From: "Bruno Mahé" <bruno(a)bmahe.net> > To: "Matthew Farrellee" <matt(a)redhat.com> > Cc: puntogil(a)libero.it, tstclair(a)redhat.com > Sent: Wednesday, June 12, 2013 1:56:57 AM > Subject: Re: Packaging Hadoop in the Fedora community > > +Tim > > Hi Matthew and Tim, > > I noticed Tim's email on the Apache Bigtop/Hadoop mailing list > and was wondering how we could all collaborate and work together to > improve the state of Apache Hadoop (and its ecosystem) on > Fedora/CentOS/RHEL. > > Thanks, > Bruno > > On 04/26/2013 10:38 AM, Bruno Mahé wrote: > > On 04/25/2013 06:49 AM, Matthew Farrellee wrote: > >> On 04/25/2013 01:55 AM, Bruno Mahé wrote: > >>> On 04/24/2013 05:40 AM, Matthew Farrellee wrote: > >>>> Hello Big Data SIG! > >>>> > >>>> A few of us in the Fedora community have been working to package Apache > >>>> Hadoop for Fedora. The feature page is, > >>>> > >>>> https://fedoraproject.org/wiki/Features/Hadoop > >>>> > >>>> You'll find what we plan to package, our progress working through all > >>>> the dependencies, a process for how we're attacking dependencies and a > >>>> pre-Fedora repo of the packages in progress. > >>>> > >>>> It's a big effort, but will form a foundation for packaging more Hadoop > >>>> ecosystem packages for Fedora. > >>>> > >>>> We're at a point where we need a better place to engage the community > >>>> and the Big Data SIG is a natural choice. I hope you will welcome us to > >>>> the list and IRC, and maybe lend a hand! > >>>> _______________________________________________ > >>>> bigdata mailing list > >>>> bigdata(a)lists.fedoraproject.org > >>>> https://admin.fedoraproject.org/mailman/listinfo/bigdata > >>> > >>> > >>> Hi, > >>> > >>> I am not CC-ing the Bigdata SIG as to not spam the list. > >>> > >>> One of the project I work on intersects closely with this SIG. This > >>> project is Apache Bigtop (http://bigtop.apache.org/). > >>> The goal of Apache Bigtop is three folds: > >>> 1/ Provide top notch packages for Apache Hadoop related projects > >>> 2/ Provide a point of integration and testing for all these projects > >>> 3/ Provide means to reliably deploy a complete stack. > >>> > >>> So there is quite a bit of overlapping between Apache Bigtop, this SIG > >>> and your efforts. > >>> > >>> Would you see any way we could collaborate? > >>> > >>> > >>> Please, see this previous message for an explanation about Apache Bigtop > >>> with regards to this SIG and Fedora: > >>> http://lists.fedoraproject.org/pipermail/bigdata/2013-March/000012.html > >>> > >>> > >>> Thanks, > >>> Bruno > >> > >> Definitely. I hold an assumption that Bigtop is a good place to - > >> > >> 0) drive consistency across the Hadoop ecosystem > >> 1) provide deliverables that are integrated into various OS platforms > >> > >> For (0), our packaging effort is finding old and inconsistent (different > >> versions of the same) dependencies across projects. > >> > >> For (1), at a minimum I assume that the packaging work done for Fedora > >> can be contributed to Bigtop. This would provide integration with Fedora > >> at a level that does not currently exist. > >> > >> Beyond that. I'm interested in the testing framework that Bigtop > >> provides. Immediately, for the Fedora packaging effort, we need a way to > >> make sure the packages we provide aren't breaking core functionality. I > >> don't think TestDFSIO and some GridMix2 will be sufficient. > >> > >> Best, > >> > >> > >> matt > >> > > > > Hi, > > > > For (0), we are already working on this. We also spend a lot of efforts > > helping verifying that release candidates do not break their downstream > > projects. So the more people helping on that front, the merrier! > > > > For (1), we would definitely gladly accepts all contributions! And we > > would look at the best way to integrate your work. > > The sticky point may be related to Fedora specific macros. Our spec file > > works across a bunch of RPM distributions, so Fedora specific macros > > would have to be inside some ifdef or encapsulated or any other solution > > (even using Apache Bigtop as a common effort and forking the specs to > > add the Fedora specific macros). But in any case that would be > > implementation details and we can look at it whenever you feel to > > contribute your work. > > > > Apache Bigtop has some smoke tests, package tests (verify they can be > > installed, removed, upgraded, permissions are rights...) as well as > > integration tests. So I believe that could be leveraged by you whether > > or not you decide to put in common the efforts to work on packaging or > > not. If you have any issue using the tests, feel free to ping me or to > > send an email to the mailing list. > > > > > > > > Thanks, > > Bruno > >

1 0

Re: Package Review Requests for Hadoop dependencies
by Tim St. Clair 11 Jun '13

11 Jun '13

Many thanks Björn! ----- Original Message ----- > From: "Björn Esser" <bjoern.esser(a)gmail.com> > To: "Robert Rati" <rrati(a)redhat.com> > Cc: devel(a)lists.fedoraproject.org, bigdata(a)lists.fedoraproject.org > Sent: Tuesday, June 11, 2013 10:23:12 AM > Subject: Re: Package Review Requests for Hadoop dependencies > > Hi Rob! > > You're welcome. > > Just approved ZooKeeper; BookKeeper is last to be reviewed (and > currently FTBFS) on the list. So hadoop should reach it's target F20 in > time :) > > Cheers, > Björn > > Am Dienstag, den 11.06.2013, 10:21 -0400 schrieb Robert Rati: > > Hi Bjorn, > > > > Thanks so much for picking these up and reviewing them! > > > > Rob > > > > On 06/11/2013 03:04 AM, Björn Esser wrote: > > > Am Montag, den 10.06.2013, 15:42 -0400 schrieb Robert Rati: > > >> I'm working with the Big Data SIG to package hadoop for Fedora. We're > > >> targeting Fedora 20, but right now there are a number of packages that > > >> hadoop will depend upon that are awaiting review. All dependent > > >> packages have review BZs logged. > > >> > > >> Packages needing review for inclusion into Fedora: > > >> bookkeeper (BZ948589) > > >> groovy (BZ858127) > > >> jets3t (BZ847109) > > >> jspc-compiler (BZ960720) > > >> maven-native (BZ864084) > > >> zookeeper (BZ823122) > > >> > > >> One of those packages also has a dependency which also needs to be > > >> reviewed: > > >> jtoaster (zookeeper, BZ957337) > > >> > > >> If we are to get hadoop into Fedora 20 we need some movement on the > > >> review requests for these packages. Would anyone, or a group of people, > > >> be willing to review these packages and help us get them into Fedora? > > >> > > >> Rob > > > > > > Hello Rob! > > > > > > I'll take them one by one during this week. Would be fine if another > > > volunteer will join-in, though. > > > > > > Cheers, > > > Björn > > > > > > -- > devel mailing list > devel(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel

1 0

Re: fas name
by Tim St. Clair 11 Jun '13

11 Jun '13

You can add me. FAS Name: Tstclair Cheers, Tim ----- Original Message ----- > From: "gil" <puntogil(a)libero.it> > To: "Robert Rati" <rrati(a)redhat.com>, "Tim St Clair" <tstclair(a)redhat.com> > Sent: Tuesday, June 11, 2013 5:18:26 AM > Subject: fas name > > hi > can you give me your fas name for append this one to the fiels owner for > package: > bookkeeper > zookeeper > and for all those who feel appropriate > thanks > regards > gil > >

1 0

Re: Package Review Requests for Hadoop dependencies
by Robert Rati 11 Jun '13

11 Jun '13

Hi Bjorn, Thanks so much for picking these up and reviewing them! Rob On 06/11/2013 03:04 AM, Björn Esser wrote: > Am Montag, den 10.06.2013, 15:42 -0400 schrieb Robert Rati: >> I'm working with the Big Data SIG to package hadoop for Fedora. We're >> targeting Fedora 20, but right now there are a number of packages that >> hadoop will depend upon that are awaiting review. All dependent >> packages have review BZs logged. >> >> Packages needing review for inclusion into Fedora: >> bookkeeper (BZ948589) >> groovy (BZ858127) >> jets3t (BZ847109) >> jspc-compiler (BZ960720) >> maven-native (BZ864084) >> zookeeper (BZ823122) >> >> One of those packages also has a dependency which also needs to be reviewed: >> jtoaster (zookeeper, BZ957337) >> >> If we are to get hadoop into Fedora 20 we need some movement on the >> review requests for these packages. Would anyone, or a group of people, >> be willing to review these packages and help us get them into Fedora? >> >> Rob > > Hello Rob! > > I'll take them one by one during this week. Would be fine if another > volunteer will join-in, though. > > Cheers, > Björn >

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

bigdata June 2013