HortonWorks is using BigTop: no more secrets!

As my former colleague John Kreisa nicely put in the HortonWorks 1.0 release announcement here (my warmest regards and best wishes to you guys!):

Those who have followed Hortonworks since our initial launch already know that we are absolutely committed to open source and the Apache Software Foundation. You will be glad to know that our commitment remains the same today. We don’t hold anything back. No proprietary code is being developed at Hortonworks.

And indeed. I have asked this questions about HortonWorks using BigTop to power up their platform offering some time ago and later pretty much repeated it in the form of comment to Shaun Connolly blog. To his credit, my question has been answered directly:

As far as BigTop goes, we at Hortonworks are using parts of BigTop for the HDP platform builds, so thanks for the efforts there!

I have meet the gentleman in person at the recent Hadoop Summit and we have a short yet nice chat about enterprise stacks and the role an open-source technology plays there.

So, it is time to put my initial question to rest as the fully answered one.

P.S. On a separate note: I have left a slightly different comment on Cloudera’ blog. Somehow, the comment doesn’t appear to be visible (at least I don’t see anything but “2 comments” line) nor had it been answered publicly (again, perhaps, it has been but I don’t see in on the page). In the Cloudera’s defense I have to say that I got an answering email from one of their execs, which I can’t publish for it was a private message.

HortonWorks distribution: secretly powered by iTest and BigTop?

As I’ve mentioned here, and here BigTop is a real neat concept that I and a couple of my friends have put together a couple of years ago. Interestingly enough, second incarnation of the concept (known as iTest) made (back then) my manager to accusing me of stealing software in favor of ASF, followed by forceful departure from Cloudera (kick in the ass, with ‘I am sorry’ kinda smile on their face). But this all this in the past. The present however is much more interesting…

Some evidences were found that BigTop project (see first two links above) is seen as a power behind commercial offerings of some of the leading Hadoop vendors. Here how interesting it gets:

• you can get HortonWorks’ stack AMI from this link to play with and learn about Hadoop and stuff.

• now let’s see what they are using to power-up their distro

$ grep -i bigtopinstall_oozie.sh:     –extra-dir=DIR    path to Bigtop distribution files
install_oozie.sh:     –build-dir=DIR    path to Bigtop distribution files
install_oozie.sh:if [ -e /usr/libexec/bigtop-detect-javahome ]; then
install_oozie.sh:  . /usr/libexec/bigtop-detect-javahome
install_oozie.sh:elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
install_oozie.sh:  . /usr/lib/bigtop-utils/bigtop-detect-javahome
oozie.init:if [ -e /usr/libexec/bigtop-detect-javahome ]; then
oozie.init:  . /usr/libexec/bigtop-detect-javahome
oozie.init:elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
oozie.init:  . /usr/lib/bigtop-utils/bigtop-detect-javahome
oozie.spec:#Requires: bigtop-utils

now if you look a bit further, there some more hints that BigTop has been used big time, but that hasn’t been hidden away very well…

oozie.init:if [ -e /usr/libexec/bigtop-detect-javahome ]; then
oozie.spec:#Requires: bigtop-utils

Wow, I feel proud, guys, I really am. Now I can officially claim that my ideas are giving the push to some of the biggest vendors of what is considered the hottest technology on the market 😉 Although, they are unlikely to admit this publicly, predicating on the above example.

I will keep you posted on the development.