Hi all,
since few days I'm trying hekafs and i really like it. Neverthless, I found some issues wich I described below.
The configuration I used is the following : - 4 (virtualized ) servers named glusterfs1,2,3,4 all running centos (I use the epel repo for the rpms) or all running fedora 16 - 1 (virtualized) client named glusterfs5 running centos or fedora - the host runs fedora 17 and the network was bridged for all the virtual machines.
My goal was to achieve a "replicated" (-r 2) hekafs with ssl. Each server having its own key and public certificate.
The first problem I had was with documentation :
- man 8 hekafs : for creating keys, it is written "openssl genrsa 1024 -out server.key". In fact the number of bits must be at the end of the command or the output file will not be created so the line has to be "openssl genrsa -out server.key 1024".
- following the link given in fedora wiki (https://fedoraproject.org/wiki/Features/CloudFS), I have access to the file named README.ssl. But when I cloned the git repository, this file does not exist. I tried to search in the git historybut I can not find it (but my knowledge of git is (very) low so I could have missed something).
For my configuration, I got the information that the different files server*.pem must be concatened in a file named "root.pem". But, this information is not in the man of hekafs. So does the file containing the different (server) certificate must be named "root.pem" or can we set another name for it ?
- from the hfs_mount manpage, I was not able to use the data key parameter. Does this parameter work only with branch aes from the git repository ?
During my testing, maybe the most annoying thing was that if something went wrong during a hfs_mount command, no information that a problem occured is given. For example, if you launch a hfs_mount command with a wrong password or bad client's certificate, you always get 0 with echo $? . The log file contains something like this : " [2012-06-01 13:37:39.634914] E [graph.c:526:glusterfs_graph_activate] 0-graph: init failed [2012-06-01 13:37:39.635219] W [glusterfsd.c:727:cleanup_and_exit] (-->/usr/sbin/glusterfs(main+0x295) [0x405e85] (-->/usr/sbin/glusterfs(glusterfs_volumes_init+0x145) [0x404d45] (-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x198) [0x404bf8]))) 0-: received signum (0), shutting down [2012-06-01 13:37:39.635290] I [fuse-bridge.c:3727:fini] 0-fuse: Unmounting '/gluster/'. " So fuse silently unmounts "/gluster" (which succeeds), and $? contains then the 0 value instead of a code error. Maybe, in this case the function which unmounts the folder should "remember" that something went wrong ?
I also had other issues but I suspect that it is rather due to my ignorance of hekafs :) So I prefer to ask if I observed "normal" behaviours :) :
- when I want to do a hfs_add_node (cli), I get a SSH error "The authenticity of host '192.168.1.199 (192.168.1.199)' can't be established". So I create a ssh key and copy it to the host by a ssh-copy-id command and after that the hfs_add_node works without any problem.
-in order to test the self-healing facility, I shutted down a server and copy some file from the client. When i turn back on the server, the "missing" files do not appear (I waited 10 minutes since I read that glusterfs runs a self-healing every 600s). On the client, I need to unmount and mount the folder (then files appears on the server but have 0 size) and then launch a "find /gluster/ -noleaf -print0 | xargs --null stat >/dev/null" for "really" having the files on the server.
- I try to compile hekafs from source via the fedora-ize script but it fails. It seems that some folders have beem renamed (pkg->packaging ?). A make fedora in the packaging folder does the trick.
- each server is running django at the 8080 port. But I could not find a way to turn it off (except by a iptable rule). I think it could be a security problem because a local attaker can run an nmap scan on then local network and then can have access to the configuration of the servers.
- In glusterfs, you could share files via acls to "few people" . Is it possible to do such kinds of thing with different tenants ? ( I suspect the answer to be no since on the server side each tenant has its own folder).
I am writing some documentation of my setup. Do you think it could be interesting to other users?
Sorry for this quite long mail and many thanks to all for this great peace of software .
Best regards, Yves
Sorry for the delay getting back to you.
On Fri, 1 Jun 2012 15:48:48 +0200 Yves Pagani ypagani@aps.edu.pl wrote:
- man 8 hekafs :
for creating keys, it is written "openssl genrsa 1024 -out server.key". In fact the number of bits must be at the end of the command or the output file will not be created so the line has to be "openssl genrsa -out server.key 1024".
Correct.
- following the link given in fedora wiki
(https://fedoraproject.org/wiki/Features/CloudFS), I have access to the file named README.ssl. But when I cloned the git repository, this file does not exist. I tried to search in the git historybut I can not find it (but my knowledge of git is (very) low so I could have missed something).
http://git.fedorahosted.org/git/?p=CloudFS.git;a=blob;f=scripts/README.ssl;h... works for me. Did it not work for you? I've attached the file, but be aware that it's slightly out of date. Some of the updated information is now in the main man page, and some is implicit in the following commands so see theirs:
hfs_update_cert hfs_start_volume hfs_mount
For my configuration, I got the information that the different files server*.pem must be concatened in a file named "root.pem". But, this information is not in the man of hekafs. So does the file containing the different (server) certificate must be named "root.pem" or can we set another name for it ?
The combination of certificates (specified with hfs_update_cert of through the GUI) is done automatically in hfs_start_volume.
- from the hfs_mount manpage, I was not able to use the data key
parameter. Does this parameter work only with branch aes from the git repository ?
What error did you get? The data key must be in a specific format, and our diagnostics for an incorrect format are probably not very good.
During my testing, maybe the most annoying thing was that if something went wrong during a hfs_mount command, no information that a problem occured is given. For example, if you launch a hfs_mount command with a wrong password or bad client's certificate, you always get 0 with echo $? . The log file contains something like this : " [2012-06-01 13:37:39.634914] E [graph.c:526:glusterfs_graph_activate] 0-graph: init failed [2012-06-01 13:37:39.635219] W [glusterfsd.c:727:cleanup_and_exit] (-->/usr/sbin/glusterfs(main+0x295) [0x405e85] (-->/usr/sbin/glusterfs(glusterfs_volumes_init+0x145) [0x404d45] (-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x198) [0x404bf8]))) 0-: received signum (0), shutting down [2012-06-01 13:37:39.635290] I [fuse-bridge.c:3727:fini] 0-fuse: Unmounting '/gluster/'. " So fuse silently unmounts "/gluster" (which succeeds), and $? contains then the 0 value instead of a code error. Maybe, in this case the function which unmounts the folder should "remember" that something went wrong ?
I'll look into that.
- when I want to do a hfs_add_node (cli), I get a SSH error "The
authenticity of host '192.168.1.199 (192.168.1.199)' can't be established". So I create a ssh key and copy it to the host by a ssh-copy-id command and after that the hfs_add_node works without any problem.
This should only affect first-time setup. We need to be able to enable the HekaFS daemon on the remote node before we can do that. The default implementation of make_remote (in hfs_utils.py) does this using ssh, but it should be easy to make it use any other method you want.
-in order to test the self-healing facility, I shutted down a server and copy some file from the client. When i turn back on the server, the "missing" files do not appear (I waited 10 minutes since I read that glusterfs runs a self-healing every 600s). On the client, I need to unmount and mount the folder (then files appears on the server but have 0 size) and then launch a "find /gluster/ -noleaf -print0 | xargs --null stat >/dev/null" for "really" having the files on the server.
Yes, in GlusterFS 3.2.x (on which HekaFS is based), there's no automatic self-heal so an explicit find/ls/whatever is necessary to touch the files. GlusterFS 3.3 does have automatic self-heal, but is currently incompatible with HekaFS. We're trying to avoid doing a 3.3 version of HekaFS, because all of that work would become obsolete when the functionality is fully integrated into GlusterFS itself (probably in 3.4 but I can't promise that).
- I try to compile hekafs from source via the fedora-ize script but
it fails. It seems that some folders have beem renamed (pkg->packaging ?). A make fedora in the packaging folder does the trick.
Yes, Kaleb sort of went a different route with the packaging/* stuff, so "fedora-ize" is basically obsolete.
- each server is running django at the 8080 port. But I could not
find a way to turn it off (except by a iptable rule). I think it could be a security problem because a local attaker can run an nmap scan on then local network and then can have access to the configuration of the servers.
Yes, this is an issue we've inherited from GlusterFS (which is similarly insecure). A while ago I did some work to enable SSL for the management interfaces, and Pete Zaitcev did something similar, but neither has made it into the master branch yet.
- In glusterfs, you could share files via acls to "few people" . Is
it possible to do such kinds of thing with different tenants ? ( I suspect the answer to be no since on the server side each tenant has its own folder).
Tenants are *fully* isolated from one another, by design. Thus, there is no sharing at all between them.
I am writing some documentation of my setup. Do you think it could be interesting to other users?
I'm sure it would.
Sorry for this quite long mail and many thanks to all for this great peace of software .
You're quite welcome, and thank you for your feedback.
Hi, On Mon, Jun 04, 2012 at 10:30:35AM -0400, Jeff Darcy wrote:
Sorry for the delay getting back to you.
No problem : days only have 24h :) and with the release of glusterfs 3.3, I think you were a litte bit busy :)
On Fri, 1 Jun 2012 15:48:48 +0200 Yves Pagani ypagani@aps.edu.pl wrote:
- man 8 hekafs :
for creating keys, it is written "openssl genrsa 1024 -out server.key". In fact the number of bits must be at the end of the command or the output file will not be created so the line has to be "openssl genrsa -out server.key 1024".
Correct.
- following the link given in fedora wiki
(https://fedoraproject.org/wiki/Features/CloudFS), I have access to the file named README.ssl. But when I cloned the git repository, this file does not exist. I tried to search in the git historybut I can not find it (but my knowledge of git is (very) low so I could have missed something).
http://git.fedorahosted.org/git/?p=CloudFS.git;a=blob;f=scripts/README.ssl;h... works for me. Did it not work for you? I've attached the file, but be aware that it's slightly out of date. Some of the updated information is now in the main man page, and some is implicit in the following commands so see theirs:
hfs_update_cert hfs_start_volume hfs_mount
Thanks for the file. The link you gave works, but when I clone the repo ( git clone -v git://git.fedorahosted.org/CloudFS.git ), I can't find this file. I tried : - git checkout 3547f6b354d8b3455359ea0 (thinking that the hash in the link was the "commit hash ") - git log - gitk just in case I missed something
without success.
For my configuration, I got the information that the different files server*.pem must be concatened in a file named "root.pem". But, this information is not in the man of hekafs. So does the file containing the different (server) certificate must be named "root.pem" or can we set another name for it ?
The combination of certificates (specified with hfs_update_cert of through the GUI) is done automatically in hfs_start_volume.
Thanks for this information. It is great that hfs_start_volume does all the required magic :)
- from the hfs_mount manpage, I was not able to use the data key
parameter. Does this parameter work only with branch aes from the git repository ?
What error did you get? The data key must be in a specific format, and our diagnostics for an incorrect format are probably not very good.
I did a dd if=/dev/random of=data.key bs=1M count=1... After your mail, I digged a little bit in the source code and by doing a openssl enc -aes-256-cbc -k secret -P -md sha1 and copying the (key part only) outcome into a file, I was able to do a hfs_mount. Neverthless I obtained a strange result : from the client, I copied some file to the mount point, files were copied on the different servers. On the client, I then unmounted the filesystem and remounted it without giving the data key. And of course, the files were encrypted as expected. But on the servers, the files are always in cleared form ! Is it a normal behaviour ? I thought that when I copied the files to the servers, they were encrypted before being sent.
During my testing, maybe the most annoying thing was that if something went wrong during a hfs_mount command, no information that a problem occured is given. For example, if you launch a hfs_mount command with a wrong password or bad client's certificate, you always get 0 with echo $? . The log file contains something like this : " [2012-06-01 13:37:39.634914] E [graph.c:526:glusterfs_graph_activate] 0-graph: init failed [2012-06-01 13:37:39.635219] W [glusterfsd.c:727:cleanup_and_exit] (-->/usr/sbin/glusterfs(main+0x295) [0x405e85] (-->/usr/sbin/glusterfs(glusterfs_volumes_init+0x145) [0x404d45] (-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x198) [0x404bf8]))) 0-: received signum (0), shutting down [2012-06-01 13:37:39.635290] I [fuse-bridge.c:3727:fini] 0-fuse: Unmounting '/gluster/'. " So fuse silently unmounts "/gluster" (which succeeds), and $? contains then the 0 value instead of a code error. Maybe, in this case the function which unmounts the folder should "remember" that something went wrong ?
I'll look into that.
If you need more verbose logs, I can send them.
- when I want to do a hfs_add_node (cli), I get a SSH error "The
authenticity of host '192.168.1.199 (192.168.1.199)' can't be established". So I create a ssh key and copy it to the host by a ssh-copy-id command and after that the hfs_add_node works without any problem.
This should only affect first-time setup. We need to be able to enable the HekaFS daemon on the remote node before we can do that. The default implementation of make_remote (in hfs_utils.py) does this using ssh, but it should be easy to make it use any other method you want.
Ok
-in order to test the self-healing facility, I shutted down a server and copy some file from the client. When i turn back on the server, the "missing" files do not appear (I waited 10 minutes since I read that glusterfs runs a self-healing every 600s). On the client, I need to unmount and mount the folder (then files appears on the server but have 0 size) and then launch a "find /gluster/ -noleaf -print0 | xargs --null stat >/dev/null" for "really" having the files on the server.
Yes, in GlusterFS 3.2.x (on which HekaFS is based), there's no automatic self-heal so an explicit find/ls/whatever is necessary to touch the files. GlusterFS 3.3 does have automatic self-heal, but is currently incompatible with HekaFS. We're trying to avoid doing a 3.3 version of HekaFS, because all of that work would become obsolete when the functionality is fully integrated into GlusterFS itself (probably in 3.4 but I can't promise that).
Ok. It will be very nice to seem them merged :)
- I try to compile hekafs from source via the fedora-ize script but
it fails. It seems that some folders have beem renamed (pkg->packaging ?). A make fedora in the packaging folder does the trick.
Yes, Kaleb sort of went a different route with the packaging/* stuff, so "fedora-ize" is basically obsolete.
Ok
- each server is running django at the 8080 port. But I could not
find a way to turn it off (except by a iptable rule). I think it could be a security problem because a local attaker can run an nmap scan on then local network and then can have access to the configuration of the servers.
Yes, this is an issue we've inherited from GlusterFS (which is similarly insecure). A while ago I did some work to enable SSL for the management interfaces, and Pete Zaitcev did something similar, but neither has made it into the master branch yet.
- In glusterfs, you could share files via acls to "few people" . Is
it possible to do such kinds of thing with different tenants ? ( I suspect the answer to be no since on the server side each tenant has its own folder).
Tenants are *fully* isolated from one another, by design. Thus, there is no sharing at all between them.
Ok
I am writing some documentation of my setup. Do you think it could be interesting to other users?
I'm sure it would.
I hope I will finish this in a few days. And I will send it to the mailing list. Which format do you prefer : simple text or latex+pdf ?
Sorry for this quite long mail and many thanks to all for this great peace of software .
You're quite welcome, and thank you for your feedback.
Thanks your time and answers.
On 06/05/2012 08:01 AM, Yves Pagani wrote:
I did a dd if=/dev/random of=data.key bs=1M count=1... After your mail, I digged a little bit in the source code and by doing a openssl enc -aes-256-cbc -k secret -P -md sha1 and copying the (key part only) outcome into a file, I was able to do a hfs_mount. Neverthless I obtained a strange result : from the client, I copied some file to the mount point, files were copied on the different servers. On the client, I then unmounted the filesystem and remounted it without giving the data key. And of course, the files were encrypted as expected. But on the servers, the files are always in cleared form ! Is it a normal behaviour ? I thought that when I copied the files to the servers, they were encrypted before being sent.
That is indeed what's supposed to happen, and I just rechecked to make sure it does. Your result seems very strange to me, because if you mount without a data key then the crypt translator won't even be loaded and therefore there should be no transformations on the data . . . and yet you report that the data read that way appears to be encrypted. If it had been stored plain, who's encrypting it in that case? Can you send the client volfiles (from the logs) that are being used when you mount with and without an encryption key?
(Sorry Jeff for sending this mail only to you instead of to the mailing list)
On Tue, Jun 05, 2012 at 10:44:42AM -0400, Jeff Darcy wrote:
On 06/05/2012 08:01 AM, Yves Pagani wrote:
I did a dd if=/dev/random of=data.key bs=1M count=1... After your mail, I digged a little bit in the source code and by doing a openssl enc -aes-256-cbc -k secret -P -md sha1 and copying the (key part only) outcome into a file, I was able to do a hfs_mount. Neverthless I obtained a strange result : from the client, I copied some file to the mount point, files were copied on the different servers. On the client, I then unmounted the filesystem and remounted it without giving the data key. And of course, the files were encrypted as expected. But on the servers, the files are always in cleared form ! Is it a normal behaviour ? I thought that when I copied the files to the servers, they were encrypted before being sent.
That is indeed what's supposed to happen, and I just rechecked to make sure it does. Your result seems very strange to me, because if you mount without a data key then the crypt translator won't even be loaded and therefore there should be no transformations on the data . . . and yet you report that the data read that way appears to be encrypted. If it had been stored plain, who's encrypting it in that case? Can you send the client volfiles (from the logs) that are being used when you mount with and without an encryption key?
I attached the client volfiles and glusterlog for 3 cases : - with the data key - without the key - with a bad key
I also did more testing and I saw stranges results which are given below. Please note that in my configuration I have 4 servers glusterfs1,glusterfs2, glusterfs3, glusterfs4 and that glusterfs2 replicates glusterfs1 and glusterfs4 replicates glusterfs3.
- I created a file named hello1.txt (which contains only the word Hello), copied it to have 5 more identically files named hello2..6.txt md5sum hello*.txt gives : 09f7e02f1290be211da707a266f153b3 hello1.txt 09f7e02f1290be211da707a266f153b3 hello2.txt 09f7e02f1290be211da707a266f153b3 hello3.txt 09f7e02f1290be211da707a266f153b3 hello4.txt 09f7e02f1290be211da707a266f153b3 hello5.txt 09f7e02f1290be211da707a266f153b3 hello6.txt
- I mounted the filesystem (hfs_mount -d ~/data.key -p ~/server.key -c ~/tenant1.pem -s ~/root.pem glusterfs1 wtf toto toto /gluster/) and copied the files (copy hello*.txt /gluster). On each server I did a md5sum hello*.txt and I obtained the following results :
on glusterfs1 : 09f7e02f1290be211da707a266f153b3 /gluster/toto/hello3.txt 09f7e02f1290be211da707a266f153b3 /gluster/toto/hello5.txt 09f7e02f1290be211da707a266f153b3 /gluster/toto/hello6.txt
on glusterfs2 : 1494788811b7db4b2a5fc374c7331a67 /gluster/toto/hello3.txt e2aee1de34f01b1bc99888bc20eaaef2 /gluster/toto/hello5.txt e13ff05c010f78240a564192b3f03d40 /gluster/toto/hello6.txt
on glusterfs3 : 93e3e8cbef5041c58eef1af2091f4bdc /gluster/toto/hello1.txt 683f04b2e888387dff6a1c75ebf53e3a /gluster/toto/hello2.txt 2b73353d9b0fc05f66f50c93157986db /gluster/toto/hello4.txt
on glusterfs4 : 09f7e02f1290be211da707a266f153b3 /gluster/toto/hello1.txt 09f7e02f1290be211da707a266f153b3 /gluster/toto/hello2.txt 09f7e02f1290be211da707a266f153b3 /gluster/toto/hello4.txt
on the client : 09f7e02f1290be211da707a266f153b3 /gluster/hello1.txt 683f04b2e888387dff6a1c75ebf53e3a /gluster/hello2.txt 09f7e02f1290be211da707a266f153b3 /gluster/hello3.txt 09f7e02f1290be211da707a266f153b3 /gluster/hello4.txt e2aee1de34f01b1bc99888bc20eaaef2 /gluster/hello5.txt 09f7e02f1290be211da707a266f153b3 /gluster/hello6.txt
So we can observe that : - on replicated servers, one server always has the files in clear form (confirmed by a cat hello) and the second has them in an encrypted form - some of the files seen by the client are encrypted !
When i unmount and mount the filesystem without the encryption key, I obtained the same results. Neverthless when I mount the filesystem with a "bad" encryption key, the client sees all the files encrypted (which is normal in my opinion since it means that some encryption with the encryption key is done)
But I have no idea why I obtained such results.
On Tue, Jun 05, 2012 at 10:44:42AM -0400, Jeff Darcy wrote:
On 06/05/2012 08:01 AM, Yves Pagani wrote:
I did a dd if=/dev/random of=data.key bs=1M count=1... After your mail, I digged a little bit in the source code and by doing a openssl enc -aes-256-cbc -k secret -P -md sha1 and copying the (key part only) outcome into a file, I was able to do a hfs_mount. Neverthless I obtained a strange result : from the client, I copied some file to the mount point, files were copied on the different servers. On the client, I then unmounted the filesystem and remounted it without giving the data key. And of course, the files were encrypted as expected. But on the servers, the files are always in cleared form ! Is it a normal behaviour ? I thought that when I copied the files to the servers, they were encrypted before being sent.
That is indeed what's supposed to happen, and I just rechecked to make sure it does. Your result seems very strange to me, because if you mount without a data key then the crypt translator won't even be loaded and therefore there should be no transformations on the data . . . and yet you report that the data read that way appears to be encrypted. If it had been stored plain, who's encrypting it in that case? Can you send the client volfiles (from the logs) that are being used when you mount with and without an encryption key?
my previous mail with logs is under moderation so I had time to test the case where I have 4 servers without replication (but with ssl) : all is working well :
- the files on the servers are encrypted - the client can read all the files - when I unmount and mount without the key, the client sees the files as encrypted - when I mount with a "bad" key, the client sees the files as encrypted.
It seems that replication breaks something when using a encryption key. I hope the logs I sent will help. If more testing/configuration/logs are needed, I will be happy to help :)
On 06/06/2012 08:06 AM, Yves Pagani wrote:
my previous mail with logs is under moderation so I had time to test the case where I have 4 servers without replication (but with ssl) : all is working well :
- the files on the servers are encrypted
- the client can read all the files
- when I unmount and mount without the key, the client sees the files as encrypted
- when I mount with a "bad" key, the client sees the files as encrypted.
It seems that replication breaks something when using a encryption key. I hope the logs I sent will help. If more testing/configuration/logs are needed, I will be happy to help :)
OK, I see what the problem is. The crypt translator is loaded *below* AFR, for reasons I don't recall at the moment. It encrypts the write buffer in place. Therefore, we get the following:
* afr-0 gets the request, sends it to crypt-0 and crypt-1
* crypt-0 encrypts the buffer, sends it to client-0
* crypt-1 encrypts the buffer (reverting to plaintext), sends it to client-1
So yes, it's definitely a problem with the combination of crypt and AFR. The obvious solution is to load crypt *above* AFR so that the buffer's only encrypted once, but first I'll have to figure out why that didn't work before and fix that.
cloudfs-general@lists.stg.fedorahosted.org
-
Jeff Darcy -
yp