Hello,
Nowadays, there is almost no support for OVAL content development. The developers have to edit the XML files manually, there is no analysis tool, no debugger or any other tool that could make the content development faster and easier. I have started working on a new utility that will be able to debug OVAL checks, show step-by-step how they are evaluated, browse collected objects and system characteristics, communicate with probes, etc. I'm working on design of the utility now. I'm considering many alternatives, so I would like to ask you a few questions. * Can you describe most common problems that you have to face when you create the OVAL content? * How do you usually solve these problems? * Can you tell me an example when did you run into issues with OVAL? * Imagine an ideal tool for an OVAL developer. What should it be able to do? What would be its features? I would appreciate any suggestions, remarks or other inputs. Thank you very much for your reply.
Regards
Jan Černý Security Technologies | Red Hat, Inc.
That seems great. I'm more than interested to help. Have you selected the target programming language/framework yet? If you are working on mockups, OmniGraffle-like diagrams or so, please kindly let me know.
NB: In addition to the OVAL Guidances documents, these lesson learned could help https://datatracker.ietf.org/doc/draft-hansbury-sacm-oval-info-model-mapping...
Cheers
2015-06-04 10:15 GMT+03:00 Jan Cerny jcerny@redhat.com:
Hello,
Nowadays, there is almost no support for OVAL content development. The developers have to edit the XML files manually, there is no analysis tool, no debugger or any other tool that could make the content development faster and easier. I have started working on a new utility that will be able to debug OVAL checks, show step-by-step how they are evaluated, browse collected objects and system characteristics, communicate with probes, etc. I'm working on design of the utility now. I'm considering many alternatives, so I would like to ask you a few questions.
- Can you describe most common problems that you have to face when you create the OVAL content?
- How do you usually solve these problems?
- Can you tell me an example when did you run into issues with OVAL?
- Imagine an ideal tool for an OVAL developer. What should it be able to do? What would be its features?
I would appreciate any suggestions, remarks or other inputs. Thank you very much for your reply.
Regards
Jan Černý Security Technologies | Red Hat, Inc. -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
Hello Jan,
----- Original Message -----
From: "Jan Cerny" jcerny@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Thursday, June 4, 2015 9:15:42 AM Subject: A tool for support of SCAP content development
Hello,
Nowadays, there is almost no support for OVAL content development.
Agree in this point.
The developers have to edit the XML files manually, there is no analysis tool, no debugger or any other tool that could make the content development faster and easier. I have started working on a new utility that will be able to debug OVAL checks, show step-by-step how they are evaluated, browse collected objects and system characteristics, communicate with probes, etc.
Two questions: * is this new tool intended to become part of the openscap-* package, or rather the plan being it to be a separate tool?, * above you speak about debugger -- is the plan to add also profiler?
I'm working on design of the utility now. I'm considering many alternatives, so I would like to ask you a few questions.
- Can you describe most common problems that you have to face when you create the OVAL content?
Absence of ability to debug a particular OVAL check. Right now it's possible to use 'oscap oval eval / oscap oval collect' for particular rule to determine what got actually collected on the system. This approach is working for simple rules. But it does not scale properly (if the underlying OVAL object consist of another OVAL variable, that variable using some OVAL functions on other variables and so on, it's hard to determine which OVAL object failed to be collected at the beginning -- which OVAL object is responsible for the whole rule failure).
Next step is XCCDF evaluation - while the corresponding OVAL check might work properly, it's possible (due to combination of "check", "check_existence", "entity_check" attributes, order and / or logical operations between the different OVAL criterions) the final XCCDF rules is still failing even in the case "some" OVAL objects have been collected on the system.
Right now oscap provides 'oscap xccdf eval .. --oval-results' combination alllowing to inspect, why the rule as a whole did not pass (e.g. wrong "check" / "check_existence" attributes etc.).
But it would be good if the new tool would support XCCDF debugging too (something like 'oscap xccdf eval ... --oval-results' is allowing now, but enhanced).
- How do you usually solve these problems?
See above. By separating the rule intended to be debugged. Then running: * oscap oval eval --id * oscap oval collect --id * oscap xccdf eval .. --oval-results
combination of commands (depending on the level [e.g. if OVAL itself, or at XCCDF level] the OVAL check is failing).
- Can you tell me an example when did you run into issues with OVAL?
Basically any a bit more complex OVAL check needs to be designed in stages (from top to bottom). Consider OVAL criterion requiring presence just some OVAL object, this object to consist of multiple stages, e.g.:
* you first collect something on the system into object, save the subexpression of it into <local_variable>, * apply couple of functions to that variable, finally create new final object from the last variable. This final object is then used in the OVAL test.
Right now it needs to be done from bottom to the top (at each level, at each <local_variable> you mostly need to verify it returns appropriate results). Other issue is OVAL standard reporting functionality. AFAICT the standard requires the rule to return 'error' also in the case the particular OVAL rule wasn't able to obtain child results (results for the child objects compositing the final object). Having seen this 'error' result for the first time, it's not clear if: * OVAL check truly has a syntax error, * or OVAL check is correct, but the engine wasn't able to determine value for some of the OVAL variables above in the evaluation tree.
Another issue / RFE in 'oscap oval eval / collect' functionality is the '--id' parameter. IIRC as of right now it supports specification of an ID just for OVAL objects (therefore having OVAL variable manipulating some previously collected OVAL object via some OVAL function, it's impossible to see the retrieved values of that variable directly.
You need to see the values of the original OVAL object, then create another OVAL object from that OVAL variable using the OVAL function, to be able to see the true values, that OVAL variable is holding).
Not to speak to abstract. Consider the following:
<ind:textfilecontent54_object id="my_object" ..> ind:filepath..</ind:filepath> <ind:pattern ..>..</ind:pattern> <ind:instance ..>..</ind:instance> </ind:textfilecontent54_object>
<local_variable id="my_variable> <some_oval_function_here, e.g. "split", "concat", whatever> <object_component object_ref="my_object" /> .. </some_oval_function_here> </local_variable>
Right now it's not possible to perform the following:
oscap oval collect --id "my_variable" --syschar /tmp/out
because "my_variable" doesn't constitute an OVAL object (therefore to bypass it, it's necessary to define new artificial OVAL object from that "my_variable" local variable, and try to collect it).
Therefore it would be very handy if 'collect' could be enhanced to support variables too.
- Imagine an ideal tool for an OVAL developer. What should it be able to do? What would be its features?
Maybe above being a sketch. From the above it's not clear if the new tool should be just debugger, or profiler too. If the debugger, basically it would be able to: * report values of particular OVAL objects, variables at any time of OVAL evaluation, * report the branch OVAL check has been evaluated (like OVAL check went through criterion #1, the rest of the criteria weren't taken into account, since criterion #1 already succeeded), * + the requests as formulated above.
If we would talk also about profiling capabilities - it would be able to report things like CPU usage in %, virtual memory usage, and or count of system files processed when evaluating particular rule.
This would be handy to optimize current form in many rules.
I would appreciate any suggestions, remarks or other inputs. Thank you very much for your reply.
Should you need any other clarification wrt to some of the points above, just ask.
Thank you && Regards, Jan -- Jan iankko Lieskovsky / Red Hat Security Technologies Team
Regards
Jan Černý Security Technologies | Red Hat, Inc. -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
Hello,
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "Jan Cerny" jcerny@redhat.com Cc: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Friday, September 11, 2015 11:39:40 AM Subject: Re: A tool for support of SCAP content development
Hello Jan,
----- Original Message -----
From: "Jan Cerny" jcerny@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Thursday, June 4, 2015 9:15:42 AM Subject: A tool for support of SCAP content development
Hello,
Nowadays, there is almost no support for OVAL content development.
Agree in this point.
The developers have to edit the XML files manually, there is no analysis tool, no debugger or any other tool that could make the content development faster and easier. I have started working on a new utility that will be able to debug OVAL checks, show step-by-step how they are evaluated, browse collected objects and system characteristics, communicate with probes, etc.
Two questions:
- is this new tool intended to become part of the openscap-* package, or rather the plan being it to be a separate tool?,
It was decided to be as part of the openscap. Official instructions approved by university says "implement this in the OpenSCAP project by form of editing existing source code." For university instructions, see Description in Czech on this page: https://diplomky.redhat.com/topic/show/335/a-tool-for-development-of-oval-de... I should definitely put the link on the wiki.
- above you speak about debugger -- is the plan to add also profiler?
It is not in my plan now, but thank your very much for your idea!
I'm working on design of the utility now. I'm considering many alternatives, so I would like to ask you a few questions.
- Can you describe most common problems that you have to face when you create the OVAL content?
Absence of ability to debug a particular OVAL check. Right now it's possible to use 'oscap oval eval / oscap oval collect' for particular rule to determine what got actually collected on the system. This approach is working for simple rules. But it does not scale properly (if the underlying OVAL object consist of another OVAL variable, that variable using some OVAL functions on other variables and so on, it's hard to determine which OVAL object failed to be collected at the beginning -- which OVAL object is responsible for the whole rule failure).
Next step is XCCDF evaluation - while the corresponding OVAL check might work properly, it's possible (due to combination of "check", "check_existence", "entity_check" attributes, order and / or logical operations between the different OVAL criterions) the final XCCDF rules is still failing even in the case "some" OVAL objects have been collected on the system.
Right now oscap provides 'oscap xccdf eval .. --oval-results' combination alllowing to inspect, why the rule as a whole did not pass (e.g. wrong "check" / "check_existence" attributes etc.).
But it would be good if the new tool would support XCCDF debugging too (something like 'oscap xccdf eval ... --oval-results' is allowing now, but enhanced).
I am working now on enhancing the debug infos and introduce "--verbose" option in oscap tool. I think it is a good first step.
- How do you usually solve these problems?
See above. By separating the rule intended to be debugged. Then running:
- oscap oval eval --id
- oscap oval collect --id
- oscap xccdf eval .. --oval-results
combination of commands (depending on the level [e.g. if OVAL itself, or at XCCDF level] the OVAL check is failing).
- Can you tell me an example when did you run into issues with OVAL?
Basically any a bit more complex OVAL check needs to be designed in stages (from top to bottom). Consider OVAL criterion requiring presence just some OVAL object, this object to consist of multiple stages, e.g.:
- you first collect something on the system into object, save the subexpression of it into <local_variable>,
- apply couple of functions to that variable, finally create new final object from the last variable. This final object is then used in the OVAL test.
Right now it needs to be done from bottom to the top (at each level, at each <local_variable> you mostly need to verify it returns appropriate results). Other issue is OVAL standard reporting functionality. AFAICT the standard requires the rule to return 'error' also in the case the particular OVAL rule wasn't able to obtain child results (results for the child objects compositing the final object). Having seen this 'error' result for the first time, it's not clear if:
- OVAL check truly has a syntax error,
- or OVAL check is correct, but the engine wasn't able to determine value for some of the OVAL variables above in the evaluation tree.
Another issue / RFE in 'oscap oval eval / collect' functionality is the '--id' parameter. IIRC as of right now it supports specification of an ID just for OVAL objects (therefore having OVAL variable manipulating some previously collected OVAL object via some OVAL function, it's impossible to see the retrieved values of that variable directly.
You need to see the values of the original OVAL object, then create another OVAL object from that OVAL variable using the OVAL function, to be able to see the true values, that OVAL variable is holding).
Not to speak to abstract. Consider the following:
<ind:textfilecontent54_object id="my_object" ..> ind:filepath..</ind:filepath> <ind:pattern ..>..</ind:pattern> <ind:instance ..>..</ind:instance> </ind:textfilecontent54_object>
<local_variable id="my_variable> <some_oval_function_here, e.g. "split", "concat", whatever> <object_component object_ref="my_object" /> ..
</some_oval_function_here> </local_variable>
Right now it's not possible to perform the following:
oscap oval collect --id "my_variable" --syschar /tmp/out
because "my_variable" doesn't constitute an OVAL object (therefore to bypass it, it's necessary to define new artificial OVAL object from that "my_variable" local variable, and try to collect it).
Therefore it would be very handy if 'collect' could be enhanced to support variables too.
This is a nice use-case.
- Imagine an ideal tool for an OVAL developer. What should it be able to do? What would be its features?
Maybe above being a sketch. From the above it's not clear if the new tool should be just debugger, or profiler too. If the debugger, basically it would be able to:
- report values of particular OVAL objects, variables at any time of OVAL evaluation,
- report the branch OVAL check has been evaluated (like OVAL check went through criterion #1, the rest of the criteria weren't taken into account, since criterion #1 already succeeded),
I am going to achieve this by collecting all the possible information during evaluating the content and processing those information to have a meaningful output.
- the requests as formulated above.
If we would talk also about profiling capabilities - it would be able to report things like CPU usage in %, virtual memory usage, and or count of system files processed when evaluating particular rule.
I would consider this, it can be useful feature to avoid bugs with excessive memory consumption caused by badly written content.
This would be handy to optimize current form in many rules.
I would appreciate any suggestions, remarks or other inputs. Thank you very much for your reply.
Should you need any other clarification wrt to some of the points above, just ask.
Thank you for your input.
Jan Černý
Thank you && Regards, Jan
Jan iankko Lieskovsky / Red Hat Security Technologies Team
Regards
Jan Černý Security Technologies | Red Hat, Inc. -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
+1 on work to support OVAL authoring.
Greg Elin
On Mon, Sep 14, 2015 at 3:17 AM, Jan Cerny jcerny@redhat.com wrote:
Hello,
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "Jan Cerny" jcerny@redhat.com Cc: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Friday, September 11, 2015 11:39:40 AM Subject: Re: A tool for support of SCAP content development
Hello Jan,
----- Original Message -----
From: "Jan Cerny" jcerny@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Thursday, June 4, 2015 9:15:42 AM Subject: A tool for support of SCAP content development
Hello,
Nowadays, there is almost no support for OVAL content development.
Agree in this point.
The developers have to edit the XML files manually, there is no analysis tool, no debugger or any other tool that could make the content development faster and easier. I have started working on a new utility that will be able to debug OVAL checks, show step-by-step how they are evaluated, browse collected objects and system characteristics, communicate with probes, etc.
Two questions:
- is this new tool intended to become part of the openscap-* package, or rather the plan being it to be a separate tool?,
It was decided to be as part of the openscap. Official instructions approved by university says "implement this in the OpenSCAP project by form of editing existing source code." For university instructions, see Description in Czech on this page:
https://diplomky.redhat.com/topic/show/335/a-tool-for-development-of-oval-de... I should definitely put the link on the wiki.
- above you speak about debugger -- is the plan to add also profiler?
It is not in my plan now, but thank your very much for your idea!
I'm working on design of the utility now. I'm considering many alternatives, so I would like to ask you a few questions.
- Can you describe most common problems that you have to face when you create the OVAL content?
Absence of ability to debug a particular OVAL check. Right now it's possible to use 'oscap oval eval / oscap oval collect' for particular rule to determine what got actually collected on the system. This approach is working for simple rules. But it does not scale properly (if the underlying OVAL object consist of another OVAL variable, that variable using some OVAL functions on other variables and so on, it's hard to determine which OVAL object failed to be collected at the beginning -- which OVAL object is responsible for the whole rule failure).
Next step is XCCDF evaluation - while the corresponding OVAL check might work properly, it's possible (due to combination of "check", "check_existence", "entity_check" attributes, order and / or logical operations between the different OVAL criterions) the final XCCDF rules is still failing even in the case "some" OVAL objects have been collected on the system.
Right now oscap provides 'oscap xccdf eval .. --oval-results' combination alllowing to inspect, why the rule as a whole did not pass (e.g. wrong "check" / "check_existence" attributes etc.).
But it would be good if the new tool would support XCCDF debugging too (something like 'oscap xccdf eval ... --oval-results' is allowing now, but enhanced).
I am working now on enhancing the debug infos and introduce "--verbose" option in oscap tool. I think it is a good first step.
- How do you usually solve these problems?
See above. By separating the rule intended to be debugged. Then running:
- oscap oval eval --id
- oscap oval collect --id
- oscap xccdf eval .. --oval-results
combination of commands (depending on the level [e.g. if OVAL itself, or at XCCDF level] the OVAL check is failing).
- Can you tell me an example when did you run into issues with OVAL?
Basically any a bit more complex OVAL check needs to be designed in stages (from top to bottom). Consider OVAL criterion requiring presence just some OVAL object, this object to consist of multiple stages, e.g.:
- you first collect something on the system into object, save the subexpression of it into <local_variable>,
- apply couple of functions to that variable, finally create new final object from the last variable. This final object is then used in the OVAL test.
Right now it needs to be done from bottom to the top (at each level, at each <local_variable> you mostly need to verify it returns appropriate results). Other issue is OVAL standard reporting functionality. AFAICT the standard requires the rule to return 'error' also in the case the particular OVAL rule wasn't able to obtain child results (results for the child objects compositing the final object). Having seen this 'error' result for the first time, it's not clear if:
- OVAL check truly has a syntax error,
- or OVAL check is correct, but the engine wasn't able to determine value for some of the OVAL variables above in the evaluation tree.
Another issue / RFE in 'oscap oval eval / collect' functionality is the '--id' parameter. IIRC as of right now it supports specification of an ID just for OVAL objects (therefore having OVAL variable manipulating some previously collected OVAL object via some OVAL function, it's impossible to see the retrieved values of that variable directly.
You need to see the values of the original OVAL object, then create another OVAL object from that OVAL variable using the OVAL function, to be able to see the true values, that OVAL variable is holding).
Not to speak to abstract. Consider the following:
<ind:textfilecontent54_object id="my_object" ..> ind:filepath..</ind:filepath> <ind:pattern ..>..</ind:pattern> <ind:instance ..>..</ind:instance> </ind:textfilecontent54_object>
<local_variable id="my_variable> <some_oval_function_here, e.g. "split", "concat", whatever> <object_component object_ref="my_object" /> ..
</some_oval_function_here> </local_variable>
Right now it's not possible to perform the following:
oscap oval collect --id "my_variable" --syschar /tmp/out
because "my_variable" doesn't constitute an OVAL object (therefore to bypass it, it's necessary to define new artificial OVAL object from that "my_variable" local variable, and try to collect it).
Therefore it would be very handy if 'collect' could be enhanced to support variables too.
This is a nice use-case.
- Imagine an ideal tool for an OVAL developer. What should it be able to do? What would be its features?
Maybe above being a sketch. From the above it's not clear if the new tool should be just debugger, or profiler too. If the debugger, basically it would be able to:
- report values of particular OVAL objects, variables at any time of OVAL evaluation,
- report the branch OVAL check has been evaluated (like OVAL check went through criterion #1, the rest of the criteria weren't taken into account, since criterion #1 already succeeded),
I am going to achieve this by collecting all the possible information during evaluating the content and processing those information to have a meaningful output.
- the requests as formulated above.
If we would talk also about profiling capabilities - it would be able to report things like CPU usage in %, virtual memory usage, and or count of system files processed when evaluating particular rule.
I would consider this, it can be useful feature to avoid bugs with excessive memory consumption caused by badly written content.
This would be handy to optimize current form in many rules.
I would appreciate any suggestions, remarks or other inputs. Thank you very much for your reply.
Should you need any other clarification wrt to some of the points above, just ask.
Thank you for your input.
Jan Černý
Thank you && Regards, Jan
Jan iankko Lieskovsky / Red Hat Security Technologies Team
Regards
Jan Černý Security Technologies | Red Hat, Inc. -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
scap-security-guide@lists.fedorahosted.org
-
Greg Elin -
Jan Cerny -
Jan Lieskovsky -
Jerome Athias