fixing (removing) dynamically bound variables

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

fixing (removing) dynamically bound variables

Jim Prewett

All,

I've been trying to think of a simple way to remove all dynamically bound
variables from LoGS.

My proposal (aka the best think I've come up with so far) is to have
"Environment"s be passed in to action functions, changing the lammbda list
for an action from simply (MESSAGE) to (MESSAGE &optional ENVIRONMENT).

The user would also no longer be able to bind variables by returning
things from the match function; so the ENVIRONMENT would have to be
queried by the action function.  

In this reguard, I'm also proposing that ENVIRONMENTs be changed from a
list of lists where the car represents the key (variable name) and the
cadr represents the value to a true association list (alist) so that we
could use built-in Lisp functions like assoc to access them.

This way, it is no longer mysterious where a variable's value might come
from, it should be clear.  It will also clear up if a variable is meant to
be bound at compile time (whenever the ruleset is put together) or at run
time.

What do you think?
Jim

James E. Prewett                    [hidden email] [hidden email]
Systems Team Leader           LoGS: http://www.hpc.unm.edu/~download/LoGS/ 
Designated Security Officer         OpenPGP key: pub 1024D/31816D93    
HPC Systems Engineer III   UNM HPC  505.277.8210


_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

Vijay Lakshminarayanan
On 1/6/07, Jim Prewett <[hidden email]> wrote:

>
> All,
>
> I've been trying to think of a simple way to remove all dynamically bound
> variables from LoGS.
>
> My proposal (aka the best think I've come up with so far) is to have
> "Environment"s be passed in to action functions, changing the lammbda list
> for an action from simply (MESSAGE) to (MESSAGE &optional ENVIRONMENT).
>
> The user would also no longer be able to bind variables by returning
> things from the match function; so the ENVIRONMENT would have to be
> queried by the action function.
>
> In this reguard, I'm also proposing that ENVIRONMENTs be changed from a
> list of lists where the car represents the key (variable name) and the
> cadr represents the value to a true association list (alist) so that we
> could use built-in Lisp functions like assoc to access them.
>
> This way, it is no longer mysterious where a variable's value might come
> from, it should be clear.  It will also clear up if a variable is meant to
> be bound at compile time (whenever the ruleset is put together) or at run
> time.
>
> What do you think?

Hi Jim

How would we pass the ENVIRONMENT to the actions?

Thanks
Vijay


_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

Jim Prewett

> How would we pass the ENVIRONMENT to the actions?

I was thinking of modifying run-actions in rule.lisp in partiucular to
remove the IN-GIVEN-ENVIRONMENT references and instead funcall the action
functions with an additional argument: This is the simplest thing I can
think of to do.

I also think that such a change would make testing a ruleset simpler as,
instead of having variables bound, they would be passed in in a
predictable way.

It should also make all of that scoping stuff we were talking about
resolve itself very nicely.  Variables bound by in/the Lisp code would be
clearly separated from things bound by the match functions.  Lisp already
does its scoping right and LoGS already maintains the ENVIRONMENT list
correctly; My concern before was in how these "variables" could co-exist
in the sensibly in the same namespace, which, possibly, they cannot in any
sensible way.

I've done a little playing with this and I think the modifications should
be pretty small.

Jim



_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

aashish-2
I am still quite unfamiliar with LoGS but I will definitely stress the importance of having ENVIRONMENT variables storing the dynamically bound variables.

I work with SEC rules and was getting into issues of a cascade modification problem where I would be going into each rule file and trying to determine and change paths for scripts folders or such local variables. I ended up initializing a lot of environment variables and made sec read those values in rule set. That made things a little simplified.

I think a conf file with default paths and other dynamic variables with default values would be a good feature to have.

Aashish
On Sun, Jan 07, 2007 at 12:03:52AM -0700, Jim Prewett wrote:

>
> > How would we pass the ENVIRONMENT to the actions?
>
> I was thinking of modifying run-actions in rule.lisp in partiucular to
> remove the IN-GIVEN-ENVIRONMENT references and instead funcall the action
> functions with an additional argument: This is the simplest thing I can
> think of to do.
>
> I also think that such a change would make testing a ruleset simpler as,
> instead of having variables bound, they would be passed in in a
> predictable way.
>
> It should also make all of that scoping stuff we were talking about
> resolve itself very nicely.  Variables bound by in/the Lisp code would be
> clearly separated from things bound by the match functions.  Lisp already
> does its scoping right and LoGS already maintains the ENVIRONMENT list
> correctly; My concern before was in how these "variables" could co-exist
> in the sensibly in the same namespace, which, possibly, they cannot in any
> sensible way.
>
> I've done a little playing with this and I think the modifications should
> be pretty small.
>
> Jim
>
>
>
> _______________________________________________
> LoGS-devel mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/logs-devel


_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

Jim Prewett

Hi Aashish,

Its nice to see you joined the mailing list. :)  Let me see if I can clear
some things up for you.

Let me first say that LoGS isn't quite (at all?) like the other tools out
there.  The point is to enable us to do just about anything we can dream
up when analyzing our logfiles.  So, things can be a bit strange when
moving from a tool like SEC or Logsurfer :)

On Mon, 8 Jan 2007, Aashish Sharma wrote:

> I am still quite unfamiliar with LoGS but I will definitely stress the
> importance of having ENVIRONMENT variables storing the dynamically bound
> variables.

LoGS, as is currently the latest in the CVS tree, uses what is known as
"dynamic variable binding" (where the variables are treated as,
essentially, global variables).  Lisp used to be known for using only
dynamic variable binding until it was decided that that was not a very
good idea and lexical binding was the way to go. :)

The current LoGS allows for functions to be defined where the value of a
variable used in that function could be "special" or dynamically bound -
its value would come, seemingly from nowhere.  The following would be a
perfectly acceptable "action function" (and would cause no compiler
warnings or anything).

(lambda (message)
  (declare (special server-name))
  (format t "misbehaving server: ~A~%" server-name))

this happens via a magical process where variables from a variety of
places (stored in the matching rule and ruleset(s) and returned by their
match functions) would be bound, again dynamically, and assigned values,
then the "action functions" would be run.

One of my colleagues points out that using dynamic variable binding
instead of lexical variable binding can cause debugging nightmares.  I
didn't believe him until I experienced a debugging nightmare due,
entirely, to this dynamic variable binding.

Vijay has pointed out, privately (I believe), that the use of dynamic
variable binding may present problems for multi-threaded applications -
Clearly LoGS will want to be multithreaded if possible.  For now, it is my
opinion that we won't need to go multithreaded until the large systems (as
in top 10 on top 500) grow by several orders of magnitude, so we've got a
while (but I still believe it could be worthwhile to make LoGS
multithreaded for other reasons).

So, to make a long long long sad story short, :) these are the variables I
meant by ENVIRONMENT variables.  And, I think it was actually pretty
different from what you were thinking of.

I think what I just described needs to be serioiusly re-thought,
re-worked, etc.  I would imagine that we could also get an efficiency win
by getting rid of dynamic variable bindings if we do things right.  

> I work with SEC rules and was getting into issues of a cascade
> modification problem where I would be going into each rule file and
> trying to determine and change paths for scripts folders or such local
> variables. I ended up initializing a lot of environment variables and
> made sec read those values in rule set. That made things a little
> simplified.

While this isn't the sort of thing that is built into LoGS per-se, it
should be pretty easy to do since you've got a full programming language
behind you.  You could easily define a variable like:
(defvar *MY-OUTPUT-DIRECTORY* #P"/var/log/myLoGS/")

Then you could freely use that variable throughout your ruleset.

> I think a conf file with default paths and other dynamic variables with
> default values would be a good feature to have.

Well, LoGS is a sort of an interesting beast. :)

We don't really have that feature, but we already have that feature a
little differently.  :)

IMO, Lisp provides plenty of perfectly good ways for working with paths,
I/O, etc.  LoGS provides only a couple of convienance functions for
writing messages and the contents of contexts, etc. out to files,
specifically, the WRITE-TO-FILE methods in the LoGS package.

One could have a "ruleset" file which would contain only variable
definitions.  ruleset files are loaded in the order listed on the
command-line, so a command line like the following could be used to first
load some path variables from path-definitions.lisp that are then used in
my-ruleset.lisp

lisp -core /path/to/LoGS.core \
--ruleset /my/LoGS/dir/definitions/path-definitions.lisp \
--ruleset /my/LoGS/dir/rulesets/my-ruleset.lisp

path-definitions.lisp might contain something like:

(defvar *output-directory* #P"/my/output/dir/")

and my-ruleset.lisp might contain something like:

(LoGS::enqueue
  LoGS::*root-ruleset*
  (rule named 'default-rule
        matching regexp ".+"
        doing
        (lambda (message)
         (write-to-file
          (merge-pathnames *output-directory* "out.log")
          message))))

which would match any message that had one or more characters (a regexp is
a less-than-optional choice here, but this is just for the purposes of
demonstration) and would write matching messages to the file out.log in
the directory specified in the variable *output-directory*.

Also, since LoGS is completely programmable, these ruleset files could
also contain function defintions.  You could, in theory, write a function
like:

(let ((filename "out.log"))
   (defun write-message-to-my-logfile (message)
     (write-to-file
      (merge-pathnames *output-directory* filename)
      message)))

Then, your rule could look like:

(LoGS::enqueue
  LoGS::*root-ruleset*
  (rule named 'default-rule
        matching regexp ".+"
        doing
        (lambda (message)
         (write-message-to-my-logfile message))))

or even

(LoGS::enqueue
  LoGS::*root-ruleset*
  (rule named 'default-rule
        matching regexp ".+"
        doing
        #'write-message-to-my-logfile))

make sense?

Gosh, I sure hope this (coredump) helps...
Jim

> Aashish
> On Sun, Jan 07, 2007 at 12:03:52AM -0700, Jim Prewett wrote:
> >
> > > How would we pass the ENVIRONMENT to the actions?
Ã> >

> > I was thinking of modifying run-actions in rule.lisp in partiucular to
> > remove the IN-GIVEN-ENVIRONMENT references and instead funcall the action
> > functions with an additional argument: This is the simplest thing I can
> > think of to do.
> >
> > I also think that such a change would make testing a ruleset simpler as,
> > instead of having variables bound, they would be passed in in a
> > predictable way.
> >
> > It should also make all of that scoping stuff we were talking about
> > resolve itself very nicely.  Variables bound by in/the Lisp code would be
> > clearly separated from things bound by the match functions.  Lisp already
> > does its scoping right and LoGS already maintains the ENVIRONMENT list
> > correctly; My concern before was in how these "variables" could co-exist
> > in the sensibly in the same namespace, which, possibly, they cannot in any
> > sensible way.
…> >

> > I've done a little playing with this and I think the modifications should
> > be pretty small.
> >
> > Jim
> >
> >
> >
> > _______________________________________________
> > LoGS-devel mailing list
> > [hidden email]
> > http://lists.nongnu.org/mailman/listinfo/logs-devel
>
>
> _______________________________________________
> LoGS-devel mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/logs-devel
>

_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

Vijay Lakshminarayanan
In reply to this post by Jim Prewett
On 1/7/07, Jim Prewett <[hidden email]> wrote:
>
> > How would we pass the ENVIRONMENT to the actions?
>
> I was thinking of modifying run-actions in rule.lisp in partiucular to
> remove the IN-GIVEN-ENVIRONMENT references and instead funcall the action
> functions with an additional argument: This is the simplest thing I can
> think of to do.

I'm not sure this is exactly possible.  In cases where the action is
specified as a function (as in your example to Aashish), we would have
to do something like

((lambda (message environment)
   (declare (ignorable environment))
   (funcall action message))
 message
 environment)

which, I think, is inefficienct.  I think we can still use
IN-GIVEN-ENVIRONMENT and i've given an example below.

> I also think that such a change would make testing a ruleset simpler as,
> instead of having variables bound, they would be passed in in a
> predictable way.
>
> It should also make all of that scoping stuff we were talking about
> resolve itself very nicely.  Variables bound by in/the Lisp code would be
> clearly separated from things bound by the match functions.  Lisp already
> does its scoping right and LoGS already maintains the ENVIRONMENT list
> correctly; My concern before was in how these "variables" could co-exist
> in the sensibly in the same namespace, which, possibly, they cannot in any
> sensible way.

:-)

> I've done a little playing with this and I think the modifications should
> be pretty small.

Here is one way I can think of using IN-GIVEN-ENVIRONMENT but the
bound variables are not available in sub-actions.

(defmacro in-given-environment (env action &rest args)
  (let ((body-result (gensym "BODY-RESULT"))
        (environment (gensym "ENVIRONMENT")))
    `(let ((,environment ,env))          ; <-- not GENSYM'd
      (flet ((getvar (var) (cadr (assoc var ,environment))))
        (let ((,body-result ,action))
          (if (functionp ,body-result)
              (funcall ,body-result ,@args)
              ,body-result))))))

CL-USER> (in-given-environment '((a "one") (b 'ii))
           #'(lambda (arg)
               (list (getvar 'a) (getvar 'b) arg))
           3)
("one" 'II 3)

So A and B of this ENVIRONMENT will not be available below.  But as we
discussed earlier, that is possible only with dynamic variables.

Of course, the above macro won't work in real application code.  It
needs a slight modification.  We cannot write (flet ((GETVAR (var)...)
It would have to be (flet ((,(intern "GETVAR") (var) ...) so that
GETVAR is available in the current package.

Or we could have ORG.PREWETT.LOGS export GETVAR, I think.

What do you think, Jim?

Thanks
Vijay


_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

Jim Prewett


> On 1/7/07, Jim Prewett <[hidden email]> wrote:
> >
> > > How would we pass the ENVIRONMENT to the actions?
> >
> > I was thinking of modifying run-actions in rule.lisp in partiucular to
> > remove the IN-GIVEN-ENVIRONMENT references and instead funcall the action
> > functions with an additional argument: This is the simplest thing I can
> > think of to do.
>
> I'm not sure this is exactly possible.  In cases where the action is
> specified as a function (as in your example to Aashish), we would have
> to do something like
>
> ((lambda (message environment)
>   (declare (ignorable environment))
>   (funcall action message))
> message
> environment)

I'm not quite sure what you mean.

I do believe I was originally underestimating the amount of work required
:)  I think for what I'm proposing, you would have to *CHANGE THE API* for
all rule and ruleset match functions, rule action functions, rule and
ruleset delete rule functions so that they all accepted the environment as
an argument.  All(?) of the functions that interact with those match, etc.
functions would also have to be changed.

> which, I think, is inefficienct.  I think we can still use
> IN-GIVEN-ENVIRONMENT and i've given an example below.

I think that given an API change, we won't need to jump through the hoops
you specified.  I think our confusion here is due to my original
oversimplification :)

> > I've done a little playing with this and I think the modifications should
> > be pretty small.

LOL!  Let me restate that as: the modifications shouldn't touch /every/
line of code and will *only* require a serious API change :P :)

> Here is one way I can think of using IN-GIVEN-ENVIRONMENT but the
> bound variables are not available in sub-actions.
>
> (defmacro in-given-environment (env action &rest args)
>  (let ((body-result (gensym "BODY-RESULT"))
>        (environment (gensym "ENVIRONMENT")))
>    `(let ((,environment ,env))          ; <-- not GENSYM'd
>      (flet ((getvar (var) (cadr (assoc var ,environment))))
>        (let ((,body-result ,action))
>          (if (functionp ,body-result)
>              (funcall ,body-result ,@args)
>              ,body-result))))))
>
> CL-USER> (in-given-environment '((a "one") (b 'ii))
>           #'(lambda (arg)(defmacro in-given-environment (env action &rest args)
>               (list (getvar 'a) (getvar 'b) arg))
>           3)
> ("one" 'II 3)
>
> So A and B of this ENVIRONMENT will not be available below.  But as we
> discussed earlier, that is possible only with dynamic variables.

>
> Of course, the above macro won't work in real application code.  It
> needs a slight modification.  We cannot write (flet ((GETVAR (var)...)
> It would have to be (flet ((,(intern "GETVAR") (var) ...) so that
> GETVAR is available in the current package.
>
> Or we could have ORG.PREWETT.LOGS export GETVAR, I think.
>
> What do you think, Jim?

I'm not sure there is actually a whole lot of point :)

Imagine if we had an exported function, getvar, that possibly looked like
this:

(defun getvar (variable-name environment)
  (cadr
   (assoc variable-name environment)))

Also imagine that the API has been changed as per above.  If we want a
rule that matches ssh login messages and, in response, prints a
message indicating the name of the user and the host the login
occurred on (a horribly contrived example :), we could write a rule like
this (for clarity, without using RDL macros):

(make-instance
 'rule
 :match
 (lambda (message environment)
   (multiple-value-bind
         (matches sub-matches)
       (scan-to-strings
        "(\\S+) sshd\\[\\d+\\] session opened for user (\\S+)"
        (message message))
     (when matches
       (values matches
               `((host ,(aref sub-matches 0))
                 (user ,(aref sub-matches 1)))))))
 :actions
 (list
  (lambda (message environment)
    (format t "Saw user: ~A login on host: ~A~%"
            (getvar 'user environment)
            (getvar 'host environment)))))

Then, not only can we call sub-actions (which, could, in my dreams, come
from a vendor who provides only pre-compiled Lisp files) as long as we
remember to propogate the environment to them, we can also, as a user (and
without mucking with any of the internals of LoGS) modify that environment
for the purposes of calling these sub-actions (which, frankly, I'm not
sure how it could ever be used :) .

I think the macro you're proposing, while possibly quite useful when
writing our action functions, doesn't really solve the problem.  the
current codebase expects in-given-environment to work with not only code
snippets, but references to actual functions like #'foo; I think the API
change is the only way to sensibly remove the dynamic variables and still
keep the same functionality.

am I making any sense? :)

Jim


_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

Vijay Lakshminarayanan
On 1/10/07, Jim Prewett <[hidden email]> wrote:

> > ((lambda (message environment)
> >   (declare (ignorable environment))
> >   (funcall action message))
> > message
> > environment)
>
> I'm not quite sure what you mean.
>
> I do believe I was originally underestimating the amount of work required
> :)  I think for what I'm proposing, you would have to *CHANGE THE API* for
> all rule and ruleset match functions, rule action functions, rule and
> ruleset delete rule functions so that they all accepted the environment as
> an argument.  All(?) of the functions that interact with those match, etc.
> functions would also have to be changed.

((lambda (x) (print x)) "hello, world!") is just a fancy way of
FUNCALLing the lambda function.  This way is more efficient (the
compiler just inlines it) is what Paul Graham says in On-Lisp.

> I think that given an API change, we won't need to jump through the hoops
> you specified.  I think our confusion here is due to my original
> oversimplification :)
>

I agree.  An API change is the best option.  I can't believe I even
came up with such an outlandish suggestion.  I blame CL.  What I
suggested wouldn't be possible in another language (Perl is the only
other mainstream language I don't know, so I wouldn't know about it)
:-)

I'll check out the latest versions of LoGS tomorrow.

Thanks
Vijay


_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel
Reply | Threaded
Open this post in threaded view
|

Re: fixing (removing) dynamically bound variables

Jim Prewett

> I agree.  An API change is the best option.  I can't believe I even
> came up with such an outlandish suggestion.  I blame CL.  What I
> suggested wouldn't be possible in another language (Perl is the only
> other mainstream language I don't know, so I wouldn't know about it)
> :-)

Perl has borrowed some of the good ideas from Lisp, but its no Lisp :P :)

> I'll check out the latest versions of LoGS tomorrow.

I tagged 0.1.3 so we could have a "last of" the 0.1.x series and a clear
point when the API would be changed.  0.2.0-proposed is what I'm proposing
to become the 0.2.0 branch.  0.1.3 shouldn't be very interesting :)

I'd really like to hear what you think of 0.2.0-proposed.  I think I've
actually got it committed to CVS now (Friday, 1/12 11:52 PM MST)

Jim


_______________________________________________
LoGS-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/logs-devel