mir/apparmor
2
0
Fork 0
mirror of https://gitlab.com/apparmor/apparmor synced 2025-08-22 10:07:12 +00:00
Code Issues Releases Wiki Activity
apparmor/parser/libapparmor_re/hfa.cc

1757 lines
50 KiB
C++
Raw Normal View History

Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/*
update copyright dates
2007-04-11 08:12:51 +00:00
* (C) 2006, 2007 Andreas Gruenbacher <agruen@suse.de>
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
* Copyright (c) 2003-2008 Novell, Inc. (All rights reserved)
* Copyright 2009-2010 Canonical Ltd.
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
*
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
* The libapparmor library is licensed under the terms of the GNU
* Lesser General Public License, version 2.1. Please see the file
* COPYING.LGPL.
Update and expand comments on regex tree normalization
2010-08-04 10:23:22 -07:00
*
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
Update and expand comments on regex tree normalization
2010-08-04 10:23:22 -07:00
*
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
* You should have received a copy of the GNU Lesser General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
Update and expand comments on regex tree normalization
2010-08-04 10:23:22 -07:00
*
*
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
* Base of implementation based on the Lexical Analysis chapter of:
* Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman:
* Compilers: Principles, Techniques, and Tools (The "Dragon Book"),
* Addison-Wesley, 1986.
Improve dfa generation. Apply tree factoring and simplification techniques to reduce the number of states used in computing the dfa. This can have an exponential impact on both space and time for dfa generation.
2008-11-07 13:00:05 +00:00
*/
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
#include <list>
#include <vector>
#include <stack>
#include <set>
#include <map>
#include <ostream>
#include <iostream>
#include <fstream>
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
#include <string.h>
#include <getopt.h>
#include <assert.h>
#include <arpa/inet.h>
#include <iostream>
#include <fstream>
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
#include "expr-tree.h"
#include "parse.h"
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
#include "../immunix.h"
Convert Nodemap comparision to use a hash value. This uses a little more memory than just using the NodeSet size to short circuit comparison but it improves on the case where compared sets have the same size. It is possible that this will slow down small dfa generation slightly but the trade off for large dfa's (which are the slow ones to generate) is worth it. This results in another performance bump over using the NodeSize is NodeSet comparison, and the amount of improvement increases with larger dfas
2010-11-09 11:20:08 -08:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
class State;
/**
* State cases are identical to NodesCases except they map to State *
* instead of NodeSet.
* Out-edges from a state to another: we store the follow State
* for each input character that is not a default match in cases and
* default matches in otherwise as well as in all matching explicit cases
* This avoids enumerating all the explicit tranitions for default matches.
*/
typedef struct Cases {
typedef map<uchar, State *>::iterator iterator;
iterator begin() { return cases.begin(); }
iterator end() { return cases.end(); }
Cases() : otherwise(0) { }
map<uchar, State *> cases;
State *otherwise;
} Cases;
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
typedef list<State *> Partition;
Move state label, nodes, and permission setting into the State constructor Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:14:12 -08:00
uint32_t accept_perms(NodeSet *state, uint32_t *audit_ctl, int *error);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
/*
* State - DFA individual state information
Embedded the temporary computed nodes as part of the state Embedding the nodes are part of the state gives fast back reference from the state to the nodes that created it. This is useful for the state to nodes mapping dump as it lets us output the states in order. It will also let us avoid certain nodemap lookup in the future. Overlay the nodes field (used only in dfa construction) with the partition field which is only used during dfa minimization to avoid making the state any larger. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:08:02 -08:00
* label: a unique label to identify the state used for pretty printing
* the non-matching state is setup to have label == 0 and
* the start state is setup to have label == 1
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
* audit: the audit permission mask for the state
* accept: the accept permissions for the state
* cases: set of transitions from this state
Embedded the temporary computed nodes as part of the state Embedding the nodes are part of the state gives fast back reference from the state to the nodes that created it. This is useful for the state to nodes mapping dump as it lets us output the states in order. It will also let us avoid certain nodemap lookup in the future. Overlay the nodes field (used only in dfa construction) with the partition field which is only used during dfa minimization to avoid making the state any larger. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:08:02 -08:00
* parition: Is a temporary work variable used during dfa minimization.
* it can be replaced with a map, but that is slower and uses more
* memory.
* nodes: Is a temporary work variable used during dfa creation. It can
* be replaced by using the nodemap, but that is slower
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
*/
class State {
public:
Move state label, nodes, and permission setting into the State constructor Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:14:12 -08:00
State() : label (0), audit(0), accept(0), cases(), nodes(NULL) { };
State(int l): label (l), audit(0), accept(0), cases(), nodes(NULL) { };
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
State(int l, NodeSet *n) throw (int):
Move state label, nodes, and permission setting into the State constructor Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:14:12 -08:00
label(l), audit(0), accept(0), cases(), nodes(n)
{
int error;
/* Compute permissions associated with the State. */
accept = accept_perms(nodes, &audit, &error);
if (error) {
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
cerr << "Failing on accept perms " << error << "\n";
throw error;
Move state label, nodes, and permission setting into the State constructor Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:14:12 -08:00
}
};
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
int label;
uint32_t audit, accept;
Cases cases;
Embedded the temporary computed nodes as part of the state Embedding the nodes are part of the state gives fast back reference from the state to the nodes that created it. This is useful for the state to nodes mapping dump as it lets us output the states in order. It will also let us avoid certain nodemap lookup in the future. Overlay the nodes field (used only in dfa construction) with the partition field which is only used during dfa minimization to avoid making the state any larger. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:08:02 -08:00
union {
Partition *partition;
NodeSet *nodes;
};
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
};
ostream& operator<<(ostream& os, const State& state)
{
Fix comment about what state information is being dumped Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:06:52 -08:00
/* dump the state label */
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
os << '{';
os << state.label;
os << '}';
return os;
}
Convert Nodemap comparision to use a hash value. This uses a little more memory than just using the NodeSet size to short circuit comparison but it improves on the case where compared sets have the same size. It is possible that this will slow down small dfa generation slightly but the trade off for large dfa's (which are the slow ones to generate) is worth it. This results in another performance bump over using the NodeSize is NodeSet comparison, and the amount of improvement increases with larger dfas
2010-11-09 11:20:08 -08:00
typedef map<pair<unsigned long, NodeSet *>, State *, deref_less_than > NodeMap;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/* Transitions in the DFA. */
Factor adding a new state to the dfa the map into its own function Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:16:38 -08:00
/* dfa_stats - structure to group various stats about dfa creation
* duplicates - how many duplicate NodeSets where encountered and discarded
* proto_max - maximum length of a NodeSet encountered during dfa construction
* proto_sum - sum of NodeSet length during dfa construction. Used to find
* average length.
*/
typedef struct dfa_stats {
unsigned int duplicates, proto_max, proto_sum;
} dfa_stats_t;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
class DFA {
Embedded the temporary computed nodes as part of the state Embedding the nodes are part of the state gives fast back reference from the state to the nodes that created it. This is useful for the state to nodes mapping dump as it lets us output the states in order. It will also let us avoid certain nodemap lookup in the future. Overlay the nodes field (used only in dfa construction) with the partition field which is only used during dfa minimization to avoid making the state any larger. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:08:02 -08:00
void dump_node_to_dfa(void);
Factor adding a new state to the dfa the map into its own function Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:16:38 -08:00
State* add_new_state(NodeMap &nodemap, pair <unsigned long, NodeSet *> index, NodeSet *nodes, dfa_stats_t &stats);
Make the work_queue be a work_queue of states that need finished computing With the addition of the nodes field to the state we can make the work queue, be based off of the state instead of the node, and avoid doing the node to map lookup to get back to the state. This means that the NodeMap is now only used for duplicate elimination. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:19:47 -08:00
void update_state_transitions(NodeMap &nodemap, list <State *> &work_queue, State *state, dfa_stats_t &stats);
Rework the code so that update for nodes is now a function The other changes have made it so that using a macro really isn't justified so rework the code to get rid of the hiddeous update_for_nodes macro. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:20:32 -08:00
State *find_target_state(NodeMap &nodemap, list <State *> &work_queue,
NodeSet *nodes, dfa_stats_t &stats);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
public:
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
DFA(Node *root, dfaflags_t flags);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
virtual ~DFA();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
void remove_unreachable(dfaflags_t flags);
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
bool same_mappings(State *s1, State *s2);
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
size_t hash_trans(State *s);
void minimize(dfaflags_t flags);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
void dump(ostream& os);
void dump_dot_graph(ostream& os);
Add ability to dump unique permission sets
2010-11-09 11:56:28 -08:00
void dump_uniq_perms(const char *s);
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
map<uchar, uchar> equivalence_classes(dfaflags_t flags);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
void apply_equivalence_classes(map<uchar, uchar>& eq);
Node *root;
State *nonmatching, *start;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Partition states;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
};
Factor adding a new state to the dfa the map into its own function Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:16:38 -08:00
State* DFA::add_new_state(NodeMap &nodemap, pair <unsigned long, NodeSet *> index, NodeSet *nodes, dfa_stats_t &stats)
{
State *state = new State(nodemap.size(), nodes);
states.push_back(state);
nodemap.insert(make_pair(index, state));
stats.proto_sum += nodes->size();
if (nodes->size() > stats.proto_max)
stats.proto_max = nodes->size();
return state;
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Rework the code so that update for nodes is now a function The other changes have made it so that using a macro really isn't justified so rework the code to get rid of the hiddeous update_for_nodes macro. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:20:32 -08:00
State *DFA::find_target_state(NodeMap &nodemap, list <State *> &work_queue,
NodeSet *nodes, dfa_stats_t &stats)
{
State *target;
pair <unsigned long, NodeSet *> index = make_pair(hash_NodeSet(nodes), nodes);
map<pair <unsigned long, NodeSet *>, State *, deref_less_than>::iterator x = nodemap.find(index);
if (x == nodemap.end()) {
/* set of nodes isn't known so create new state, and nodes to
* state mapping
*/
target = add_new_state(nodemap, index, nodes, stats);
work_queue.push_back(target);
} else {
/* set of nodes already has a mapping so free this one */
stats.duplicates++;
delete (nodes);
target = x->second;
}
return target;
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Factor updating the state transitions into its own fn Factoring the updating of the state transitions doesn't save on any code but it provides a nice logical seperation and makes the dfa work_queue loop and the updating of the state transitions easier to understand as units. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:18:48 -08:00
void DFA::update_state_transitions(NodeMap &nodemap,
Make the work_queue be a work_queue of states that need finished computing With the addition of the nodes field to the state we can make the work queue, be based off of the state instead of the node, and avoid doing the node to map lookup to get back to the state. This means that the NodeMap is now only used for duplicate elimination. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:19:47 -08:00
list <State *> &work_queue, State *state,
Factor updating the state transitions into its own fn Factoring the updating of the state transitions doesn't save on any code but it provides a nice logical seperation and makes the dfa work_queue loop and the updating of the state transitions easier to understand as units. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:18:48 -08:00
dfa_stats_t &stats)
{
/* Compute possible transitions for state->nodes. This is done by
* iterating over all the nodes in state->nodes and combining the
* transitions.
*
* The resultant transition set is a mapping of characters to
* sets of nodes.
*/
NodeCases cases;
for (NodeSet::iterator i = state->nodes->begin(); i != state->nodes->end(); i++)
(*i)->follow(cases);
/* Now for each set of nodes in the computed transitions, make
* sure that there is a state that maps to it, and add the
* matching case to the state.
*/
/* check the default transition first */
Rework the code so that update for nodes is now a function The other changes have made it so that using a macro really isn't justified so rework the code to get rid of the hiddeous update_for_nodes macro. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:20:32 -08:00
if (cases.otherwise)
state->cases.otherwise = find_target_state(nodemap, work_queue,
cases.otherwise,
stats);;
Factor updating the state transitions into its own fn Factoring the updating of the state transitions doesn't save on any code but it provides a nice logical seperation and makes the dfa work_queue loop and the updating of the state transitions easier to understand as units. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:18:48 -08:00
/* For each transition from *from, check if the set of nodes it
* transitions to already has been mapped to a state
*/
for (NodeCases::iterator j = cases.begin(); j != cases.end(); j++) {
State *target;
Rework the code so that update for nodes is now a function The other changes have made it so that using a macro really isn't justified so rework the code to get rid of the hiddeous update_for_nodes macro. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:20:32 -08:00
target = find_target_state(nodemap, work_queue, j->second,
stats);
Factor updating the state transitions into its own fn Factoring the updating of the state transitions doesn't save on any code but it provides a nice logical seperation and makes the dfa work_queue loop and the updating of the state transitions easier to understand as units. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:18:48 -08:00
/* Don't insert transition that the default transition
* already covers
*/
if (target != state->cases.otherwise)
state->cases.cases[j->first] = target;
}
}
Embedded the temporary computed nodes as part of the state Embedding the nodes are part of the state gives fast back reference from the state to the nodes that created it. This is useful for the state to nodes mapping dump as it lets us output the states in order. It will also let us avoid certain nodemap lookup in the future. Overlay the nodes field (used only in dfa construction) with the partition field which is only used during dfa minimization to avoid making the state any larger. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:08:02 -08:00
/* WARNING: This routine can only be called from within DFA creation as
* the nodes value is only valid during dfa construction.
*/
void DFA::dump_node_to_dfa(void)
Add the ability to dump NodeSet to dfa state mapping
2010-11-09 11:55:40 -08:00
{
cerr << "Mapping of States to expr nodes\n"
" State <= Nodes\n"
"-------------------\n";
Embedded the temporary computed nodes as part of the state Embedding the nodes are part of the state gives fast back reference from the state to the nodes that created it. This is useful for the state to nodes mapping dump as it lets us output the states in order. It will also let us avoid certain nodemap lookup in the future. Overlay the nodes field (used only in dfa construction) with the partition field which is only used during dfa minimization to avoid making the state any larger. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:08:02 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++)
cerr << " " << (*i)->label << " <= " << *(*i)->nodes << "\n";
Add the ability to dump NodeSet to dfa state mapping
2010-11-09 11:55:40 -08:00
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/**
* Construct a DFA from a syntax tree.
*/
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
DFA::DFA(Node *root, dfaflags_t flags) : root(root)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
Group dfa stats into a single structure. Move the dfa stats into a structure to provide a single access point to them. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:12:50 -08:00
dfa_stats_t stats = { 0, 0, 0 };
int i = 0;
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (flags & DFA_DUMP_PROGRESS)
fprintf(stderr, "Creating dfa:\r");
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (depth_first_traversal i(root); i; i++) {
(*i)->compute_nullable();
(*i)->compute_firstpos();
(*i)->compute_lastpos();
}
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (flags & DFA_DUMP_PROGRESS)
fprintf(stderr, "Creating dfa: followpos\r");
for (depth_first_traversal i(root); i; i++) {
(*i)->compute_followpos();
}
NodeMap nodemap;
NodeSet *emptynode = new NodeSet;
Factor adding a new state to the dfa the map into its own function Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:16:38 -08:00
nonmatching = add_new_state(nodemap,
make_pair(hash_NodeSet(emptynode), emptynode),
emptynode, stats);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
NodeSet *first = new NodeSet(root->firstpos);
Factor adding a new state to the dfa the map into its own function Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:16:38 -08:00
start = add_new_state(nodemap, make_pair(hash_NodeSet(first), first),
first, stats);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Make the work_queue be a work_queue of states that need finished computing With the addition of the nodes field to the state we can make the work queue, be based off of the state instead of the node, and avoid doing the node to map lookup to get back to the state. This means that the NodeMap is now only used for duplicate elimination. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:19:47 -08:00
/* the work_queue contains the states that need to have their
* transitions computed. This could be done with a recursive
* algorithm instead of a work_queue, but it would be slightly slower
* and consume more memory.
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
*
* TODO: currently the work_queue is treated in a breadth first
* search manner. Test using the work_queue in a depth first
* manner, this may help reduce the number of entries on the
* work_queue at any given time, thus reducing peak memory use.
*/
Make the work_queue be a work_queue of states that need finished computing With the addition of the nodes field to the state we can make the work queue, be based off of the state instead of the node, and avoid doing the node to map lookup to get back to the state. This means that the NodeMap is now only used for duplicate elimination. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:19:47 -08:00
list<State *> work_queue;
work_queue.push_back(start);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
while (!work_queue.empty()) {
if (i % 1000 == 0 && (flags & DFA_DUMP_PROGRESS))
Group dfa stats into a single structure. Move the dfa stats into a structure to provide a single access point to them. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:12:50 -08:00
fprintf(stderr, "\033[2KCreating dfa: queue %ld\tstates %ld\teliminated duplicates %d\r", work_queue.size(), states.size(), stats.duplicates);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
i++;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Make the work_queue be a work_queue of states that need finished computing With the addition of the nodes field to the state we can make the work queue, be based off of the state instead of the node, and avoid doing the node to map lookup to get back to the state. This means that the NodeMap is now only used for duplicate elimination. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:19:47 -08:00
State *from = work_queue.front();
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
work_queue.pop_front();
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Factor updating the state transitions into its own fn Factoring the updating of the state transitions doesn't save on any code but it provides a nice logical seperation and makes the dfa work_queue loop and the updating of the state transitions easier to understand as units. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:18:48 -08:00
/* Update 'from's transitions, and if it transitions to any
* unknown State create it and add it to the work_queue
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
*/
Factor updating the state transitions into its own fn Factoring the updating of the state transitions doesn't save on any code but it provides a nice logical seperation and makes the dfa work_queue loop and the updating of the state transitions easier to understand as units. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:18:48 -08:00
update_state_transitions(nodemap, work_queue, from, stats);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
} /* for (NodeSet *nodes ... */
/* cleanup Sets of nodes used computing the DFA as they are no longer
* needed.
*/
for (depth_first_traversal i(root); i; i++) {
(*i)->firstpos.clear();
(*i)->lastpos.clear();
(*i)->followpos.clear();
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
Add the ability to dump NodeSet to dfa state mapping
2010-11-09 11:55:40 -08:00
if (flags & DFA_DUMP_NODE_TO_DFA)
Embedded the temporary computed nodes as part of the state Embedding the nodes are part of the state gives fast back reference from the state to the nodes that created it. This is useful for the state to nodes mapping dump as it lets us output the states in order. It will also let us avoid certain nodemap lookup in the future. Overlay the nodes field (used only in dfa construction) with the partition field which is only used during dfa minimization to avoid making the state any larger. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:08:02 -08:00
dump_node_to_dfa();
Add the ability to dump NodeSet to dfa state mapping
2010-11-09 11:55:40 -08:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (NodeMap::iterator i = nodemap.begin(); i != nodemap.end(); i++)
Convert Nodemap comparision to use a hash value. This uses a little more memory than just using the NodeSet size to short circuit comparison but it improves on the case where compared sets have the same size. It is possible that this will slow down small dfa generation slightly but the trade off for large dfa's (which are the slow ones to generate) is worth it. This results in another performance bump over using the NodeSize is NodeSet comparison, and the amount of improvement increases with larger dfas
2010-11-09 11:20:08 -08:00
delete i->first.second;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
nodemap.clear();
When creating the dfa the sets firstpos, lastpos, and followpos are computed for each expression tree node and then used as input to create the dfa states. Currently they are not being freed until the nodes are destroyed, but the information is no longer needed once the dfa has been created. Cleaning them up early reduces peak memory usage.
2010-07-10 17:47:25 -07:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (flags & (DFA_DUMP_STATS))
Group dfa stats into a single structure. Move the dfa stats into a structure to provide a single access point to them. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-11-11 16:12:50 -08:00
fprintf(stderr, "\033[2KCreated dfa: states %ld,\teliminated duplicates %d,\tprotostate sets: longest %u, avg %u\n", states.size(), stats.duplicates, stats.proto_max, (unsigned int) (stats.proto_sum/states.size()));
When creating the dfa the sets firstpos, lastpos, and followpos are computed for each expression tree node and then used as input to create the dfa states. Currently they are not being freed until the nodes are destroyed, but the information is no longer needed once the dfa has been created. Cleaning them up early reduces peak memory usage.
2010-07-10 17:47:25 -07:00
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
DFA::~DFA()
{
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
delete *i;
}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
class MatchFlag : public AcceptNode {
public:
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
MatchFlag(uint32_t flag, uint32_t audit) : flag(flag), audit(audit) {}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
ostream& dump(ostream& os)
{
return os << '<' << flag << '>';
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
uint32_t flag;
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
uint32_t audit;
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
};
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
class ExactMatchFlag : public MatchFlag {
public:
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
ExactMatchFlag(uint32_t flag, uint32_t audit) : MatchFlag(flag, audit) {}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
};
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
class DenyMatchFlag : public MatchFlag {
public:
DenyMatchFlag(uint32_t flag, uint32_t quiet) : MatchFlag(flag, quiet) {}
};
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Add ability to dump unique permission sets
2010-11-09 11:56:28 -08:00
void DFA::dump_uniq_perms(const char *s)
{
set < pair<uint32_t, uint32_t> > uniq;
for (Partition::iterator i = states.begin(); i != states.end(); i++)
uniq.insert(make_pair((*i)->accept, (*i)->audit));
cerr << "Unique Permission sets: " << s << " (" << uniq.size() << ")\n";
cerr << "----------------------\n";
for (set< pair<uint32_t, uint32_t> >::iterator i = uniq.begin();
i != uniq.end(); i++) {
cerr << " " << hex << i->first << " " << i->second << dec <<"\n";
}
}
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
/* Remove dead or unreachable states */
void DFA::remove_unreachable(dfaflags_t flags)
{
set <State *> reachable;
list <State *> work_queue;
/* find the set of reachable states */
reachable.insert(nonmatching);
work_queue.push_back(start);
while (!work_queue.empty()) {
State *from = work_queue.front();
work_queue.pop_front();
reachable.insert(from);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (from->cases.otherwise &&
(reachable.find(from->cases.otherwise) == reachable.end()))
work_queue.push_back(from->cases.otherwise);
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Cases::iterator j = from->cases.begin();
j != from->cases.end(); j++) {
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (reachable.find(j->second) == reachable.end())
work_queue.push_back(j->second);
}
}
/* walk the set of states and remove any that aren't reachable */
if (reachable.size() < states.size()) {
int count = 0;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Partition::iterator i;
Partition::iterator next;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
for (i = states.begin(); i != states.end(); i = next) {
next = i;
next++;
if (reachable.find(*i) == reachable.end()) {
if (flags & DFA_DUMP_UNREACHABLE) {
cerr << "unreachable: "<< **i;
if (*i == start)
cerr << " <==";
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if ((*i)->accept) {
cerr << " (0x" << hex << (*i)->accept
<< " " << (*i)->audit << dec << ')';
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
cerr << endl;
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
State *current = *i;
states.erase(i);
delete(current);
count++;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
}
if (count && (flags & DFA_DUMP_STATS))
cerr << "DFA: states " << states.size() << " removed "
<< count << " unreachable states\n";
}
}
/* test if two states have the same transitions under partition_map */
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
bool DFA::same_mappings(State *s1, State *s2)
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
{
When hashing Nodes ensure that cases.otherwise == NULL is treated the same as pointing to the nonmatching state. Having this mix shouldn't currently exist but adding the extra check makes the code more robust.
2010-11-09 11:25:44 -08:00
if (s1->cases.otherwise && s1->cases.otherwise != nonmatching) {
if (!s2->cases.otherwise || s2->cases.otherwise == nonmatching)
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
return false;
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
Partition *p1 = s1->cases.otherwise->partition;
Partition *p2 = s2->cases.otherwise->partition;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (p1 != p2)
return false;
When hashing Nodes ensure that cases.otherwise == NULL is treated the same as pointing to the nonmatching state. Having this mix shouldn't currently exist but adding the extra check makes the code more robust.
2010-11-09 11:25:44 -08:00
} else if (s2->cases.otherwise && s2->cases.otherwise != nonmatching) {
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
return false;
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (s1->cases.cases.size() != s2->cases.cases.size())
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
return false;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Cases::iterator j1 = s1->cases.begin(); j1 != s1->cases.end();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
j1++){
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Cases::iterator j2 = s2->cases.cases.find(j1->first);
if (j2 == s2->cases.end())
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
return false;
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
Partition *p1 = j1->second->partition;
Partition *p2 = j2->second->partition;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (p1 != p2)
return false;
}
return true;
}
/* Do simple djb2 hashing against a States transition cases
* this provides a rough initial guess at state equivalence as if a state
* has a different number of transitions or has transitions on different
* cases they will never be equivalent.
* Note: this only hashes based off of the alphabet (not destination)
* as different destinations could end up being equiv
*/
size_t DFA::hash_trans(State *s)
{
unsigned long hash = 5381;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Cases::iterator j = s->cases.begin(); j != s->cases.end(); j++){
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
hash = ((hash << 5) + hash) + j->first;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
State *k = j->second;
hash = ((hash << 5) + hash) + k->cases.cases.size();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (s->cases.otherwise && s->cases.otherwise != nonmatching) {
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
hash = ((hash << 5) + hash) + 5381;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
State *k = s->cases.otherwise;
hash = ((hash << 5) + hash) + k->cases.cases.size();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
Update hash calculation Update the hash calculation to guarentee that states with a different number of transition entries will be placed in seperate partitions. This will allow for a better character transition based state comparison.
2010-01-20 05:10:38 -08:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
hash = (hash << 8) | s->cases.cases.size();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
return hash;
}
/* minimize the number of dfa states */
void DFA::minimize(dfaflags_t flags)
{
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
map <pair <uint64_t, size_t>, Partition *> perm_map;
list <Partition *> partitions;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
Split dfa minimizing hashing into two seperately controllable hashes. The first hash does hashing on state just state transitions, which always results in a performance improvement. The second does hashing based off of accept permissions, which can create more initial states but can result in not being able to achieve a true minimum dfa. This can also lead to slowing down total dfa creation because while minimization, compression can take longer if the dfa isn't completely minimized. permission hashing is currently required, as minimization does not accumulate redundant Node permissions.
2010-11-09 11:22:54 -08:00
/* Set up the initial partitions
* minimium of - 1 non accepting, and 1 accepting
* if trans hashing is used the accepting and non-accepting partitions
* can be further split based on the number and type of transitions
* a state makes.
* If permission hashing is enabled the accepting partitions can
* be further divided by permissions. This can result in not
* obtaining a truely minimized dfa but comes close, and can speedup
* minimization.
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
*/
int accept_count = 0;
Update state progress/stats output to dump the number of accepting states/partitions occur in the minimized dfa.
2010-11-09 11:26:50 -08:00
int final_accept = 0;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++) {
Split dfa minimizing hashing into two seperately controllable hashes. The first hash does hashing on state just state transitions, which always results in a performance improvement. The second does hashing based off of accept permissions, which can create more initial states but can result in not being able to achieve a true minimum dfa. This can also lead to slowing down total dfa creation because while minimization, compression can take longer if the dfa isn't completely minimized. permission hashing is currently required, as minimization does not accumulate redundant Node permissions.
2010-11-09 11:22:54 -08:00
uint64_t perm_hash = 0;
if (flags & DFA_CONTROL_MINIMIZE_HASH_PERMS) {
/* make every unique perm create a new partition */
perm_hash = ((uint64_t)(*i)->audit)<<32 |
(uint64_t)(*i)->accept;
} else if ((*i)->audit || (*i)->accept) {
/* combine all perms together into a single parition */
perm_hash = 1;
} /* else not an accept state so 0 for perm_hash */
size_t trans_hash = 0;
if (flags & DFA_CONTROL_MINIMIZE_HASH_TRANS)
trans_hash = hash_trans(*i);
pair <uint64_t, size_t> group = make_pair(perm_hash, trans_hash);
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
map <pair <uint64_t, size_t>, Partition *>::iterator p = perm_map.find(group);
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (p == perm_map.end()) {
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
Partition *part = new Partition();
Move partitions from using sets to lists as this is a better match for what is being done.
2010-01-31 23:19:54 -08:00
part->push_back(*i);
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
perm_map.insert(make_pair(group, part));
partitions.push_back(part);
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
(*i)->partition = part;
Split dfa minimizing hashing into two seperately controllable hashes. The first hash does hashing on state just state transitions, which always results in a performance improvement. The second does hashing based off of accept permissions, which can create more initial states but can result in not being able to achieve a true minimum dfa. This can also lead to slowing down total dfa creation because while minimization, compression can take longer if the dfa isn't completely minimized. permission hashing is currently required, as minimization does not accumulate redundant Node permissions.
2010-11-09 11:22:54 -08:00
if (perm_hash)
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
accept_count++;
} else {
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
(*i)->partition = p->second;
Move partitions from using sets to lists as this is a better match for what is being done.
2010-01-31 23:19:54 -08:00
p->second->push_back(*i);
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if ((flags & DFA_DUMP_PROGRESS) &&
(partitions.size() % 1000 == 0))
Update state progress/stats output to dump the number of accepting states/partitions occur in the minimized dfa.
2010-11-09 11:26:50 -08:00
cerr << "\033[2KMinimize dfa: partitions " << partitions.size() << "\tinit " << partitions.size() << " (accept " << accept_count << ")\r";
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
Cleanup the perm_map as soon as it is no longer needed. Cleaning up the map before the end of the functions reduces the peak memory of the function
2010-11-09 11:26:18 -08:00
/* perm_map is no longer needed so free the memory it is using.
* Don't remove - doing it manually here helps reduce peak memory usage.
*/
perm_map.clear();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
int init_count = partitions.size();
if (flags & DFA_DUMP_PROGRESS)
Update state progress/stats output to dump the number of accepting states/partitions occur in the minimized dfa.
2010-11-09 11:26:50 -08:00
cerr << "\033[2KMinimize dfa: partitions " << partitions.size() << "\tinit " << init_count << " (accept " << accept_count << ")\r";
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
/* Now do repartitioning until each partition contains the set of
* states that are the same. This will happen when the partition
* splitting stables. With a worse case of 1 state per partition
* ie. already minimized.
*/
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
Partition *new_part;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
int new_part_count;
do {
new_part_count = 0;
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
for (list <Partition *>::iterator p = partitions.begin();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
p != partitions.end(); p++) {
new_part = NULL;
State *rep = *((*p)->begin());
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
Partition::iterator next;
for (Partition::iterator s = ++(*p)->begin();
Rework the partitioning to take advantage of Partitions now being a list
2010-01-31 23:21:00 -08:00
s != (*p)->end(); ) {
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
if (same_mappings(rep, *s)) {
Rework the partitioning to take advantage of Partitions now being a list
2010-01-31 23:21:00 -08:00
++s;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
continue;
Rework the partitioning to take advantage of Partitions now being a list
2010-01-31 23:21:00 -08:00
}
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (!new_part) {
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
new_part = new Partition;
Rework the partitioning to take advantage of Partitions now being a list
2010-01-31 23:21:00 -08:00
list <Partition *>::iterator tmp = p;
partitions.insert(++tmp, new_part);
new_part_count++;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
Move partitions from using sets to lists as this is a better match for what is being done.
2010-01-31 23:19:54 -08:00
new_part->push_back(*s);
Rework the partitioning to take advantage of Partitions now being a list
2010-01-31 23:21:00 -08:00
s = (*p)->erase(s);
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
Rework the partitioning to take advantage of Partitions now being a list
2010-01-31 23:21:00 -08:00
/* remapping partition_map for new_part entries
* Do not do this above as it messes up same_mappings
*/
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (new_part) {
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
for (Partition::iterator m = new_part->begin();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
m != new_part->end(); m++) {
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
(*m)->partition = new_part;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
}
if ((flags & DFA_DUMP_PROGRESS) &&
Improve partitioning performance slightly by inserting new partitions imediately after the current partition being considered, instead of at the back of the parition list. This does two things, it makes it more likely the data is in cache, and it also in general results in more partitions being created in a single pass.
2010-01-31 23:12:33 -08:00
(partitions.size() % 100 == 0))
Update state progress/stats output to dump the number of accepting states/partitions occur in the minimized dfa.
2010-11-09 11:26:50 -08:00
cerr << "\033[2KMinimize dfa: partitions " << partitions.size() << "\tinit " << init_count << " (accept " << accept_count << ")\r";
Improve partitioning performance slightly by inserting new partitions imediately after the current partition being considered, instead of at the back of the parition list. This does two things, it makes it more likely the data is in cache, and it also in general results in more partitions being created in a single pass.
2010-01-31 23:12:33 -08:00
}
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
} while(new_part_count);
Update state progress/stats output to dump the number of accepting states/partitions occur in the minimized dfa.
2010-11-09 11:26:50 -08:00
if (partitions.size() == states.size()) {
if (flags & DFA_DUMP_STATS)
cerr << "\033[2KDfa minimization no states removed: partitions " << partitions.size() << "\tinit " << init_count << " (accept " << accept_count << ")\n";
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
goto out;
}
/* Remap the dfa so it uses the representative states
* Use the first state of a partition as the representative state
* At this point all states with in a partion have transitions
Do not use permission hashing for minimization by default. While this improves minimization performance, it can slow down total creation time and result in larger compressed dfas. This is because it results in the dfa not being completely minimized which with the current O(n2) dfa table compression algorithm can result in slower compressed dfa generation.
2010-11-09 11:27:36 -08:00
* to states within the same partitions, however this can slow
* down compressed dfa compression as there are more states,
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
*/
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
for (list <Partition *>::iterator p = partitions.begin();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
p != partitions.end(); p++) {
/* representative state for this partition */
State *rep = *((*p)->begin());
/* update representative state's transitions */
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (rep->cases.otherwise) {
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
Partition *partition = rep->cases.otherwise->partition;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
rep->cases.otherwise = *partition->begin();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Cases::iterator c = rep->cases.begin();
c != rep->cases.end(); c++) {
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
Partition *partition = c->second->partition;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
c->second = *partition->begin();
}
//if ((*p)->size() > 1)
//cerr << rep->label << ": ";
Do permission accumulation in dfa minimization. This is necessary if accept states with different permissions are to ever share a partition.
2010-11-09 11:24:51 -08:00
/* clear the state label for all non representative states,
* and accumulate permissions */
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = ++(*p)->begin(); i != (*p)->end(); i++) {
//cerr << " " << (*i)->label;
(*i)->label = -1;
Do permission accumulation in dfa minimization. This is necessary if accept states with different permissions are to ever share a partition.
2010-11-09 11:24:51 -08:00
rep->accept |= (*i)->accept;
rep->audit |= (*i)->audit;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
}
Update state progress/stats output to dump the number of accepting states/partitions occur in the minimized dfa.
2010-11-09 11:26:50 -08:00
if (rep->accept || rep->audit)
final_accept++;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
//if ((*p)->size() > 1)
//cerr << "\n";
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
Update state progress/stats output to dump the number of accepting states/partitions occur in the minimized dfa.
2010-11-09 11:26:50 -08:00
if (flags & DFA_DUMP_STATS)
cerr << "\033[2KMinimized dfa: final partitions " << partitions.size() << " (accept " << final_accept << ")" << "\tinit " << init_count << " (accept " << accept_count << ")\n";
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
/* make sure nonmatching and start state are up to date with the
* mappings */
{
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
Partition *partition = nonmatching->partition;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (*partition->begin() != nonmatching) {
nonmatching = *partition->begin();
}
Embedded the State to partition mapping into the State. Embedding the the partition mapping into the State structure significantly speeds up dfa minimization, by converting rbtree finds to straight direct references when checking for same mappings. The overall time improvement is small but it can half the time spent in minimization.
2010-11-09 11:57:43 -08:00
partition = start->partition;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (*partition->begin() != start) {
start = *partition->begin();
}
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
/* Now that the states have been remapped, remove all states
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
* that are not the representive states for their partition, they
* will have a label == -1
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
*/
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); ) {
if ((*i)->label == -1) {
Fix memory leak during dfa minimization. Dfa minimization wasn't deleting the states it eliminated during the minimization process, and hence leaking memory.
2010-03-13 02:23:23 -08:00
State *s = *i;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
i = states.erase(i);
Fix memory leak during dfa minimization. Dfa minimization wasn't deleting the states it eliminated during the minimization process, and hence leaking memory.
2010-03-13 02:23:23 -08:00
delete(s);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
} else
i++;
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
}
out:
/* Cleanup */
while (!partitions.empty()) {
Seperate Partition definition for States. This is a small step to cleaning up the code
2010-01-31 23:18:14 -08:00
Partition *p = partitions.front();
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
partitions.pop_front();
delete(p);
}
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/**
* text-dump the DFA (for debugging).
*/
void DFA::dump(ostream& os)
{
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++) {
if (*i == start || (*i)->accept) {
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
os << **i;
if (*i == start)
os << " <==";
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if ((*i)->accept) {
os << " (0x" << hex << (*i)->accept << " " << (*i)->audit << dec << ')';
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
os << endl;
}
}
os << endl;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++) {
if ((*i)->cases.otherwise)
os << **i << " -> " << (*i)->cases.otherwise << endl;
for (Cases::iterator j = (*i)->cases.begin(); j != (*i)->cases.end(); j++) {
os << **i << " -> " << j->second << ": " << j->first << endl;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
}
os << endl;
}
/**
* Create a dot (graphviz) graph from the DFA (for debugging).
*/
void DFA::dump_dot_graph(ostream& os)
{
os << "digraph \"dfa\" {" << endl;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++) {
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
if (*i == nonmatching)
continue;
os << "\t\"" << **i << "\" [" << endl;
if (*i == start) {
os << "\t\tstyle=bold" << endl;
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
uint32_t perms = (*i)->accept;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
if (perms) {
os << "\t\tlabel=\"" << **i << "\\n("
<< perms << ")\"" << endl;
}
os << "\t]" << endl;
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++) {
Cases& cases = (*i)->cases;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Chars excluded;
for (Cases::iterator j = cases.begin(); j != cases.end(); j++) {
if (j->second == nonmatching)
excluded.insert(j->first);
else {
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
os << "\t\"" << **i << "\" -> \"";
os << j->second << "\" [" << endl;
os << "\t\tlabel=\"" << j->first << "\"" << endl;
os << "\t]" << endl;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
}
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if (cases.otherwise && cases.otherwise != nonmatching) {
os << "\t\"" << **i << "\" -> \"" << cases.otherwise
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
<< "\" [" << endl;
if (!excluded.empty()) {
os << "\t\tlabel=\"[^";
for (Chars::iterator i = excluded.begin();
i != excluded.end();
i++) {
os << *i;
}
os << "]\"" << endl;
}
os << "\t]" << endl;
}
}
os << '}' << endl;
}
/**
* Compute character equivalence classes in the DFA to save space in the
* transition table.
*/
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
map<uchar, uchar> DFA::equivalence_classes(dfaflags_t flags)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
map<uchar, uchar> classes;
uchar next_class = 1;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++) {
Cases& cases = (*i)->cases;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/* Group edges to the same next state together */
map<const State *, Chars> node_sets;
for (Cases::iterator j = cases.begin(); j != cases.end(); j++)
node_sets[j->second].insert(j->first);
for (map<const State *, Chars>::iterator j = node_sets.begin();
j != node_sets.end();
j++) {
/* Group edges to the same next state together by class */
map<uchar, Chars> node_classes;
bool class_used = false;
for (Chars::iterator k = j->second.begin();
k != j->second.end();
k++) {
pair<map<uchar, uchar>::iterator, bool> x =
classes.insert(make_pair(*k, next_class));
if (x.second)
class_used = true;
pair<map<uchar, Chars>::iterator, bool> y =
node_classes.insert(make_pair(x.first->second, Chars()));
y.first->second.insert(*k);
}
if (class_used) {
next_class++;
class_used = false;
}
for (map<uchar, Chars>::iterator k = node_classes.begin();
k != node_classes.end();
k++) {
/**
* If any other characters are in the same class, move
* the characters in this class into their own new class
*/
map<uchar, uchar>::iterator l;
for (l = classes.begin(); l != classes.end(); l++) {
if (l->second == k->first &&
k->second.find(l->first) == k->second.end()) {
class_used = true;
break;
}
}
if (class_used) {
for (Chars::iterator l = k->second.begin();
l != k->second.end();
l++) {
classes[*l] = next_class;
}
next_class++;
class_used = false;
}
}
}
}
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
if (flags & DFA_DUMP_EQUIV_STATS)
fprintf(stderr, "Equiv class reduces to %d classes\n", next_class - 1);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
return classes;
}
/**
* Text-dump the equivalence classes (for debugging).
*/
void dump_equivalence_classes(ostream& os, map<uchar, uchar>& eq)
{
map<uchar, Chars> rev;
for (map<uchar, uchar>::iterator i = eq.begin(); i != eq.end(); i++) {
Chars& chars = rev.insert(make_pair(i->second,
Chars())).first->second;
chars.insert(i->first);
}
os << "(eq):" << endl;
for (map<uchar, Chars>::iterator i = rev.begin(); i != rev.end(); i++) {
os << (int)i->first << ':';
Chars& chars = i->second;
for (Chars::iterator j = chars.begin(); j != chars.end(); j++) {
os << ' ' << *j;
}
os << endl;
}
}
/**
* Replace characters with classes (which are also represented as
* characters) in the DFA transition table.
*/
void DFA::apply_equivalence_classes(map<uchar, uchar>& eq)
{
/**
* Note: We only transform the transition table; the nodes continue to
* contain the original characters.
*/
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = states.begin(); i != states.end(); i++) {
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
map<uchar, State *> tmp;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
tmp.swap((*i)->cases.cases);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
for (Cases::iterator j = tmp.begin(); j != tmp.end(); j++)
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
(*i)->cases.cases.insert(make_pair(eq[j->first], j->second));
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
}
/**
* Flip the children of all cat nodes. This causes strings to be matched
* back-forth.
*/
void flip_tree(Node *node)
{
for (depth_first_traversal i(node); i; i++) {
if (CatNode *cat = dynamic_cast<CatNode *>(*i)) {
Improve dfa generation. Apply tree factoring and simplification techniques to reduce the number of states used in computing the dfa. This can have an exponential impact on both space and time for dfa generation.
2008-11-07 13:00:05 +00:00
swap(cat->child[0], cat->child[1]);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
}
}
class TransitionTable {
typedef vector<pair<const State *, size_t> > DefaultBase;
typedef vector<pair<const State *, const State *> > NextCheck;
public:
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
TransitionTable(DFA& dfa, map<uchar, uchar>& eq, dfaflags_t flags);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
void dump(ostream& os);
void flex_table(ostream& os, const char *name);
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
void init_free_list(vector <pair<size_t, size_t> > &free_list, size_t prev, size_t start);
bool fits_in(vector <pair<size_t, size_t> > &free_list,
size_t base, Cases& cases);
void insert_state(vector <pair<size_t, size_t> > &free_list,
State *state, DFA& dfa);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
private:
vector<uint32_t> accept;
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
vector<uint32_t> accept2;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
DefaultBase default_base;
NextCheck next_check;
map<const State *, size_t> num;
map<uchar, uchar>& eq;
uchar max_eq;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
size_t first_free;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
};
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
void TransitionTable::init_free_list(vector <pair<size_t, size_t> > &free_list,
size_t prev, size_t start) {
for (size_t i = start; i < free_list.size(); i++) {
if (prev)
free_list[prev].second = i;
free_list[i].first = prev;
prev = i;
}
free_list[free_list.size() -1].second = 0;
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/**
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
* new Construct the transition table.
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
*/
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
TransitionTable::TransitionTable(DFA& dfa, map<uchar, uchar>& eq,
dfaflags_t flags)
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
: eq(eq)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
if (flags & DFA_DUMP_TRANS_PROGRESS)
Rename trans-XXXX transition to compress- compression trans- isn't a very good name for this phase of compilation. It is the compression phase, rename to trans- to compress- to reflect this.
2010-11-09 11:49:18 -08:00
fprintf(stderr, "Compressing trans table:\r");
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
if (eq.empty())
max_eq = 255;
else {
max_eq = 0;
for(map<uchar, uchar>::iterator i = eq.begin(); i != eq.end(); i++) {
if (i->second > max_eq)
max_eq = i->second;
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
/* Do initial setup adding up all the transitions and sorting by
* transition count.
*/
size_t optimal = 2;
multimap <size_t, State *> order;
vector <pair<size_t, size_t> > free_list;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = dfa.states.begin(); i != dfa.states.end(); i++) {
if (*i == dfa.start || *i == dfa.nonmatching)
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
continue;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
optimal += (*i)->cases.cases.size();
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
if (flags & DFA_CONTROL_TRANS_HIGH) {
size_t range = 0;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
if ((*i)->cases.cases.size())
range = (*i)->cases.cases.rbegin()->first - (*i)->cases.begin()->first;
size_t ord = ((256 - (*i)->cases.cases.size()) << 8) |
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
(256 - range);
/* reverse sort by entry count, most entries first */
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
order.insert(make_pair(ord, *i));
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
}
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
/* Insert the dummy nonmatching transition by hand */
next_check.push_back(make_pair(dfa.nonmatching, dfa.nonmatching));
default_base.push_back(make_pair(dfa.nonmatching, 0));
num.insert(make_pair(dfa.nonmatching, num.size()));
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
accept.resize(dfa.states.size());
accept2.resize(dfa.states.size());
next_check.resize(optimal);
free_list.resize(optimal);
Update trans table reporting to include some statistics
2010-01-08 05:29:25 -08:00
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
accept[0] = 0;
accept2[0] = 0;
first_free = 1;
init_free_list(free_list, 0, 1);
insert_state(free_list, dfa.start, dfa);
accept[1] = 0;
accept2[1] = 0;
num.insert(make_pair(dfa.start, num.size()));
int count = 2;
if (!(flags & DFA_CONTROL_TRANS_HIGH)) {
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (Partition::iterator i = dfa.states.begin(); i != dfa.states.end();
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
i++) {
if (*i != dfa.nonmatching && *i != dfa.start) {
insert_state(free_list, *i, dfa);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
accept[num.size()] = (*i)->accept;
accept2[num.size()] = (*i)->audit;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
num.insert(make_pair(*i, num.size()));
}
if (flags & (DFA_DUMP_TRANS_PROGRESS)) {
count++;
if (count % 100 == 0)
Rename trans-XXXX transition to compress- compression trans- isn't a very good name for this phase of compilation. It is the compression phase, rename to trans- to compress- to reflect this.
2010-11-09 11:49:18 -08:00
fprintf(stderr, "\033[2KCompressing trans table: insert state: %d/%ld\r", count, dfa.states.size());
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
}
}
} else {
for (multimap <size_t, State *>::iterator i = order.begin();
i != order.end(); i++) {
if (i->second != dfa.nonmatching && i->second != dfa.start) {
insert_state(free_list, i->second, dfa);
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
accept[num.size()] = i->second->accept;
accept2[num.size()] = i->second->audit;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
num.insert(make_pair(i->second, num.size()));
}
if (flags & (DFA_DUMP_TRANS_PROGRESS)) {
count++;
if (count % 100 == 0)
Rename trans-XXXX transition to compress- compression trans- isn't a very good name for this phase of compilation. It is the compression phase, rename to trans- to compress- to reflect this.
2010-11-09 11:49:18 -08:00
fprintf(stderr, "\033[2KCompressing trans table: insert state: %d/%ld\r", count, dfa.states.size());
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
}
}
}
if (flags & (DFA_DUMP_TRANS_STATS | DFA_DUMP_TRANS_PROGRESS)) {
ssize_t size = 4 * next_check.size() + 6 * dfa.states.size();
Rename trans-XXXX transition to compress- compression trans- isn't a very good name for this phase of compilation. It is the compression phase, rename to trans- to compress- to reflect this.
2010-11-09 11:49:18 -08:00
fprintf(stderr, "\033[2KCompressed trans table: states %ld, next/check %ld, optimal next/check %ld avg/state %.2f, compression %ld/%ld = %.2f %%\n", dfa.states.size(), next_check.size(), optimal, (float)next_check.size()/(float)dfa.states.size(), size, 512 * dfa.states.size(), 100.0 - ((float) size * 100.0 / (float)(512 * dfa.states.size())));
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/**
* Does <cases> fit into position <base> of the transition table?
*/
This cleans up a number of warnings that appeared after the parser rework commits were made (as well as a few other minor warnings elsewhere). The Makefile change is to avoid passing -Wstrict-prototypes and -Wnested-externs to the C++ compiler, which the compiler yells about and then ignores. Since we compile with -Wmissing-field-initializers I dropped the unreferenced zero-width fields in the header structs, and then explicitly initialized the remaining fields. I tagged several unused function parameters to silence those warnings. And finally, I dropped the unused filter_escapes() too.
2010-11-09 13:39:18 -08:00
bool TransitionTable::fits_in(vector <pair<size_t, size_t> > &free_list __attribute__((unused)),
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
size_t pos, Cases& cases)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
size_t c, base = pos - cases.begin()->first;
for (Cases::iterator i = cases.begin(); i != cases.end(); i++) {
c = base + i->first;
/* if it overflows the next_check array it fits in as we will
* resize */
if (c >= next_check.size())
This is a preparatory patch for the fix to launchpad.net/bugs/599450. It combines the two separate table resize code segments into a single functionally equivalent segment. It does not fix the bug.
2010-07-23 04:29:54 +02:00
return true;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
if (next_check[c].second)
return false;
}
return true;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
/**
* Insert <state> of <dfa> into the transition table.
*/
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
void TransitionTable::insert_state(vector <pair<size_t, size_t> > &free_list,
State *from, DFA& dfa)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
State *default_state = dfa.nonmatching;
size_t base = 0;
This is a preparatory patch for the fix to launchpad.net/bugs/599450. It combines the two separate table resize code segments into a single functionally equivalent segment. It does not fix the bug.
2010-07-23 04:29:54 +02:00
int resize;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
Cases& cases = from->cases;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
size_t c = cases.begin()->first;
size_t prev = 0;
size_t x = first_free;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
if (cases.otherwise)
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
default_state = cases.otherwise;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
if (cases.cases.empty())
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
goto do_insert;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
repeat:
This is a preparatory patch for the fix to launchpad.net/bugs/599450. It combines the two separate table resize code segments into a single functionally equivalent segment. It does not fix the bug.
2010-07-23 04:29:54 +02:00
resize = 0;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
/* get the first free entry that won't underflow */
while (x && (x < c)) {
prev = x;
x = free_list[x].second;
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
/* try inserting until we succeed. */
while (x && !fits_in(free_list, x, cases)) {
prev = x;
x = free_list[x].second;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
if (!x) {
This is the user space fix for launchpad.net/busgs/599450 It changes the table resizing so that there is always sufficient high entries in the table, preventing bounds violations from occurring. Previously the resize allocation was always based on the character set range for a state, which could be more or less than actually required, and packing would waste some space when over allocation was done. As a result this patch in general results in slightly smaller transition tables even though it enforcing the minimum required padding to avoid bounds violations.
2010-07-23 04:30:31 +02:00
resize = 256 - cases.begin()->first;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
x = free_list.size();
This is a preparatory patch for the fix to launchpad.net/bugs/599450. It combines the two separate table resize code segments into a single functionally equivalent segment. It does not fix the bug.
2010-07-23 04:29:54 +02:00
/* set prev to last free */
This is the user space fix for launchpad.net/busgs/599450 It changes the table resizing so that there is always sufficient high entries in the table, preventing bounds violations from occurring. Previously the resize allocation was always based on the character set range for a state, which could be more or less than actually required, and packing would waste some space when over allocation was done. As a result this patch in general results in slightly smaller transition tables even though it enforcing the minimum required padding to avoid bounds violations.
2010-07-23 04:30:31 +02:00
} else if (x + 255 - cases.begin()->first >= next_check.size()) {
resize = (255 - cases.begin()->first - (next_check.size() - 1 - x));
This is a preparatory patch for the fix to launchpad.net/bugs/599450. It combines the two separate table resize code segments into a single functionally equivalent segment. It does not fix the bug.
2010-07-23 04:29:54 +02:00
for (size_t y = x; y; y = free_list[y].second)
prev = y;
}
if (resize) {
/* expand next_check and free_list */
size_t old_size = free_list.size();
next_check.resize(next_check.size() + resize);
free_list.resize(free_list.size() + resize);
init_free_list(free_list, prev, old_size);
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
if (!first_free)
This is a preparatory patch for the fix to launchpad.net/bugs/599450. It combines the two separate table resize code segments into a single functionally equivalent segment. It does not fix the bug.
2010-07-23 04:29:54 +02:00
first_free = old_size;;
if (x == old_size)
goto repeat;
Speedup transition table compression. This is a basic improvement and not an algorithmic improvement. It does the same basic algorithm of test until it can insert the data, but instead of only tracking the first free entry (and recomputing it each pass). It tracks all free entries reducing the number of comparisons done and the table grows in size. This may actually result in a small loss on small tables, but is a win for larger tables.
2010-01-27 17:20:13 -08:00
}
base = x - c;
for (Cases::iterator j = cases.begin(); j != cases.end(); j++) {
next_check[base + j->first] = make_pair(j->second, from);
size_t prev = free_list[base + j->first].first;
size_t next = free_list[base + j->first].second;
if (prev)
free_list[prev].second = next;
if (next)
free_list[next].first = prev;
if (base + j->first == first_free)
first_free = next;
}
do_insert:
default_base.push_back(make_pair(default_state, base));
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
/**
* Text-dump the transition table (for debugging).
*/
void TransitionTable::dump(ostream& os)
{
map<size_t, const State *> st;
for (map<const State *, size_t>::iterator i = num.begin();
i != num.end();
i++) {
st.insert(make_pair(i->second, i->first));
}
Add extra transition table labeling to help with interpretation of the dump output.
2010-07-23 04:29:29 +02:00
os << "size=" << default_base.size() << " (accept, default, base): {state} -> {default state}" << endl;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
for (size_t i = 0; i < default_base.size(); i++) {
Make the transition table dump easier to understand by labeling each entry with its index.
2010-07-10 17:49:32 -07:00
os << i << ": ";
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
os << "(" << accept[i] << ", "
<< num[default_base[i].first] << ", "
<< default_base[i].second << ")";
if (st[i])
os << " " << *st[i];
if (default_base[i].first)
os << " -> " << *default_base[i].first;
os << endl;
}
Add extra transition table labeling to help with interpretation of the dump output.
2010-07-23 04:29:29 +02:00
os << "size=" << next_check.size() << " (next, check): {check state} -> {next state} : offset from base" << endl;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
for (size_t i = 0; i < next_check.size(); i++) {
if (!next_check[i].second)
continue;
os << i << ": ";
if (next_check[i].second) {
os << "(" << num[next_check[i].first] << ", "
<< num[next_check[i].second] << ")" << " "
<< *next_check[i].second << " -> "
<< *next_check[i].first << ": ";
size_t offs = i - default_base[num[next_check[i].second]].second;
if (eq.size())
os << offs;
else
os << (uchar)offs;
}
os << endl;
}
}
#if 0
template<class Iter>
class FirstIterator {
public:
FirstIterator(Iter pos) : pos(pos) { }
typename Iter::value_type::first_type operator*() { return pos->first; }
bool operator!=(FirstIterator<Iter>& i) { return pos != i.pos; }
void operator++() { ++pos; }
ssize_t operator-(FirstIterator<Iter> i) { return pos - i.pos; }
private:
Iter pos;
};
template<class Iter>
FirstIterator<Iter> first_iterator(Iter iter)
{
return FirstIterator<Iter>(iter);
}
template<class Iter>
class SecondIterator {
public:
SecondIterator(Iter pos) : pos(pos) { }
typename Iter::value_type::second_type operator*() { return pos->second; }
bool operator!=(SecondIterator<Iter>& i) { return pos != i.pos; }
void operator++() { ++pos; }
ssize_t operator-(SecondIterator<Iter> i) { return pos - i.pos; }
private:
Iter pos;
};
template<class Iter>
SecondIterator<Iter> second_iterator(Iter iter)
{
return SecondIterator<Iter>(iter);
}
#endif
/**
* Create a flex-style binary dump of the DFA tables. The table format
* was partly reverse engineered from the flex sources and from
* examining the tables that flex creates with its --tables-file option.
* (Only the -Cf and -Ce formats are currently supported.)
*/
#include "flex-tables.h"
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
#define YYTH_REGEX_MAGIC 0x1B5E783D
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
static inline size_t pad64(size_t i)
{
return (i + (size_t)7) & ~(size_t)7;
}
string fill64(size_t i)
{
const char zeroes[8] = { };
string fill(zeroes, (i & 7) ? 8 - (i & 7) : 0);
return fill;
}
template<class Iter>
size_t flex_table_size(Iter pos, Iter end)
{
return pad64(sizeof(struct table_header) + sizeof(*pos) * (end - pos));
}
template<class Iter>
void write_flex_table(ostream& os, int id, Iter pos, Iter end)
{
This cleans up a number of warnings that appeared after the parser rework commits were made (as well as a few other minor warnings elsewhere). The Makefile change is to avoid passing -Wstrict-prototypes and -Wnested-externs to the C++ compiler, which the compiler yells about and then ignores. Since we compile with -Wmissing-field-initializers I dropped the unreferenced zero-width fields in the header structs, and then explicitly initialized the remaining fields. I tagged several unused function parameters to silence those warnings. And finally, I dropped the unused filter_escapes() too.
2010-11-09 13:39:18 -08:00
struct table_header td = { 0, 0, 0, 0 };
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
size_t size = end - pos;
td.td_id = htons(id);
td.td_flags = htons(sizeof(*pos));
td.td_lolen = htonl(size);
os.write((char *)&td, sizeof(td));
for (; pos != end; ++pos) {
switch(sizeof(*pos)) {
case 4:
os.put((char)(*pos >> 24));
os.put((char)(*pos >> 16));
case 2:
os.put((char)(*pos >> 8));
case 1:
os.put((char)*pos);
}
}
os << fill64(sizeof(td) + sizeof(*pos) * size);
}
void TransitionTable::flex_table(ostream& os, const char *name)
{
const char th_version[] = "notflex";
This cleans up a number of warnings that appeared after the parser rework commits were made (as well as a few other minor warnings elsewhere). The Makefile change is to avoid passing -Wstrict-prototypes and -Wnested-externs to the C++ compiler, which the compiler yells about and then ignores. Since we compile with -Wmissing-field-initializers I dropped the unreferenced zero-width fields in the header structs, and then explicitly initialized the remaining fields. I tagged several unused function parameters to silence those warnings. And finally, I dropped the unused filter_escapes() too.
2010-11-09 13:39:18 -08:00
struct table_set_header th = { 0, 0, 0, 0 };
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/**
* Change the following two data types to adjust the maximum flex
* table size.
*/
typedef uint16_t state_t;
typedef uint32_t trans_t;
if (default_base.size() >= (state_t)-1) {
cerr << "Too many states (" << default_base.size() << ") for "
"type state_t" << endl;
exit(1);
}
if (next_check.size() >= (trans_t)-1) {
cerr << "Too many transitions (" << next_check.size() << ") for "
"type trans_t" << endl;
exit(1);
}
/**
* Create copies of the data structures so that we can dump the tables
* using the generic write_flex_table() routine.
*/
vector<uint8_t> equiv_vec;
if (eq.size()) {
equiv_vec.resize(256);
for (map<uchar, uchar>::iterator i = eq.begin(); i != eq.end(); i++) {
equiv_vec[i->first] = i->second;
}
}
vector<state_t> default_vec;
vector<trans_t> base_vec;
for (DefaultBase::iterator i = default_base.begin();
i != default_base.end();
i++) {
default_vec.push_back(num[i->first]);
base_vec.push_back(i->second);
}
vector<state_t> next_vec;
vector<state_t> check_vec;
for (NextCheck::iterator i = next_check.begin();
i != next_check.end();
i++) {
next_vec.push_back(num[i->first]);
check_vec.push_back(num[i->second]);
}
/* Write the actual flex parser table. */
size_t hsize = pad64(sizeof(th) + sizeof(th_version) + strlen(name) + 1);
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
th.th_magic = htonl(YYTH_REGEX_MAGIC);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
th.th_hsize = htonl(hsize);
th.th_ssize = htonl(hsize +
flex_table_size(accept.begin(), accept.end()) +
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
flex_table_size(accept2.begin(), accept2.end()) +
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
(eq.size() ?
flex_table_size(equiv_vec.begin(), equiv_vec.end()) : 0) +
flex_table_size(base_vec.begin(), base_vec.end()) +
flex_table_size(default_vec.begin(), default_vec.end()) +
flex_table_size(next_vec.begin(), next_vec.end()) +
flex_table_size(check_vec.begin(), check_vec.end()));
os.write((char *)&th, sizeof(th));
os << th_version << (char)0 << name << (char)0;
os << fill64(sizeof(th) + sizeof(th_version) + strlen(name) + 1);
write_flex_table(os, YYTD_ID_ACCEPT, accept.begin(), accept.end());
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
write_flex_table(os, YYTD_ID_ACCEPT2, accept2.begin(), accept2.end());
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
if (eq.size())
write_flex_table(os, YYTD_ID_EC, equiv_vec.begin(), equiv_vec.end());
write_flex_table(os, YYTD_ID_BASE, base_vec.begin(), base_vec.end());
write_flex_table(os, YYTD_ID_DEF, default_vec.begin(), default_vec.end());
write_flex_table(os, YYTD_ID_NXT, next_vec.begin(), next_vec.end());
write_flex_table(os, YYTD_ID_CHK, check_vec.begin(), check_vec.end());
}
Comment out some dead code
2007-03-30 15:20:57 +00:00
#if 0
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
typedef set<ImportantNode *> AcceptNodes;
map<ImportantNode *, AcceptNodes> dominance(DFA& dfa)
{
map<ImportantNode *, AcceptNodes> is_dominated;
for (States::iterator i = dfa.states.begin(); i != dfa.states.end(); i++) {
AcceptNodes set1;
for (State::iterator j = (*i)->begin(); j != (*i)->end(); j++) {
if (AcceptNode *accept = dynamic_cast<AcceptNode *>(*j))
set1.insert(accept);
}
for (AcceptNodes::iterator j = set1.begin(); j != set1.end(); j++) {
pair<map<ImportantNode *, AcceptNodes>::iterator, bool> x =
is_dominated.insert(make_pair(*j, set1));
if (!x.second) {
AcceptNodes &set2(x.first->second), set3;
for (AcceptNodes::iterator l = set2.begin();
l != set2.end();
l++) {
if (set1.find(*l) != set1.end())
set3.insert(*l);
}
set3.swap(set2);
}
}
}
return is_dominated;
}
Comment out some dead code
2007-03-30 15:20:57 +00:00
#endif
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
void dump_regexp_rec(ostream& os, Node *tree)
{
Improve dfa generation. Apply tree factoring and simplification techniques to reduce the number of states used in computing the dfa. This can have an exponential impact on both space and time for dfa generation.
2008-11-07 13:00:05 +00:00
if (tree->child[0])
dump_regexp_rec(os, tree->child[0]);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
os << *tree;
Improve dfa generation. Apply tree factoring and simplification techniques to reduce the number of states used in computing the dfa. This can have an exponential impact on both space and time for dfa generation.
2008-11-07 13:00:05 +00:00
if (tree->child[1])
dump_regexp_rec(os, tree->child[1]);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
void dump_regexp(ostream& os, Node *tree)
{
dump_regexp_rec(os, tree);
os << endl;
}
#include <sstream>
#include <ext/stdio_filebuf.h>
struct aare_ruleset {
int reverse;
Node *root;
};
Add reference counting on class Node so that we will be able to reuse AcceptNodes
2007-03-30 14:13:56 +00:00
extern "C" aare_ruleset_t *aare_new_ruleset(int reverse)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
aare_ruleset_t *container = (aare_ruleset_t *) malloc(sizeof(aare_ruleset_t));
if (!container)
return NULL;
Clean up the tree simplification code, and make the following improvements - disable charter, charset merging. This can actually hamper optimization in some cases and needs special cases added to the factoring code. The charset code is merged off into its own routines that can be reenabled at a later time. - fix a couple bugs in tree simplifications that would cause earlier exit before the tree had even reached a local minima I particular the t != c portion of the simplify_tree, would cause the loop to exit early if it didn't change but other modifications had been made. - remove the extra epsnode that was getting added to the created tree - optimize the forward factor alt loop so that it will find all left factor matches down the alt subtree without having to loop and recompare against nodes that were already checked These changes result in small improvements in most cases, but in some policies the changes result in very large wins. The early bailout of optimizations was causing 2.5* as many dfa states in one particular stress test policy.
2008-12-03 03:47:31 +00:00
container->root = NULL;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
container->reverse = reverse;
return container;
}
Add reference counting on class Node so that we will be able to reuse AcceptNodes
2007-03-30 14:13:56 +00:00
extern "C" void aare_delete_ruleset(aare_ruleset_t *rules)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
if (rules) {
Clean up the tree simplification code, and make the following improvements - disable charter, charset merging. This can actually hamper optimization in some cases and needs special cases added to the factoring code. The charset code is merged off into its own routines that can be reenabled at a later time. - fix a couple bugs in tree simplifications that would cause earlier exit before the tree had even reached a local minima I particular the t != c portion of the simplify_tree, would cause the loop to exit early if it didn't change but other modifications had been made. - remove the extra epsnode that was getting added to the created tree - optimize the forward factor alt loop so that it will find all left factor matches down the alt subtree without having to loop and recompare against nodes that were already checked These changes result in small improvements in most cases, but in some policies the changes result in very large wins. The early bailout of optimizations was causing 2.5* as many dfa states in one particular stress test policy.
2008-12-03 03:47:31 +00:00
if (rules->root)
rules->root->release();
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
free(rules);
}
}
add pix transition mode
2007-11-16 09:27:34 +00:00
static inline int diff_qualifiers(uint32_t perm1, uint32_t perm2)
{
Basis for named transitions
2008-04-16 04:44:21 +00:00
return ((perm1 & AA_EXEC_TYPE) && (perm2 & AA_EXEC_TYPE) &&
(perm1 & AA_EXEC_TYPE) != (perm2 & AA_EXEC_TYPE));
add pix transition mode
2007-11-16 09:27:34 +00:00
}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
/**
* Compute the permission flags that this state corresponds to. If we
* have any exact matches, then they override the execute and safe
* execute flags.
*/
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
uint32_t accept_perms(NodeSet *state, uint32_t *audit_ctl, int *error)
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
{
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
uint32_t perms = 0, exact_match_perms = 0, audit = 0, exact_audit = 0,
quiet = 0, deny = 0;
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (error)
*error = 0;
This patch reworks the internal structures used to compute the dfa. It is on the large side, and I experimented with different ways to split this up but in the end, anything I could do would result in a series of dependent patches that would require all of them to be applied to get meaningful functional changes. The patch structural reworks the dfa so that - there is a new State class, it takes the place of sets of nodes in the dfa, and allows storing state information within the state - removes the dfa transition table, which mapped sets of nodes to a transition table, by moving the transition into the new state class - computes dfa state permissions once (stored in the state) - expression tree nodes are independent from a created dfa. This allows computed expression trees, and sets of Nodes (used as protostates when computing the dfa). To be managed independent of the dfa life time. This will allow reducing the amount of memory used, in the future, and will also allow separating the expression tree logic out into its own file. The patch has some effect on reducing peak memory usage, and computation time. The actual amount of reduction is dependent on the number of states in the dfa with larger saving being achieved on larger dfas. Eg. for the test evince profile I was using it makes the parser about 7% faster with a peak memory usage about 12% less. This patch changes the initial partition hashing of minimization resulting in slightly smaller dfas.
2010-11-09 11:14:55 -08:00
for (NodeSet::iterator i = state->begin(); i != state->end(); i++) {
add pix transition mode
2007-11-16 09:27:34 +00:00
MatchFlag *match;
if (!(match= dynamic_cast<MatchFlag *>(*i)))
continue;
if (dynamic_cast<ExactMatchFlag *>(match)) {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
/* exact match only ever happens with x */
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
if (!is_merged_x_consistent(exact_match_perms,
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
match->flag) && error)
allow for ptrace rules
2008-04-09 09:04:08 +00:00
*error = 1;;
add pix transition mode
2007-11-16 09:27:34 +00:00
exact_match_perms |= match->flag;
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
exact_audit |= match->audit;
} else if (dynamic_cast<DenyMatchFlag *>(match)) {
deny |= match->flag;
quiet |= match->audit;
add pix transition mode
2007-11-16 09:27:34 +00:00
} else {
Dfa minimization and unreachable state removal Add basic Hopcroft based dfa minimization. It currently does a simple straight state comparison that can be quadratic in time to split partitions. This is offset however by using hashing to setup the initial partitions so that the number of states within a partition are relative few. The hashing of states for initial partition setup is linear in time. This means the closer the initial partition set is to the final set, the closer the algorithm is to completing in a linear time. The hashing works as follows: For each state we know the number of transitions that are not the default transition. For each of of these we hash the set of letters it can transition on using a simple djb2 hash algorithm. This creates a unique hash based on the number of transitions and the input it can transition on. If a state does not have the same hash we know it can not the same as another because it either has a different number of transitions or or transitions on a different set. To further distiguish states, the number of transitions of each transitions target state are added into the hash. This serves to further distiguish states as a transition to a state with a different number of transitions can not possibly be reduced to an equivalent state. A further distinction of states is made for accepting states in that we know each state with a unique set of accept permissions must be in its own partition to ensure the unique accept permissions are in the final dfa. The unreachable state removal is a basic walk of the dfa from the start state marking all states that are reached. It then sweeps any state not reached away. This does not do dead state removal where a non accepting state gets into a loop that will never result in an accepting state.
2010-01-20 03:32:34 -08:00
if (!is_merged_x_consistent(perms, match->flag) && error)
allow for ptrace rules
2008-04-09 09:04:08 +00:00
*error = 1;
add pix transition mode
2007-11-16 09:27:34 +00:00
perms |= match->flag;
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
audit |= match->audit;
add pix transition mode
2007-11-16 09:27:34 +00:00
}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
}
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
//if (audit || quiet)
//fprintf(stderr, "perms: 0x%x, audit: 0x%x exact: 0x%x eaud: 0x%x deny: 0x%x quiet: 0x%x\n", perms, audit, exact_match_perms, exact_audit, deny, quiet);
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
perms |= exact_match_perms &
change from U:G:O syntax to owner syntax and remove group permission
2007-11-29 18:06:53 +00:00
~(AA_USER_EXEC_TYPE | AA_OTHER_EXEC_TYPE);
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
if (exact_match_perms & AA_USER_EXEC_TYPE) {
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
perms = (exact_match_perms & AA_USER_EXEC_TYPE) |
(perms & ~AA_USER_EXEC_TYPE);
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
audit = (exact_audit & AA_USER_EXEC_TYPE) |
(audit & ~ AA_USER_EXEC_TYPE);
}
if (exact_match_perms & AA_OTHER_EXEC_TYPE) {
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
perms = (exact_match_perms & AA_OTHER_EXEC_TYPE) |
(perms & ~AA_OTHER_EXEC_TYPE);
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
audit = (exact_audit & AA_OTHER_EXEC_TYPE) |
(audit & ~AA_OTHER_EXEC_TYPE);
}
if (perms & AA_USER_EXEC & deny)
perms &= ~AA_USER_EXEC_TYPE;
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
if (perms & AA_OTHER_EXEC & deny)
perms &= ~AA_OTHER_EXEC_TYPE;
perms &= ~deny;
if (audit_ctl)
*audit_ctl = PACK_AUDIT_CTL(audit, quiet & deny);
// if (perms & AA_ERROR_BIT) {
// fprintf(stderr, "error bit 0x%x\n", perms);
// exit(255);
//}
extend the flags in preparation for audit control
2008-03-13 16:46:53 +00:00
//if (perms & AA_EXEC_BITS)
//fprintf(stderr, "accept perm: 0x%x\n", perms);
add pix transition mode
2007-11-16 09:27:34 +00:00
/*
if (perms & ~AA_VALID_PERMS)
yyerror(_("Internal error accumulated invalid perm 0x%llx\n"), perms);
*/
update change_hats rules to generate rules for all hats
2008-04-06 18:52:47 +00:00
//if (perms & AA_CHANGE_HAT)
// fprintf(stderr, "change_hat 0x%x\n", perms);
Update x conflict failure message Output a better failure message when a conflict of x permissions cause policy compilation to fail. We don't have enough information available to output which rules during the dfa compilation so just improve the message to let people know that it means there are conflicting x modifiers in the rules. Signed-off-by: John Johansen <john.johansen@canonical.com>
2011-02-22 03:47:03 -08:00
if (*error)
Fix compilation errors that slipped in. Yes, I realize this breaks the one translation string that was intended for regexp.y, sorry.
2011-02-23 14:40:07 -08:00
fprintf(stderr, "profile has merged rule with conflicting x modifiers\n");
Update x conflict failure message Output a better failure message when a conflict of x permissions cause policy compilation to fail. We don't have enough information available to output which rules during the dfa compilation so just improve the message to let people know that it means there are conflicting x modifiers in the rules. Signed-off-by: John Johansen <john.johansen@canonical.com>
2011-02-22 03:47:03 -08:00
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
return perms;
}
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
extern "C" int aare_add_rule(aare_ruleset_t *rules, char *rule, int deny,
This adds a basic debug dump for the conversion of each rule in a profile to its expression tree. It is limited in that it doesn't currently handle the permissions of a rule. conversion output presents an aare -> prce conversion followed by 1 or more expression tree rules, governed by what the rule does. eg. aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* eg. echo "/foo { /** rwlkmix, } " | ./apparmor_parser -QT -D rule-exprs -D expr-tree aare: /foo -> /foo aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* rule: /[^/\x00][^\x00]*\x00/[^/].* -> /[^\0000/]([^\0000])*\0000/[^/](.)* DFA: Expression Tree (/[^\0000/]([^\0000])*(((((((((((((<513>|<2>)|<4>)|<8>)|<16>)|<32>)|<64>)|<8404992>)|<32768>)|<65536>)|<131072>)|<262144>)|<524288>)|<1048576>)|/[^\0000/]([^\0000])*\0000/[^/](.)*((<16>|<32>)|<262144>)) This simple example shows many things 1. The profile name under goes pcre conversion. But since no regular expressions where found it doesn't generate any expr rules 2. /** is converted into the pcre expression /[^\0000/]([^\0000])* 3. The pcre expression /[^\0000/]([^\0000])* is converted into two rules that are then converted into expression trees. The reason for this can not be seen by the output as this is actually triggered by permissions separation for the rule. In this case the link permission is separated into what is shown as the second rule: statement. 4. DFA: Expression Tree dump shows how these rules are combined together You will notice that the rule conversion statement is fairly redundant currently as it just show pcre to expression tree pcre. This will change when direct aare parsing occurs, but currently serves to verify the pcre conversion step. It is not the prettiest patch, as its touching some ugly code that is schedule to be cleaned up/replaced. eg. convert_aaregex_to_pcre is going to replaced with native parse conversion from an aare straight to the expression tree, and dfaflag passing will become part of the rule set.
2010-07-23 13:29:35 +02:00
uint32_t perms, uint32_t audit, dfaflags_t flags)
Update the parse to emit a 0 to seperate pairs in the dfa. This was always the intended behavior and fixes a bug where the dfa will match change profile rules using // seperator.
2008-03-13 16:46:19 +00:00
{
This adds a basic debug dump for the conversion of each rule in a profile to its expression tree. It is limited in that it doesn't currently handle the permissions of a rule. conversion output presents an aare -> prce conversion followed by 1 or more expression tree rules, governed by what the rule does. eg. aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* eg. echo "/foo { /** rwlkmix, } " | ./apparmor_parser -QT -D rule-exprs -D expr-tree aare: /foo -> /foo aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* rule: /[^/\x00][^\x00]*\x00/[^/].* -> /[^\0000/]([^\0000])*\0000/[^/](.)* DFA: Expression Tree (/[^\0000/]([^\0000])*(((((((((((((<513>|<2>)|<4>)|<8>)|<16>)|<32>)|<64>)|<8404992>)|<32768>)|<65536>)|<131072>)|<262144>)|<524288>)|<1048576>)|/[^\0000/]([^\0000])*\0000/[^/](.)*((<16>|<32>)|<262144>)) This simple example shows many things 1. The profile name under goes pcre conversion. But since no regular expressions where found it doesn't generate any expr rules 2. /** is converted into the pcre expression /[^\0000/]([^\0000])* 3. The pcre expression /[^\0000/]([^\0000])* is converted into two rules that are then converted into expression trees. The reason for this can not be seen by the output as this is actually triggered by permissions separation for the rule. In this case the link permission is separated into what is shown as the second rule: statement. 4. DFA: Expression Tree dump shows how these rules are combined together You will notice that the rule conversion statement is fairly redundant currently as it just show pcre to expression tree pcre. This will change when direct aare parsing occurs, but currently serves to verify the pcre conversion step. It is not the prettiest patch, as its touching some ugly code that is schedule to be cleaned up/replaced. eg. convert_aaregex_to_pcre is going to replaced with native parse conversion from an aare straight to the expression tree, and dfaflag passing will become part of the rule set.
2010-07-23 13:29:35 +02:00
return aare_add_rule_vec(rules, deny, perms, audit, 1, &rule, flags);
Update the parse to emit a 0 to seperate pairs in the dfa. This was always the intended behavior and fixes a bug where the dfa will match change profile rules using // seperator.
2008-03-13 16:46:19 +00:00
}
add aare_reset_matchflags() to reset match flags Signed-Off-By: Kees Cook <kees.cook@canonical.com>
2009-07-24 07:33:09 +00:00
#define FLAGS_WIDTH 2
#define MATCH_FLAGS_SIZE (sizeof(uint32_t) * 8 - 1)
MatchFlag *match_flags[FLAGS_WIDTH][MATCH_FLAGS_SIZE];
DenyMatchFlag *deny_flags[FLAGS_WIDTH][MATCH_FLAGS_SIZE];
Fix two x transition conflict bugs. The is_merged_x_consistend macro was incorrect in that is tested for USER_EXEC_TYPE to determine if there was an x transition. This fails for unconfined execs so an unconfined exec would not correctly conflict with another exec type. The dfa match flag table for xtransitions was not large enough and not indexed properly for pux, and cux transitions. The index calculation did not take into account the pux flag so that pux and px aliased to the same location and cux and cx aliased to the same location. This would result in the first rule being processed defining what the transition type was for all following rules of the type following. So if a px transition was processed first all pux, transitions in the profile would be treated pux. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:52:53 -08:00
#define EXEC_MATCH_FLAGS_SIZE (AA_EXEC_COUNT *2 * 2 * 2) /* double for each of ix pux, unsafe x bits * u::o */
MatchFlag *exec_match_flags[FLAGS_WIDTH][EXEC_MATCH_FLAGS_SIZE]; /* mods + unsafe + ix + pux * u::o*/
ExactMatchFlag *exact_match_flags[FLAGS_WIDTH][EXEC_MATCH_FLAGS_SIZE];/* mods + unsafe + ix + pux *u::o*/
add aare_reset_matchflags() to reset match flags Signed-Off-By: Kees Cook <kees.cook@canonical.com>
2009-07-24 07:33:09 +00:00
extern "C" void aare_reset_matchflags(void)
{
uint32_t i, j;
#define RESET_FLAGS(group, size) { \
for (i = 0; i < FLAGS_WIDTH; i++) { \
for (j = 0; j < size; j++) { \
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
if ((group)[i][j]) delete (group)[i][j]; \
add aare_reset_matchflags() to reset match flags Signed-Off-By: Kees Cook <kees.cook@canonical.com>
2009-07-24 07:33:09 +00:00
(group)[i][j] = NULL; \
} \
} \
}
RESET_FLAGS(match_flags,MATCH_FLAGS_SIZE);
RESET_FLAGS(deny_flags,MATCH_FLAGS_SIZE);
RESET_FLAGS(exec_match_flags,EXEC_MATCH_FLAGS_SIZE);
RESET_FLAGS(exact_match_flags,EXEC_MATCH_FLAGS_SIZE);
#undef RESET_FLAGS
}
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
extern "C" int aare_add_rule_vec(aare_ruleset_t *rules, int deny,
uint32_t perms, uint32_t audit,
This adds a basic debug dump for the conversion of each rule in a profile to its expression tree. It is limited in that it doesn't currently handle the permissions of a rule. conversion output presents an aare -> prce conversion followed by 1 or more expression tree rules, governed by what the rule does. eg. aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* eg. echo "/foo { /** rwlkmix, } " | ./apparmor_parser -QT -D rule-exprs -D expr-tree aare: /foo -> /foo aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* rule: /[^/\x00][^\x00]*\x00/[^/].* -> /[^\0000/]([^\0000])*\0000/[^/](.)* DFA: Expression Tree (/[^\0000/]([^\0000])*(((((((((((((<513>|<2>)|<4>)|<8>)|<16>)|<32>)|<64>)|<8404992>)|<32768>)|<65536>)|<131072>)|<262144>)|<524288>)|<1048576>)|/[^\0000/]([^\0000])*\0000/[^/](.)*((<16>|<32>)|<262144>)) This simple example shows many things 1. The profile name under goes pcre conversion. But since no regular expressions where found it doesn't generate any expr rules 2. /** is converted into the pcre expression /[^\0000/]([^\0000])* 3. The pcre expression /[^\0000/]([^\0000])* is converted into two rules that are then converted into expression trees. The reason for this can not be seen by the output as this is actually triggered by permissions separation for the rule. In this case the link permission is separated into what is shown as the second rule: statement. 4. DFA: Expression Tree dump shows how these rules are combined together You will notice that the rule conversion statement is fairly redundant currently as it just show pcre to expression tree pcre. This will change when direct aare parsing occurs, but currently serves to verify the pcre conversion step. It is not the prettiest patch, as its touching some ugly code that is schedule to be cleaned up/replaced. eg. convert_aaregex_to_pcre is going to replaced with native parse conversion from an aare straight to the expression tree, and dfaflag passing will become part of the rule set.
2010-07-23 13:29:35 +02:00
int count, char **rulev,
dfaflags_t flags)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
Update the parse to emit a 0 to seperate pairs in the dfa. This was always the intended behavior and fixes a bug where the dfa will match change profile rules using // seperator.
2008-03-13 16:46:19 +00:00
Node *tree = NULL, *accept;
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
int exact_match;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
assert(perms != 0);
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
if (regex_parse(&tree, rulev[0]))
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
return 0;
Update the parse to emit a 0 to seperate pairs in the dfa. This was always the intended behavior and fixes a bug where the dfa will match change profile rules using // seperator.
2008-03-13 16:46:19 +00:00
for (int i = 1; i < count; i++) {
Node *subtree = NULL;
Node *node = new CharNode(0);
if (!node)
return 0;
tree = new CatNode(tree, node);
Split out parsing and expression trees from regexp.y Start of splitting regexp.y into logical components instead of the mess it is today. Split out the expr-tree and parsing components from regexp.y int expr-tree.x and parse.y and since regexp.y no longer does parsing rename it to hfa.cc Some code cleanups snuck their way into this patch and since I am to lazy to redo it, I have left them in. Signed-off-by: John Johansen <john.johansen@canonical.com> Acked-By: Steve Beattie <sbeattie@ubuntu.com>
2011-03-13 05:46:29 -07:00
if (regex_parse(&subtree, rulev[i]))
Update the parse to emit a 0 to seperate pairs in the dfa. This was always the intended behavior and fixes a bug where the dfa will match change profile rules using // seperator.
2008-03-13 16:46:19 +00:00
return 0;
tree = new CatNode(tree, subtree);
}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
/*
* Check if we have an expression with or without wildcards. This
* determines how exec modifiers are merged in accept_perms() based
* on how we split permission bitmasks here.
*/
exact_match = 1;
for (depth_first_traversal i(tree); i; i++) {
if (dynamic_cast<StarNode *>(*i) ||
dynamic_cast<PlusNode *>(*i) ||
dynamic_cast<AnyCharNode *>(*i) ||
dynamic_cast<CharSetNode *>(*i) ||
dynamic_cast<NotCharSetNode *>(*i))
exact_match = 0;
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
}
if (rules->reverse)
flip_tree(tree);
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
Fix two x transition conflict bugs. The is_merged_x_consistend macro was incorrect in that is tested for USER_EXEC_TYPE to determine if there was an x transition. This fails for unconfined execs so an unconfined exec would not correctly conflict with another exec type. The dfa match flag table for xtransitions was not large enough and not indexed properly for pux, and cux transitions. The index calculation did not take into account the pux flag so that pux and px aliased to the same location and cux and cx aliased to the same location. This would result in the first rule being processed defining what the transition type was for all following rules of the type following. So if a px transition was processed first all pux, transitions in the profile would be treated pux. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:52:53 -08:00
/* 0x7f == 4 bits x mods + 1 bit unsafe mask + 1 bit ix, + 1 pux after shift */
#define EXTRACT_X_INDEX(perm, shift) (((perm) >> (shift + 7)) & 0x7f)
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
//if (perms & ALL_AA_EXEC_TYPE && (!perms & AA_EXEC_BITS))
// fprintf(stderr, "adding X rule without MAY_EXEC: 0x%x %s\n", perms, rulev[0]);
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
extend the flags in preparation for audit control
2008-03-13 16:46:53 +00:00
//if (perms & ALL_EXEC_TYPE)
// fprintf(stderr, "adding X rule %s 0x%x\n", rulev[0], perms);
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
//if (audit)
//fprintf(stderr, "adding rule with audit bits set: 0x%x %s\n", audit, rulev[0]);
update change_hats rules to generate rules for all hats
2008-04-06 18:52:47 +00:00
//if (perms & AA_CHANGE_HAT)
// fprintf(stderr, "adding change_hat rule %s\n", rulev[0]);
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
/* the permissions set is assumed to be non-empty if any audit
* bits are specified */
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
accept = NULL;
allow for ptrace rules
2008-04-09 09:04:08 +00:00
for (unsigned int n = 0; perms && n < (sizeof(perms) * 8) ; n++) {
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
uint32_t mask = 1 << n;
if (perms & mask) {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
int ai = audit & mask ? 1 : 0;
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
perms &= ~mask;
Node *flag;
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
if (mask & ALL_AA_EXEC_TYPE)
/* these cases are covered by EXEC_BITS */
continue;
if (deny) {
if (deny_flags[ai][n]) {
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = deny_flags[ai][n];
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
} else {
//fprintf(stderr, "Adding deny ai %d mask 0x%x audit 0x%x\n", ai, mask, audit & mask);
deny_flags[ai][n] = new DenyMatchFlag(mask, audit&mask);
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = deny_flags[ai][n];
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
}
} else if (mask & AA_EXEC_BITS) {
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
uint32_t eperm = 0;
uint32_t index = 0;
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
if (mask & AA_USER_EXEC) {
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
eperm = mask | (perms & AA_USER_EXEC_TYPE);
index = EXTRACT_X_INDEX(eperm, AA_USER_SHIFT);
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
} else {
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
eperm = mask | (perms & AA_OTHER_EXEC_TYPE);
Basis for named transitions
2008-04-16 04:44:21 +00:00
index = EXTRACT_X_INDEX(eperm, AA_OTHER_SHIFT) + (AA_EXEC_COUNT << 2);
exec modes per U:G:O
2007-11-16 09:35:57 +00:00
}
extend the flags in preparation for audit control
2008-03-13 16:46:53 +00:00
//fprintf(stderr, "index %d eperm 0x%x\n", index, eperm);
add pix transition mode
2007-11-16 09:27:34 +00:00
if (exact_match) {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
if (exact_match_flags[ai][index]) {
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = exact_match_flags[ai][index];
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
} else {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
exact_match_flags[ai][index] = new ExactMatchFlag(eperm, audit&mask);
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = exact_match_flags[ai][index];
add pix transition mode
2007-11-16 09:27:34 +00:00
}
} else {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
if (exec_match_flags[ai][index]) {
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = exec_match_flags[ai][index];
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
} else {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
exec_match_flags[ai][index] = new MatchFlag(eperm, audit&mask);
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = exec_match_flags[ai][index];
add pix transition mode
2007-11-16 09:27:34 +00:00
}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
}
} else {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
if (match_flags[ai][n]) {
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = match_flags[ai][n];
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
} else {
Add Audit control to AppArmor through, the use of audit and deny key words. Deny is also used to subtract permissions from the profiles permission set. the audit key word can be prepended to any file, network, or capability rule, to force a selective audit when that rule is matched. Audit permissions accumulate just like standard permissions. eg. audit /bin/foo rw, will force an audit message when the file /bin/foo is opened for read or write. audit /etc/shadow w, /etc/shadow r, will force an audit message when /etc/shadow is opened for writing. The audit message is per permission bit so only opening the file for read access will not, force an audit message. audit can also be used in block form instead of prepending audit to every rule. audit { /bin/foo rw, /etc/shadow w, } /etc/shadow r, # don't audit r access to /etc/shadow the deny key word can be prepended to file, network and capability rules, to result in a denial of permissions when matching that rule. The deny rule specifically does 3 things - it gives AppArmor the ability to remember what has been denied so that the tools don't prompt for what has been denied in previous profiling sessions. - it subtracts globally from the allowed permissions. Deny permissions accumulate in the the deny set just as allow permissions accumulate then, the deny set is subtracted from the allow set. - it quiets known rejects. The default audit behavior of deny rules is to quiet known rejects so that audit logs are not flooded with already known rejects. To have known rejects logged prepend the audit keyword to the deny rule. Deny rules do not have a block form. eg. deny /foo/bar rw, audit deny /etc/shadow w, audit { deny owner /blah w, deny other /foo w, deny /etc/shadow w, }
2008-03-13 17:39:03 +00:00
match_flags[ai][n] = new MatchFlag(mask, audit&mask);
Reference counting of Nodes exists to shared the special accept nodes that hold permission information. We currently keep them in a table with a refcount so that they don't go away, until we delete the table. We can simulate this by getting rid of the refcount, and making dup and release virtual, and overriding it for the special accept nodes.
2010-11-09 11:28:22 -08:00
flag = match_flags[ai][n];
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
}
}
if (accept)
accept = new AltNode(accept, flag);
else
accept = flag;
}
}
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
This adds a basic debug dump for the conversion of each rule in a profile to its expression tree. It is limited in that it doesn't currently handle the permissions of a rule. conversion output presents an aare -> prce conversion followed by 1 or more expression tree rules, governed by what the rule does. eg. aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* eg. echo "/foo { /** rwlkmix, } " | ./apparmor_parser -QT -D rule-exprs -D expr-tree aare: /foo -> /foo aare: /** -> /[^/\x00][^\x00]* rule: /[^/\x00][^\x00]* -> /[^\0000/]([^\0000])* rule: /[^/\x00][^\x00]*\x00/[^/].* -> /[^\0000/]([^\0000])*\0000/[^/](.)* DFA: Expression Tree (/[^\0000/]([^\0000])*(((((((((((((<513>|<2>)|<4>)|<8>)|<16>)|<32>)|<64>)|<8404992>)|<32768>)|<65536>)|<131072>)|<262144>)|<524288>)|<1048576>)|/[^\0000/]([^\0000])*\0000/[^/](.)*((<16>|<32>)|<262144>)) This simple example shows many things 1. The profile name under goes pcre conversion. But since no regular expressions where found it doesn't generate any expr rules 2. /** is converted into the pcre expression /[^\0000/]([^\0000])* 3. The pcre expression /[^\0000/]([^\0000])* is converted into two rules that are then converted into expression trees. The reason for this can not be seen by the output as this is actually triggered by permissions separation for the rule. In this case the link permission is separated into what is shown as the second rule: statement. 4. DFA: Expression Tree dump shows how these rules are combined together You will notice that the rule conversion statement is fairly redundant currently as it just show pcre to expression tree pcre. This will change when direct aare parsing occurs, but currently serves to verify the pcre conversion step. It is not the prettiest patch, as its touching some ugly code that is schedule to be cleaned up/replaced. eg. convert_aaregex_to_pcre is going to replaced with native parse conversion from an aare straight to the expression tree, and dfaflag passing will become part of the rule set.
2010-07-23 13:29:35 +02:00
if (flags & DFA_DUMP_RULE_EXPR) {
cerr << "rule: ";
cerr << rulev[0];
for (int i = 1; i < count; i++) {
cerr << "\\x00";
cerr << rulev[i];
}
cerr << " -> ";
tree->dump(cerr);
cerr << "\n\n";
}
Clean up the tree simplification code, and make the following improvements - disable charter, charset merging. This can actually hamper optimization in some cases and needs special cases added to the factoring code. The charset code is merged off into its own routines that can be reenabled at a later time. - fix a couple bugs in tree simplifications that would cause earlier exit before the tree had even reached a local minima I particular the t != c portion of the simplify_tree, would cause the loop to exit early if it didn't change but other modifications had been made. - remove the extra epsnode that was getting added to the created tree - optimize the forward factor alt loop so that it will find all left factor matches down the alt subtree without having to loop and recompare against nodes that were already checked These changes result in small improvements in most cases, but in some policies the changes result in very large wins. The early bailout of optimizations was causing 2.5* as many dfa states in one particular stress test policy.
2008-12-03 03:47:31 +00:00
if (rules->root)
rules->root = new AltNode(rules->root, new CatNode(tree, accept));
else
rules->root = new CatNode(tree, accept);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
return 1;
user:group:any permissions
2007-11-16 09:35:31 +00:00
}
Introduce reference counting for syntax tree nodes. Use that to share AcceptNodes, and split the bits in AcceptNodes to minimize the number of states.
2007-03-30 20:38:51 +00:00
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
/* create a dfa from the ruleset
* returns: buffer contain dfa tables, @size set to the size of the tables
* else NULL on failure
*/
Add basic controls for dfa optimization
2010-01-08 04:30:56 -08:00
extern "C" void *aare_create_dfa(aare_ruleset_t *rules, size_t *size, dfaflags_t flags)
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
{
char *buffer = NULL;
label_nodes(rules->root);
Update parser to allow for multiple debug dump options via -D or --dump. This will allow turning on and off various debug dumps as needed. Multiple dump options can be specified as needed by using multiple options. eg. apparmor_parser -D variables apparmor_parser -D dfa-tree -D dfa-simple-tree The help option has also been updated to take an optional argument to display help about give parameters, currently only dump is supported. eg. apparmor_parser -h # standard help apparmor_parser -h=dump # dump info about --dump options Also Enable the dfa expression tree dumps
2010-01-07 16:21:02 -08:00
if (flags & DFA_DUMP_TREE) {
cerr << "\nDFA: Expression Tree\n";
rules->root->dump(cerr);
cerr << "\n\n";
}
The dfa flags currently are a weird mix of position and negative assertions. Its cleaner just to have them all assert one way and let the cmd line options apply them correctly.
2010-11-09 11:23:45 -08:00
if (flags & DFA_CONTROL_TREE_SIMPLE) {
Add basic controls for dfa optimization
2010-01-08 04:30:56 -08:00
rules->root = simplify_tree(rules->root, flags);
Update parser to allow for multiple debug dump options via -D or --dump. This will allow turning on and off various debug dumps as needed. Multiple dump options can be specified as needed by using multiple options. eg. apparmor_parser -D variables apparmor_parser -D dfa-tree -D dfa-simple-tree The help option has also been updated to take an optional argument to display help about give parameters, currently only dump is supported. eg. apparmor_parser -h # standard help apparmor_parser -h=dump # dump info about --dump options Also Enable the dfa expression tree dumps
2010-01-07 16:21:02 -08:00
Add basic controls for dfa optimization
2010-01-08 04:30:56 -08:00
if (flags & DFA_DUMP_SIMPLE_TREE) {
cerr << "\nDFA: Simplified Expression Tree\n";
rules->root->dump(cerr);
cerr << "\n\n";
}
Update parser to allow for multiple debug dump options via -D or --dump. This will allow turning on and off various debug dumps as needed. Multiple dump options can be specified as needed by using multiple options. eg. apparmor_parser -D variables apparmor_parser -D dfa-tree -D dfa-simple-tree The help option has also been updated to take an optional argument to display help about give parameters, currently only dump is supported. eg. apparmor_parser -h # standard help apparmor_parser -h=dump # dump info about --dump options Also Enable the dfa expression tree dumps
2010-01-07 16:21:02 -08:00
}
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
stringstream stream;
try {
DFA dfa(rules->root, flags);
if (flags & DFA_DUMP_UNIQ_PERMS)
dfa.dump_uniq_perms("dfa");
As from a library pov they should be seperately callable fns, and this will help reduce peak memory usage in some cases. Also disbale remove_unreachable, as the current dfa code isn't generating unreachable states, and minimization removes any states that are connected but redundant.
2010-11-09 11:28:56 -08:00
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
if (flags & DFA_CONTROL_MINIMIZE) {
dfa.minimize(flags);
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
if (flags & DFA_DUMP_MIN_UNIQ_PERMS)
dfa.dump_uniq_perms("minimized dfa");
}
if (flags & DFA_CONTROL_REMOVE_UNREACHABLE)
dfa.remove_unreachable(flags);
fix overlapping x modifiers perm computation
2007-12-20 12:56:50 +00:00
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
if (flags & DFA_DUMP_STATES)
dfa.dump(cerr);
Add basic dfa stats and debug dumps for equivelence classes expr tree (add stats, update parser switch) dfa transition table
2010-01-08 02:17:45 -08:00
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
if (flags & DFA_DUMP_GRAPH)
dfa.dump_dot_graph(cerr);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
map<uchar, uchar> eq;
if (flags & DFA_CONTROL_EQUIV) {
eq = dfa.equivalence_classes(flags);
dfa.apply_equivalence_classes(eq);
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
Fix error checking of conflicting x-trans during dfa construction. During some of the dfa cleanups, the checks for conflicting xtransition was removed. This adds the conflict checking back in and makes it part of dfa creation. Signed-off-by: John Johansen <john.johansen@canonical.com>
2010-12-20 11:51:20 -08:00
if (flags & DFA_DUMP_EQUIV) {
cerr << "\nDFA equivalence class\n";
dump_equivalence_classes(cerr, eq);
}
} else if (flags & DFA_DUMP_EQUIV)
cerr << "\nDFA did not generate an equivalence class\n";
TransitionTable transition_table(dfa, eq, flags);
if (flags & DFA_DUMP_TRANS_TABLE)
transition_table.dump(cerr);
transition_table.flex_table(stream, "");
} catch (int error) {
*size = 0;
return NULL;
}
Add dfa support to the parser
2007-02-27 02:29:16 +00:00
stringbuf *buf = stream.rdbuf();
buf->pubseekpos(0);
*size = buf->in_avail();
buffer = (char *)malloc(*size);
if (!buffer)
return NULL;
buf->sgetn(buffer, *size);
return buffer;
}
Reference in New Issue Copy Permalink