Caché Bitmap Indexes on Global
- Tutorial
Caché object DBMS supports bitmap and bitslice indexes. They are very simple to use in Caché classes: it is enough to specify the Bitmap or Bitslice attribute in the description of the index , and the performance of some SQL queries improves dramatically. But how does it work?
This article reveals how bitmap indexes are structured, how to create a bitmap index on an arbitrary global structure, how to use bit logic functions, and how to use them effectively when running NoSQL in Caché.
Since pre-object times, when developing applications for quickly searching for records of interest in globals, indexes of the form
set ^ Index ("Property", value, id) = "" were most often constructed
Such indexes allow you to easily and quickly find records for which the property of interest has a given value:
set id = "" for set id = $ order (^ Index ("Property", value, id)) quit: id = "" write id, !
will give identifiers of all records for which the value of the Property property is value, and you can check the existence of records with this property value, for example, using the following code:
if $ data (^ Index ("Property", value))> 1 {
write "id" _ $ order (^ Index ("Property", value, "")) _ "with" _value_ "exists"}
else {
write "records with" _value_ "do not exist"}
It is not difficult to come up with an algorithm for the efficient search for records that satisfy several conditions at once, united by the “I” logic. But for more complex queries, it was either necessary to develop (and debug each time) a more complex algorithm, or to abandon the use of indexes in favor of direct enumeration of all records in the database.
A revolutionary change was the appearance in Caché of full-fledged bit functions : $ bit (), $ bitlogic (), $ bitfind () and $ bitcount (). Now you can build bitmap indexes, in the simplest case - of the form
set $ bit (^ Index ("Property", value), id) = 1
Of course, since the length of the bit string is limited to 256K, in real applications the bitmap index must be placed not in one bit string, but in an array.
The $ bitlogic () functions make it possible to easily form the resulting bitmap index from such indices corresponding to a sample of records satisfying any arbitrarily interesting query, $ bitcount () determines the number of records included in the sample, and $ bitfind () provides convenient navigation through the resulting sample.
For example, collect records for which the “Pr1” property has the values “A” or “B”, and the “Pr2” property has the value “C”:
set ^ mtemp.resbitmap = $ bitlogic (^ Index ("Pr1", "A ") | ^ Index (" Pr1 "," B "))
set ^ mtemp.resbitmap = $ bitlogic (^ mtemp.resbitmap & ^ Index (" Pr2 "," C "
if '$ bitcount (^ mtemp.resbitmap) write "No records found" ,!
else set id = 0 for set id = $ bitfind (^ mtemp.resbitmap, 1, id) quit: 'id write id ,!
This is not all that can be squeezed out of bit functions in Caché. To calculate the values of the grouping function (sum, arithmetic mean, minimum, maximum), all the same, in a cycle we had to sort through all the records included in the sample, which was not fast with a sufficient sample size. But a small modification of the structure of bitmap indexes significantly speeds up this procedure as well - if the property values are integer or with a fixed decimal point.
The essence of this modification is the construction of separate bitmap indices for each bit of the property value. Such constructions are called bit-slices:
set $ bit (^ Index ("Property", N), id) = NthBitOfValue
In reality, again, an array is used instead of the bit string.
When using bitslices, it’s a little more complicated, for example, searching for restrictions “more” or “less”, but the gain in calculating grouping functions significantly outweighs this inconvenience. In fact, to calculate the sum of the values of the property of interest for the selected records, the record loop is no longer needed. It is enough to sum with the corresponding weights $ bitcount () in the resulting bitmap index for all bits the values of the property of interest. With a sufficient sample size, the performance gain can be called, without exaggeration, terrific.
The attached code contains a complete set of functions for developing a data warehouse using bitmap indexes and bitslices in classes and routines, supporting all the required manipulations with these structures. Also included is a demo method for filling test storage with data, during which the value of the grouping function — the sum — is directly calculated for the subsequent test request:
do ## class (User.BitMapSlice) .Populate (1000000); million records
and a method for executing this test request, in which the value of the same grouping function is calculated using bitslices:
do ## class (User.BitMapSlice) .TestGetData ()
You can compare both the result and the time spent to get it.
For those who are accustomed to the abbreviated COS syntax, I remind you that Caché Studio, starting with version 5.2, allows you to easily convert the selected code fragment from the full form to the short form - Ctrl-Shift-E and vice versa - Ctrl-E .
In conclusion, it is necessary to comment on one subtlety. It is recommended to build intermediate and resulting bitmap indexes for fetching in globals with ^ CacheTemp or ^ mtemp prefixes. Such globals are physically located in the CACHETEMP database and due to the fact that they are not logged , which we have no need for,working with them is more productive than even with local variables. The use of local variables for this purpose is also undesirable for another reason - the ability to run into a 16-megabyte memory limit of the user process.
This article reveals how bitmap indexes are structured, how to create a bitmap index on an arbitrary global structure, how to use bit logic functions, and how to use them effectively when running NoSQL in Caché.
Since pre-object times, when developing applications for quickly searching for records of interest in globals, indexes of the form
set ^ Index ("Property", value, id) = "" were most often constructed
Such indexes allow you to easily and quickly find records for which the property of interest has a given value:
set id = "" for set id = $ order (^ Index ("Property", value, id)) quit: id = "" write id, !
will give identifiers of all records for which the value of the Property property is value, and you can check the existence of records with this property value, for example, using the following code:
if $ data (^ Index ("Property", value))> 1 {
write "id" _ $ order (^ Index ("Property", value, "")) _ "with" _value_ "exists"}
else {
write "records with" _value_ "do not exist"}
It is not difficult to come up with an algorithm for the efficient search for records that satisfy several conditions at once, united by the “I” logic. But for more complex queries, it was either necessary to develop (and debug each time) a more complex algorithm, or to abandon the use of indexes in favor of direct enumeration of all records in the database.
A revolutionary change was the appearance in Caché of full-fledged bit functions : $ bit (), $ bitlogic (), $ bitfind () and $ bitcount (). Now you can build bitmap indexes, in the simplest case - of the form
set $ bit (^ Index ("Property", value), id) = 1
Of course, since the length of the bit string is limited to 256K, in real applications the bitmap index must be placed not in one bit string, but in an array.
The $ bitlogic () functions make it possible to easily form the resulting bitmap index from such indices corresponding to a sample of records satisfying any arbitrarily interesting query, $ bitcount () determines the number of records included in the sample, and $ bitfind () provides convenient navigation through the resulting sample.
For example, collect records for which the “Pr1” property has the values “A” or “B”, and the “Pr2” property has the value “C”:
set ^ mtemp.resbitmap = $ bitlogic (^ Index ("Pr1", "A ") | ^ Index (" Pr1 "," B "))
set ^ mtemp.resbitmap = $ bitlogic (^ mtemp.resbitmap & ^ Index (" Pr2 "," C "
if '$ bitcount (^ mtemp.resbitmap) write "No records found" ,!
else set id = 0 for set id = $ bitfind (^ mtemp.resbitmap, 1, id) quit: 'id write id ,!
This is not all that can be squeezed out of bit functions in Caché. To calculate the values of the grouping function (sum, arithmetic mean, minimum, maximum), all the same, in a cycle we had to sort through all the records included in the sample, which was not fast with a sufficient sample size. But a small modification of the structure of bitmap indexes significantly speeds up this procedure as well - if the property values are integer or with a fixed decimal point.
The essence of this modification is the construction of separate bitmap indices for each bit of the property value. Such constructions are called bit-slices:
set $ bit (^ Index ("Property", N), id) = NthBitOfValue
In reality, again, an array is used instead of the bit string.
When using bitslices, it’s a little more complicated, for example, searching for restrictions “more” or “less”, but the gain in calculating grouping functions significantly outweighs this inconvenience. In fact, to calculate the sum of the values of the property of interest for the selected records, the record loop is no longer needed. It is enough to sum with the corresponding weights $ bitcount () in the resulting bitmap index for all bits the values of the property of interest. With a sufficient sample size, the performance gain can be called, without exaggeration, terrific.
Class code
/// A set of methods for creating and working with bitmap and bit-slice indices
///
/// structure of a bitmap index
/// an array of the form array (value), the elements of which are
/// bit strings with 1 at positions corresponding to numbers records,
/// for which a certain parameter takes the value value,
/// broken into an array of strings with a maximum length of $$$ MAXBITLENGTH
/// record numbering - from 1, array elements - from 0,
/// (hereinafter, for brevity - bitmaps).
/// example of filling
/// let the record have idfact number par is val
/// set idf0 = idfact-1 \ $$$ MAXBITLENGTH, idf1 = idfact-1 # $$$ MAXBITLENGTH + 1
/// set $ bit (^ BM ("I", par, val, idf0), idf1) = 1
/// set $ bit (^ BM ("EXIST", idf0), idf1) = 1
/// the index of existing records is useful, firstly, because
/// makes it easy to perform logical deletion, and secondly,
// / makes it possible to find the number of records in the repository without
/// accessing other structures, which is necessary for the correct
/// operation of the BMNot procedure (see below)
/// structure of the bit-slice index:
/// array of bit-map indexes. The 1st element of the array is the index of signs
/// values, the 2nd is the index of the least significant bit (1/0), the 3rd is
/// the index of the penultimate bit (2/0), etc.
///
/// example of filling
/// let the record have idfact parameter number par has integer
/// value val
/// set idf0 = idfact-1 \ $$$ MAXBITLENGTH, idf1 = idfact-1 # $$$ MAXBITLENGTH + 1
/// set val = $$ CvtToBin (val)
/// for ix = 1: 1: $ length (val) set: $ extract (val, ix) $ bit (^ BM ("S", par, ix, idf0), idf1) = 1
/// when mass filling, to avoid a catastrophic drop in
/// performance, it is recommended to overlay the corresponding
/// code fragment with the functions $ sortbegin (^ BM) and $ sortend (^ BM)
/// to select facts according to given restrictions and to calculate the
sum, maximum, and minimum "
/// the functions BSSum (), BSMax () and BSMin ()
Class User are intended .BitMapSlice Extends% RegisteredObject [ProcedureBlock]
{
/// Name ($ na) of global where to store bitmaps and bitslices
/// Default value is "^ BM"
Property BMGLOB As% String (TRUNCATE = 1) [InitialExpression = "^ BM"];
/// Maximal length of $ bit string to use
Parameter MAXBITLENGTH = 64000;
/// Creates and returns the name ($ na ()) of the temporary global subnode
/// for storing intermediate bitmaps
ClassMethod GetNewTMP () As% String
{
quit $ name (^ CacheTemp ("BM", $ job_ $ zutil (110), $ increment (^ CacheTemp ("BM", $ job_ $ zutil (110)))))
}
/// Deletes all temporary subnodes created by this process
ClassMethod KillAllTMP ()
{
kill ^ CacheTemp ("BM", $ job_ $ zutil (110)) quit
}
/// Operations on bitmaps
/// hereinafter, for the sake of flexibility,
/// arrays and slices are passed by name ($ na ())
///
/// finds the position following the pos with a single bit
/// in bm - the transmitted by name bit array
Method BMOrder (bm As% String, pos As % String) As% String
{
set sub = pos \ .. # MAXBITLENGTH, ix = $ bitfind ($ get (@ bm @ (sub)), 1, pos # .. # MAXBITLENGTH + 1)
quit: ix sub *. . # MAXBITLENGTH + ix
for set sub = $ order (@ bm @ (sub)) quit: 'sub set ix = $ bitfind ($ get (@ bm @ (sub)), 1) quit: ix
quit: ix sub * .. # MAXBITLENGTH + ix
quit ""
}
/// bmdest = bmdest & bmsrc
Method BMAnd (bmdest As% String, bmsrc As% String)
{
set sub1 = $ order (@bmdest @ ($ char (0)), - 1), sub = $ order (@bmsrc @ ($ char (0)), - 1)
set: sub
for ix = 0: 1: sub set: $ data (@ bmdest @ (ix)) & $ data (@ bmsrc @ (ix)) @ bmdest @ (ix) = $ bitlogic (@ bmdest @ (ix) & @ bmsrc @ (ix)) kill: '$ data (@ bmsrc @ (ix)) @ bmdest @ (ix)
quit
}
/// bmdest = bmdest | bmsrc
Method BMOr (bmdest As% String, bmsrc As% String)
{
set sub1 = $ order (@bmdest @ ($ char (0)), - 1), sub = $ order (@bmsrc @ ($ char (0) ), - 1)
set: sub
for ix = 0: 1: sub set: $ data (@ bmsrc @ (ix)) @ bmdest @ (ix) = $ select ($ data (@ bmdest @ (ix)): $ bitlogic (@ bmdest @ (ix) | @ bmsrc @ (ix)), 1: @ bmsrc @ (ix))
quit
}
Method BMNot (bm As% String)
{
set maxblk = $ order (@ (.. BMGLOB) @ ("EXIST", "") , -1), blklen = $ bitcount (@ (.. BMGLOB) @ ("EXIST", maxblk))
for ix = maxblk: -1: 0 set blk = $ get (@ bm @ (ix)) set: $ bitcount (blk)
do ..BMAnd (bm, $ name (@ (.. BMGLOB) @ ("EXIST")))
quit
}
/// returns the number of unit bits in the
Method BMCount (bm As% String) As% Integer
{
set ix = "", bmcret = 0
for set ix = $ order (@ bm @ (ix)) quit: ix '= + ix set bmcret = bmcret + $ bitcount (@ bm @ (ix), 1)
quit bmcret
}
/// puts to the bmdest bitmap, select from the bitmap index vbmsrc,
/// where the parameter value is val
///
Method BMEq (bmdest As% String, vbmsrc As% String, val As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (val)
quit
}
/// places a selection from the bitmap index vbmsrc in the bmdest bitmap,
/// where the value of the parameter is not equal val
Method BMNe (bmdest As% String, vbmsrc As% String, val As% String)
{
do ..BMEq (bmdest, vbmsrc, val) do ..BMNot (bmdest)
quit
}
// / puts a selection from the bitmap index vbmsrc in the bmdest bitmap,
/// where the parameter value is less (sorted to) val
/// bmdest: = U (vbmsrc (v): v
Method BMLt (bmdest As% String, vbmsrc As% String, val As% String)
{
kill @bmdest set ix = val
for set ix = $ order (@ vbmsrc @ (ix), - 1) quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similar to BMLt () but "less than or equal to"
Method BMLe (bmdest As% String, vbmsrc As% String, value As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (val) set ix = val
for set ix = $ order (@ vbmsrc @ (ix), - 1) quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similar to BMLe, but "greater than or equal to"
Method BMGe (bmdest As% String, vbmsrc As% String, val As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (val) set ix = val
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, the values are greater than or equal to min and less than or equal to max
Method BMGeLe (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (min) set ix = min
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix]] max quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similar to BMGe (), but "strictly greater"
Method BMGt (bmdest As% String, vbmsrc As% String, val As% String)
{
kill @bmdest set ix = val
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix = " "do ..BMOr (bmdest,$ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, values greater than min and less than max
Method BMGtLt (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest set ix = min
for set ix = $ order (@ vbmsrc @ (ix)) quit: max ']] ix quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, the values are greater than or equal to min and less than max
Method BMGeLt (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (min) set ix = min
for set ix = $ order ( @ vbmsrc @ (ix)) quit: max ']] ix quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, values greater than min and less or equal to max
Method BMGtLe (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest set ix = min
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix]] max quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// Operations on Bit-Sliced data
/// converts an integer value to a text bit string
/// { sign, 1,2,4, ..., 2 ** N}
ClassMethod CvtToBin (value As% Integer) As% String
{
set value = $ fnumber (+ value, "", 0), res = (value <0 )
for quit: 'value set res = res_ (value # 2), value = value \ 2
quit res
}
/// converts the bit list $ lb (sign, 1,2,4, ... 2 ** N) to integer
ClassMethod CvtFromSlice (slice As% String) As% Integer
{
set res = 0
for i = $ listlength (slice): - 1: 2 set res = res + res set res = res + $ listget (slice, i, 0)
quit $ select ($ listget (slice): - res, 1: res)
}
/// puts the bits from the pos position of the vbs bitlice to the list ($ lb ())
Method GetSlice (vbs As% String, pos As% Integer) As% String
{
set sub = pos \ .. # MAXBITLENGTH, ix = pos-1 # .. # MAXBITLENGTH + 1
for i = 1: 1: $ order (@vbs @ (""), - 1) set $ list (slice, i) = $ bit ($ get (@vbs @ (i, sub)), ix)
quit slice
}
/// puts in the bmdest bit array a sample from the bit-index vbs,
/// where the parameter value is val
Method BSEq (bmdest As% String, vbs As% String, val As% Integer)
{
set bswork = .. GetNewTMP () set vbit = .. CvtToBin (val)
kill @bmdest merge @bmdest = @ (.. BMGLOB) @ ("EXIST")
set maxbit = $ order (@vbs @ (""), - 1) set: maxbit <$ length (vbit) maxbit = $ length ( vbit)
for ix = 1: 1: maxbit kill @bswork merge @ bswork = @ vbs @ (ix) do: '$ extract (vbit, ix) ..BMNot (bswork) do ..BMAnd (bmdest, bswork)
kill @ bswork quit
}
/// places in the bmdest bitmap a sample from the bit-index vbs,
/// where the parameter value is greater than or equal to val
Method BSGe (bmdest As% String, vbs As% String, val As% Integer)
{
do .. BSLt (bmdest, vbs, val), .. BMNot (bmdest) quit
}
/// puts in the bmdest bit array a sample from the bitrate index vbs,
/// where the parameter value is not equal to val
Method BSNe (bmdest As% String, vbs As% String,val As% Integer)
{
do ..BSEq (bmdest, vbs, val), .. BMNot (bmdest)
quit
}
/// puts in the bmdest bit array a sample from the bit-index vbs,
/// where the parameter value is greater than val
///
Method BSGt ( bmdest As% String, vbs As% String, val As% Integer)
{
do ..BSLe (bmdest, vbs, val), .. BMNot (bmdest) quit
}
/// places a selection from the bbs bit-index in the bmdest bit array ,
/// where the parameter sign is sign (zero is considered positive)
Method BSSign (bmdest As% String, vbs As% String, sign As% Integer)
{
set bswork = .. GetNewTMP () kill @bmdest
merge @bmdest = @ ( ..BMGLOB) @ ("EXIST"), @ bswork = @ vbs @ (1)
do: $ get (sign) '<0 ..BMNot (bswork) do ..BMAnd (bmdest, bswork)
kill @bswork quit
}
/// puts in the bmdest bit array a sample from the bit rate index vbs,
/// where the parameter is less than or equal to val
Method BSLe (bmdest As% String, vbs As% String, val As% Integer)
{
set tmpLe = .. GetNewTMP ()
do ..BSLtAbs (bmdest, vbs, val), .. BSSign (tmpLe, vbs, -1)
if val '<0 do ..BMOr (bmdest, tmpLe), .. BSEq (tmpLe, vbs, val), .. BMOr (bmdest, tmpLe) if 1
else do ..BMNot (bmdest), ..BMAnd (bmdest, tmpLe)
kill @tmpLe quit
}
/// puts in the bmdest bit array a sample from the bitrate index vbs,
/// where the parameter value is less than val
Method BSLt (bmdest As% String, vbs As% String, val As% Integer)
{
set tmpLt = .. GetNewTMP ()
do ..BSLtAbs (bmdest, vbs, val), .. BSSign (tmpLt, vbs, -1)
if val '<0 do ..BMOr (bmdest, tmpLt) if 1
ELSE do ..BMNot (bmdest), .. BMAnd (bmdest, tmpLt), .. BSNe (tmpLt, vbs, val), .. BMAnd (bmdest , tmpLt)
kill @tmpLt quit
}
/// puts into the bmdest bitmap a selection from the bitrate index vbs,
/// where the parameter value is greater than or equal to val1 and less than or equal to val2
Method BSGeLe (bmdest As% String, vbs As% String , val1 As% Integer, val2 As% Integer)
{
set tmpGeLe = .. GetNewTMP ()
do ..BSGe (bmdest, vbs, val1), .. BSLe (tmpGeLe, vbs, val2), .. BMAnd (bmdest, tmpGeLe )
kill @tmpGeLe quit
}
/// places a sample from the bit-slice index vbs into the bmdest bitmap,
/// where the parameter value is greater than or equal to val1 and less than val2
Method BSGeLt (bmdest As% String, vbs As% String, val1 As% Integer, val2 As % Integer)
{
set tmpGeLt = .. GetNewTMP ()
do ..BSGe (bmdest, vbs, val1), .. BSLt (tmpGeLt, vbs, val2), .. BMAnd (bmdest, tmpGeLt)
kill @tmpGeLt quit
}
// / puts in the bmdest bitmap a sample from the bit-index vbs,
/// where the parameter value is greater than val1 and less than or equal to val2
Method BSGtLe (bmdest As% String, vbs As% String, val1 As% Integer, val2 As% Integer)
{
set tmpGtLe = .. GetNewTMP ()
do ..BSGt (bmdest, vbs, val1), .. BSLe (tmpGtLe, vbs, val2), .. BMAnd (bmdest, tmpGtLe)
kill @tmpGtLe quit
}
/// places a selection from the vbs bit-slice index into the bmdest bit array
/// where the parameter value is greater than val1 and less than val2
Method BSGtLt (bmdest As% String, vbs As% String, val1 As% Integer , val2 As% Integer)
{
set tmpGtLt = .. GetNewTMP ()
do ..BSGt (bmdest, vbs, val1), .. BSLt (tmpGtLt, vbs, val2), .. BMAnd (bmdest, tmpGtLt)
kill @tmpGtLt quit
}
/// places in the bmdest bit array a sample from the vbs bit-slice index,
/// where the parameter value in absolute value is less than val
Method BSLtAbs (bmdest As% String, vbs As% String, val As% Integer)
{
set bswork = .. GetNewTMP (), test = .. GetNewTMP ()
kill @bmdest set vbit = .. CvtToBin (val), ixmax = $ order (@vbs @ (""), - 1)
kill @test merge @test = @ (.. BMGLOB) @ ("EXIST")
if ixmax <$ length (vbit) {
merge @ bmdest = @ test
} else {
for ix = ixmax: -1: 2 {
kill @bswork merge @ bswork = @ vbs @ (ix)
do ..BMNot (bswork), .. BMAnd (bswork, test)
do: $ extract (vbit, ix) ..BMOr (bmdest, bswork), .. BMNot (bswork)
do ..BMAnd (test, bswork)
}
}
kill @ test, @ bswork quit
}
// / grouping functions for bit-slice indices
/// returns the maximum value (with the optional parameter
/// bsmin '= 0 - the minimum) for selecting a bitmap from the vbs bit-slice index
Method BSMax (vbs As% String, bitmap As% String, bsmin As% String) As% Integer
{
set bsmin = '' $ get (bsmin), bswork = .. GetNewTMP ()
set resBSM = .. GetNewTMP (), tmpBSM = .. GetNewTMP ()
merge @ resBSM = @ vbs @ (1) do: 'bsmin ..BMNot (resBSM) do ..BMAnd (resBSM, bitmap)
if ..BMCount (resBSM) set min = 0
ELSE set min = 1 kill @resBSM merge @ resBSM = @ vbs @ (1) do: bsmin ..BMNot (resBSM) do ..BMAnd (resBSM, bitmap)
for ix = $ order (@vbs @ (""), - 1): -1: 2 {
kill @ tmpBSM, @ bswork merge @ tmpBSM = @ resBSM, @ bswork = @ vbs @ (ix)
do: min ..BMNot (bswork) do ..BMAnd (tmpBSM, bswork)
if ..BMCount ( tmpBSM) kill @resBSM merge @ resBSM = @ tmpBSM
}
set pos = .. BMOrder (resBSM, 0) quit: 'pos 0
set val = .. CvtFromSlice (.. GetSlice (vbs, pos))
kill @ bswork, @ resBSM, @ tmpBSM quit val
}
Method BSMin (vbs As% String, bitmap As% String) As% Integer
{
quit ..BSMax ( vbs, bitmap, 1)
}
/// returns the sum of the values for a bitmap sample from the bit rate index vbs
Method BSSum (vbs As% String, bitmap As% String) As% Integer
{
set bswork = .. GetNewTMP (), resBSSum =. .GetNewTMP (), tmpBSSum = .. GetNewTMP ()
merge @ resBSSum = @ vbs @ (1) do ..BMNot (resBSSum), .. BMAnd (resBSSum, bitmap) set slice = ""
for ix = 2: 1: $ order (@vbs @ (""), - 1) kill @tmpBSSum merge @ tmpBSSum = @ vbs @ (ix) do ..BMAnd (tmpBSSum, resBSSum) set $ list (slice, ix) = .. BMCount (tmpBSSum )
set val = .. CvtFromSlice (slice)
kill @resBSSum merge @ resBSSum = @ vbs @ (1) do ..BMAnd (resBSSum, bitmap) set slice = ""
for ix = 2: 1: $ order (@vbs @ (""), - 1) kill @ tmpBSSum merge @ tmpBSSum = @ vbs @ (ix) do ..BMAnd (tmpBSSum, resBSSum) set $ list (slice, ix) = .. BMCount (tmpBSSum)
set val = val - .. CvtFromSlice (slice)
kill @bswork, @ resBSSum, @ tmpBSSum quit val
}
/// methods for filling the bitmap and bit-slice indices
///
/// sets the corresponding bit in the bitmap index
/// of the given value of this property
/// with the optional parameter setexist = 1 also sets bit in the
/// bitmap index of existing records (necessary for the correct
/// operation of the BMNot () method)
Method SetBitMap (idfact As% Integer, property As% String, value As% String, setexist As% String)
{
set idf0 = idfact-1 \ .. # MAXBITLENGTH, idf1 = idfact-1 # .. # MAXBITLENGTH + 1
set: $ get (setexist) $ bit (@ (.. BMGLOB) @ ("EXIST", idf0), idf1) = 1
set $ bit (@ (.. BMGLOB) @ ("I", property, value, idf0), idf1) = 1
quit
}
/// sets the corresponding bit in the bit-slice index
/// of the given value of this property.
/// Optional parameter setexist - similar to SetBitMap () method SetBitSlice (
idfact As% Integer, property As% String, value As% Integer, setexist As% String)
{
set idf0 = idfact-1 \ .. # MAXBITLENGTH, idf1 = idfact-1 # .. # MAXBITLENGTH + 1
set: $ get (setexist) $ bit (@ (.. BMGLOB) @ ("EXIST", idf0), idf1) = 1
set v = .. CvtToBin (+ value)
for ix = 1: 1: $ length (v ) set: $ extract (v, ix) $ bit (@ (.. BMGLOB) @ ("S", property, ix, idf0), idf1) = 1
quit
}
/// returns the name of the subnode of the index global containing
// / bitmap index (if slice = 0 or not defined) or bitslice (ghb slice '= 0)
/// for this property. If property is empty or undefined -
/// then the bitmap index of existing records
Method GetBitMapName (property As% String, slice As% String) As% String
{
quit: $ get (property) = "" $ name (@ (.. BMGLOB) @ ("EXIST"))
quit $ name (@ (.. BMGLOB) @ ($ select (+ $ get (slice): "S", 1: "I"), property))
}
/// demo filling and fetching
///
ClassMethod Populate (count As% String = 10000)
{
set ix = ..% New ()
set ix.BMGLOB = $ name (^ BMI)
set count = $ get (count, 10000 ), m = 0
set names = $ listbuild ("SantaClause", "Crocodile", "Simba")
set colors = $ listbuild ("Cyan", "Magenta", "Yellow", "Black")
if $ sortbegin (^ BMI)
for idfact = 1: 1: count {
set name = $ list (names, 1 + $ random ($ listlength (names)))
set color = $ list (colors, 1 + $ random ($ listlength (colors)) )
set length = 10 + $ random (90)
set weight = (10 + $ random (40)) * 100
; for color, create a bitmap index
do ix.SetBitMap (idfact, "
; for weight and length we create bit-slice indices
do ix.SetBitSlice (idfact, "L", length)
do ix.SetBitSlice (idfact, "W", weight)
; to test the test sample (see below), we consider
; total weight of black and yellow crocodiles 45-70 cm long
; inclusive
set: ((color = "Black")! (color = "Yellow")) & (length '<45) & (length'> 70) m = m + weight
}
if $ sortend (^ BMI)
write m_ " gramms total ",!
kill ix quit
}
/// test sample
ClassMethod TestGetData ()
{
set ix = ..% New ()
set ix.BMGLOB = $ name (^ BMI)
set b = .. GetNewTMP (), b1 = .. GetNewTMP ()
do ix.BMEq (b, ix.
do ix.BMEq (b1, ix.GetBitMapName ("C"), "Yellow")
do ix.BMOr (b, b1)
do ix.BSGeLe (b1, ix.GetBitMapName ("L", 1), 45.70 )
do ix.BMAnd (b, b1)
set count = ix.BMCount (b)
write count_ "items selected" ,!
; determine the total weight of the selected
write ix.BSSum (ix.GetBitMapName ("W", 1), b) _ "gramms total" ,!
do ..KillAllTMP ()
kill ix quit
}
}
///
/// structure of a bitmap index
/// an array of the form array (value), the elements of which are
/// bit strings with 1 at positions corresponding to numbers records,
/// for which a certain parameter takes the value value,
/// broken into an array of strings with a maximum length of $$$ MAXBITLENGTH
/// record numbering - from 1, array elements - from 0,
/// (hereinafter, for brevity - bitmaps).
/// example of filling
/// let the record have idfact number par is val
/// set idf0 = idfact-1 \ $$$ MAXBITLENGTH, idf1 = idfact-1 # $$$ MAXBITLENGTH + 1
/// set $ bit (^ BM ("I", par, val, idf0), idf1) = 1
/// set $ bit (^ BM ("EXIST", idf0), idf1) = 1
/// the index of existing records is useful, firstly, because
/// makes it easy to perform logical deletion, and secondly,
// / makes it possible to find the number of records in the repository without
/// accessing other structures, which is necessary for the correct
/// operation of the BMNot procedure (see below)
/// structure of the bit-slice index:
/// array of bit-map indexes. The 1st element of the array is the index of signs
/// values, the 2nd is the index of the least significant bit (1/0), the 3rd is
/// the index of the penultimate bit (2/0), etc.
///
/// example of filling
/// let the record have idfact parameter number par has integer
/// value val
/// set idf0 = idfact-1 \ $$$ MAXBITLENGTH, idf1 = idfact-1 # $$$ MAXBITLENGTH + 1
/// set val = $$ CvtToBin (val)
/// for ix = 1: 1: $ length (val) set: $ extract (val, ix) $ bit (^ BM ("S", par, ix, idf0), idf1) = 1
/// when mass filling, to avoid a catastrophic drop in
/// performance, it is recommended to overlay the corresponding
/// code fragment with the functions $ sortbegin (^ BM) and $ sortend (^ BM)
/// to select facts according to given restrictions and to calculate the
sum, maximum, and minimum "
/// the functions BSSum (), BSMax () and BSMin ()
Class User are intended .BitMapSlice Extends% RegisteredObject [ProcedureBlock]
{
/// Name ($ na) of global where to store bitmaps and bitslices
/// Default value is "^ BM"
Property BMGLOB As% String (TRUNCATE = 1) [InitialExpression = "^ BM"];
/// Maximal length of $ bit string to use
Parameter MAXBITLENGTH = 64000;
/// Creates and returns the name ($ na ()) of the temporary global subnode
/// for storing intermediate bitmaps
ClassMethod GetNewTMP () As% String
{
quit $ name (^ CacheTemp ("BM", $ job_ $ zutil (110), $ increment (^ CacheTemp ("BM", $ job_ $ zutil (110)))))
}
/// Deletes all temporary subnodes created by this process
ClassMethod KillAllTMP ()
{
kill ^ CacheTemp ("BM", $ job_ $ zutil (110)) quit
}
/// Operations on bitmaps
/// hereinafter, for the sake of flexibility,
/// arrays and slices are passed by name ($ na ())
///
/// finds the position following the pos with a single bit
/// in bm - the transmitted by name bit array
Method BMOrder (bm As% String, pos As % String) As% String
{
set sub = pos \ .. # MAXBITLENGTH, ix = $ bitfind ($ get (@ bm @ (sub)), 1, pos # .. # MAXBITLENGTH + 1)
quit: ix sub *. . # MAXBITLENGTH + ix
for set sub = $ order (@ bm @ (sub)) quit: 'sub set ix = $ bitfind ($ get (@ bm @ (sub)), 1) quit: ix
quit: ix sub * .. # MAXBITLENGTH + ix
quit ""
}
/// bmdest = bmdest & bmsrc
Method BMAnd (bmdest As% String, bmsrc As% String)
{
set sub1 = $ order (@bmdest @ ($ char (0)), - 1), sub = $ order (@bmsrc @ ($ char (0)), - 1)
set: sub
quit
}
/// bmdest = bmdest | bmsrc
Method BMOr (bmdest As% String, bmsrc As% String)
{
set sub1 = $ order (@bmdest @ ($ char (0)), - 1), sub = $ order (@bmsrc @ ($ char (0) ), - 1)
set: sub
quit
}
Method BMNot (bm As% String)
{
set maxblk = $ order (@ (.. BMGLOB) @ ("EXIST", "") , -1), blklen = $ bitcount (@ (.. BMGLOB) @ ("EXIST", maxblk))
for ix = maxblk: -1: 0 set blk = $ get (@ bm @ (ix)) set: $ bitcount (blk)
quit
}
/// returns the number of unit bits in the
Method BMCount (bm As% String) As% Integer
{
set ix = "", bmcret = 0
for set ix = $ order (@ bm @ (ix)) quit: ix '= + ix set bmcret = bmcret + $ bitcount (@ bm @ (ix), 1)
quit bmcret
}
/// puts to the bmdest bitmap, select from the bitmap index vbmsrc,
/// where the parameter value is val
///
Method BMEq (bmdest As% String, vbmsrc As% String, val As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (val)
quit
}
/// places a selection from the bitmap index vbmsrc in the bmdest bitmap,
/// where the value of the parameter is not equal val
Method BMNe (bmdest As% String, vbmsrc As% String, val As% String)
{
do ..BMEq (bmdest, vbmsrc, val) do ..BMNot (bmdest)
quit
}
// / puts a selection from the bitmap index vbmsrc in the bmdest bitmap,
/// where the parameter value is less (sorted to) val
/// bmdest: = U (vbmsrc (v): v
{
kill @bmdest set ix = val
for set ix = $ order (@ vbmsrc @ (ix), - 1) quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similar to BMLt () but "less than or equal to"
Method BMLe (bmdest As% String, vbmsrc As% String, value As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (val) set ix = val
for set ix = $ order (@ vbmsrc @ (ix), - 1) quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similar to BMLe, but "greater than or equal to"
Method BMGe (bmdest As% String, vbmsrc As% String, val As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (val) set ix = val
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, the values are greater than or equal to min and less than or equal to max
Method BMGeLe (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (min) set ix = min
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix]] max quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similar to BMGe (), but "strictly greater"
Method BMGt (bmdest As% String, vbmsrc As% String, val As% String)
{
kill @bmdest set ix = val
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix = " "do ..BMOr (bmdest,$ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, values greater than min and less than max
Method BMGtLt (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest set ix = min
for set ix = $ order (@ vbmsrc @ (ix)) quit: max ']] ix quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, the values are greater than or equal to min and less than max
Method BMGeLt (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest merge @ bmdest = @ vbmsrc @ (min) set ix = min
for set ix = $ order ( @ vbmsrc @ (ix)) quit: max ']] ix quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// similarly, values greater than min and less or equal to max
Method BMGtLe (bmdest As% String, vbmsrc As% String, min As% String, max As% String)
{
kill @bmdest set ix = min
for set ix = $ order (@ vbmsrc @ (ix)) quit: ix]] max quit: ix = "" do ..BMOr (bmdest, $ name (@ vbmsrc @ (ix)))
quit
}
/// Operations on Bit-Sliced data
/// converts an integer value to a text bit string
/// { sign, 1,2,4, ..., 2 ** N}
ClassMethod CvtToBin (value As% Integer) As% String
{
set value = $ fnumber (+ value, "", 0), res = (value <0 )
for quit: 'value set res = res_ (value # 2), value = value \ 2
quit res
}
/// converts the bit list $ lb (sign, 1,2,4, ... 2 ** N) to integer
ClassMethod CvtFromSlice (slice As% String) As% Integer
{
set res = 0
for i = $ listlength (slice): - 1: 2 set res = res + res set res = res + $ listget (slice, i, 0)
quit $ select ($ listget (slice): - res, 1: res)
}
/// puts the bits from the pos position of the vbs bitlice to the list ($ lb ())
Method GetSlice (vbs As% String, pos As% Integer) As% String
{
set sub = pos \ .. # MAXBITLENGTH, ix = pos-1 # .. # MAXBITLENGTH + 1
for i = 1: 1: $ order (@vbs @ (""), - 1) set $ list (slice, i) = $ bit ($ get (@vbs @ (i, sub)), ix)
quit slice
}
/// puts in the bmdest bit array a sample from the bit-index vbs,
/// where the parameter value is val
Method BSEq (bmdest As% String, vbs As% String, val As% Integer)
{
set bswork = .. GetNewTMP () set vbit = .. CvtToBin (val)
kill @bmdest merge @bmdest = @ (.. BMGLOB) @ ("EXIST")
set maxbit = $ order (@vbs @ (""), - 1) set: maxbit <$ length (vbit) maxbit = $ length ( vbit)
for ix = 1: 1: maxbit kill @bswork merge @ bswork = @ vbs @ (ix) do: '$ extract (vbit, ix) ..BMNot (bswork) do ..BMAnd (bmdest, bswork)
kill @ bswork quit
}
/// places in the bmdest bitmap a sample from the bit-index vbs,
/// where the parameter value is greater than or equal to val
Method BSGe (bmdest As% String, vbs As% String, val As% Integer)
{
do .. BSLt (bmdest, vbs, val), .. BMNot (bmdest) quit
}
/// puts in the bmdest bit array a sample from the bitrate index vbs,
/// where the parameter value is not equal to val
Method BSNe (bmdest As% String, vbs As% String,val As% Integer)
{
do ..BSEq (bmdest, vbs, val), .. BMNot (bmdest)
quit
}
/// puts in the bmdest bit array a sample from the bit-index vbs,
/// where the parameter value is greater than val
///
Method BSGt ( bmdest As% String, vbs As% String, val As% Integer)
{
do ..BSLe (bmdest, vbs, val), .. BMNot (bmdest) quit
}
/// places a selection from the bbs bit-index in the bmdest bit array ,
/// where the parameter sign is sign (zero is considered positive)
Method BSSign (bmdest As% String, vbs As% String, sign As% Integer)
{
set bswork = .. GetNewTMP () kill @bmdest
merge @bmdest = @ ( ..BMGLOB) @ ("EXIST"), @ bswork = @ vbs @ (1)
do: $ get (sign) '<0 ..BMNot (bswork) do ..BMAnd (bmdest, bswork)
kill @bswork quit
}
/// puts in the bmdest bit array a sample from the bit rate index vbs,
/// where the parameter is less than or equal to val
Method BSLe (bmdest As% String, vbs As% String, val As% Integer)
{
set tmpLe = .. GetNewTMP ()
do ..BSLtAbs (bmdest, vbs, val), .. BSSign (tmpLe, vbs, -1)
if val '<0 do ..BMOr (bmdest, tmpLe), .. BSEq (tmpLe, vbs, val), .. BMOr (bmdest, tmpLe) if 1
else do ..BMNot (bmdest), ..BMAnd (bmdest, tmpLe)
kill @tmpLe quit
}
/// puts in the bmdest bit array a sample from the bitrate index vbs,
/// where the parameter value is less than val
Method BSLt (bmdest As% String, vbs As% String, val As% Integer)
{
set tmpLt = .. GetNewTMP ()
do ..BSLtAbs (bmdest, vbs, val), .. BSSign (tmpLt, vbs, -1)
if val '<0 do ..BMOr (bmdest, tmpLt) if 1
ELSE do ..BMNot (bmdest), .. BMAnd (bmdest, tmpLt), .. BSNe (tmpLt, vbs, val), .. BMAnd (bmdest , tmpLt)
kill @tmpLt quit
}
/// puts into the bmdest bitmap a selection from the bitrate index vbs,
/// where the parameter value is greater than or equal to val1 and less than or equal to val2
Method BSGeLe (bmdest As% String, vbs As% String , val1 As% Integer, val2 As% Integer)
{
set tmpGeLe = .. GetNewTMP ()
do ..BSGe (bmdest, vbs, val1), .. BSLe (tmpGeLe, vbs, val2), .. BMAnd (bmdest, tmpGeLe )
kill @tmpGeLe quit
}
/// places a sample from the bit-slice index vbs into the bmdest bitmap,
/// where the parameter value is greater than or equal to val1 and less than val2
Method BSGeLt (bmdest As% String, vbs As% String, val1 As% Integer, val2 As % Integer)
{
set tmpGeLt = .. GetNewTMP ()
do ..BSGe (bmdest, vbs, val1), .. BSLt (tmpGeLt, vbs, val2), .. BMAnd (bmdest, tmpGeLt)
kill @tmpGeLt quit
}
// / puts in the bmdest bitmap a sample from the bit-index vbs,
/// where the parameter value is greater than val1 and less than or equal to val2
Method BSGtLe (bmdest As% String, vbs As% String, val1 As% Integer, val2 As% Integer)
{
set tmpGtLe = .. GetNewTMP ()
do ..BSGt (bmdest, vbs, val1), .. BSLe (tmpGtLe, vbs, val2), .. BMAnd (bmdest, tmpGtLe)
kill @tmpGtLe quit
}
/// places a selection from the vbs bit-slice index into the bmdest bit array
/// where the parameter value is greater than val1 and less than val2
Method BSGtLt (bmdest As% String, vbs As% String, val1 As% Integer , val2 As% Integer)
{
set tmpGtLt = .. GetNewTMP ()
do ..BSGt (bmdest, vbs, val1), .. BSLt (tmpGtLt, vbs, val2), .. BMAnd (bmdest, tmpGtLt)
kill @tmpGtLt quit
}
/// places in the bmdest bit array a sample from the vbs bit-slice index,
/// where the parameter value in absolute value is less than val
Method BSLtAbs (bmdest As% String, vbs As% String, val As% Integer)
{
set bswork = .. GetNewTMP (), test = .. GetNewTMP ()
kill @bmdest set vbit = .. CvtToBin (val), ixmax = $ order (@vbs @ (""), - 1)
kill @test merge @test = @ (.. BMGLOB) @ ("EXIST")
if ixmax <$ length (vbit) {
merge @ bmdest = @ test
} else {
for ix = ixmax: -1: 2 {
kill @bswork merge @ bswork = @ vbs @ (ix)
do ..BMNot (bswork), .. BMAnd (bswork, test)
do: $ extract (vbit, ix) ..BMOr (bmdest, bswork), .. BMNot (bswork)
do ..BMAnd (test, bswork)
}
}
kill @ test, @ bswork quit
}
// / grouping functions for bit-slice indices
/// returns the maximum value (with the optional parameter
/// bsmin '= 0 - the minimum) for selecting a bitmap from the vbs bit-slice index
Method BSMax (vbs As% String, bitmap As% String, bsmin As% String) As% Integer
{
set bsmin = '' $ get (bsmin), bswork = .. GetNewTMP ()
set resBSM = .. GetNewTMP (), tmpBSM = .. GetNewTMP ()
merge @ resBSM = @ vbs @ (1) do: 'bsmin ..BMNot (resBSM) do ..BMAnd (resBSM, bitmap)
if ..BMCount (resBSM) set min = 0
ELSE set min = 1 kill @resBSM merge @ resBSM = @ vbs @ (1) do: bsmin ..BMNot (resBSM) do ..BMAnd (resBSM, bitmap)
for ix = $ order (@vbs @ (""), - 1): -1: 2 {
kill @ tmpBSM, @ bswork merge @ tmpBSM = @ resBSM, @ bswork = @ vbs @ (ix)
do: min ..BMNot (bswork) do ..BMAnd (tmpBSM, bswork)
if ..BMCount ( tmpBSM) kill @resBSM merge @ resBSM = @ tmpBSM
}
set pos = .. BMOrder (resBSM, 0) quit: 'pos 0
set val = .. CvtFromSlice (.. GetSlice (vbs, pos))
kill @ bswork, @ resBSM, @ tmpBSM quit val
}
Method BSMin (vbs As% String, bitmap As% String) As% Integer
{
quit ..BSMax ( vbs, bitmap, 1)
}
/// returns the sum of the values for a bitmap sample from the bit rate index vbs
Method BSSum (vbs As% String, bitmap As% String) As% Integer
{
set bswork = .. GetNewTMP (), resBSSum =. .GetNewTMP (), tmpBSSum = .. GetNewTMP ()
merge @ resBSSum = @ vbs @ (1) do ..BMNot (resBSSum), .. BMAnd (resBSSum, bitmap) set slice = ""
for ix = 2: 1: $ order (@vbs @ (""), - 1) kill @tmpBSSum merge @ tmpBSSum = @ vbs @ (ix) do ..BMAnd (tmpBSSum, resBSSum) set $ list (slice, ix) = .. BMCount (tmpBSSum )
set val = .. CvtFromSlice (slice)
kill @resBSSum merge @ resBSSum = @ vbs @ (1) do ..BMAnd (resBSSum, bitmap) set slice = ""
for ix = 2: 1: $ order (@vbs @ (""), - 1) kill @ tmpBSSum merge @ tmpBSSum = @ vbs @ (ix) do ..BMAnd (tmpBSSum, resBSSum) set $ list (slice, ix) = .. BMCount (tmpBSSum)
set val = val - .. CvtFromSlice (slice)
kill @bswork, @ resBSSum, @ tmpBSSum quit val
}
/// methods for filling the bitmap and bit-slice indices
///
/// sets the corresponding bit in the bitmap index
/// of the given value of this property
/// with the optional parameter setexist = 1 also sets bit in the
/// bitmap index of existing records (necessary for the correct
/// operation of the BMNot () method)
Method SetBitMap (idfact As% Integer, property As% String, value As% String, setexist As% String)
{
set idf0 = idfact-1 \ .. # MAXBITLENGTH, idf1 = idfact-1 # .. # MAXBITLENGTH + 1
set: $ get (setexist) $ bit (@ (.. BMGLOB) @ ("EXIST", idf0), idf1) = 1
set $ bit (@ (.. BMGLOB) @ ("I", property, value, idf0), idf1) = 1
quit
}
/// sets the corresponding bit in the bit-slice index
/// of the given value of this property.
/// Optional parameter setexist - similar to SetBitMap () method SetBitSlice (
idfact As% Integer, property As% String, value As% Integer, setexist As% String)
{
set idf0 = idfact-1 \ .. # MAXBITLENGTH, idf1 = idfact-1 # .. # MAXBITLENGTH + 1
set: $ get (setexist) $ bit (@ (.. BMGLOB) @ ("EXIST", idf0), idf1) = 1
set v = .. CvtToBin (+ value)
for ix = 1: 1: $ length (v ) set: $ extract (v, ix) $ bit (@ (.. BMGLOB) @ ("S", property, ix, idf0), idf1) = 1
quit
}
/// returns the name of the subnode of the index global containing
// / bitmap index (if slice = 0 or not defined) or bitslice (ghb slice '= 0)
/// for this property. If property is empty or undefined -
/// then the bitmap index of existing records
Method GetBitMapName (property As% String, slice As% String) As% String
{
quit: $ get (property) = "" $ name (@ (.. BMGLOB) @ ("EXIST"))
quit $ name (@ (.. BMGLOB) @ ($ select (+ $ get (slice): "S", 1: "I"), property))
}
/// demo filling and fetching
///
ClassMethod Populate (count As% String = 10000)
{
set ix = ..% New ()
set ix.BMGLOB = $ name (^ BMI)
set count = $ get (count, 10000 ), m = 0
set names = $ listbuild ("SantaClause", "Crocodile", "Simba")
set colors = $ listbuild ("Cyan", "Magenta", "Yellow", "Black")
if $ sortbegin (^ BMI)
for idfact = 1: 1: count {
set name = $ list (names, 1 + $ random ($ listlength (names)))
set color = $ list (colors, 1 + $ random ($ listlength (colors)) )
set length = 10 + $ random (90)
set weight = (10 + $ random (40)) * 100
; for color, create a bitmap index
do ix.SetBitMap (idfact, "
; for weight and length we create bit-slice indices
do ix.SetBitSlice (idfact, "L", length)
do ix.SetBitSlice (idfact, "W", weight)
; to test the test sample (see below), we consider
; total weight of black and yellow crocodiles 45-70 cm long
; inclusive
set: ((color = "Black")! (color = "Yellow")) & (length '<45) & (length'> 70) m = m + weight
}
if $ sortend (^ BMI)
write m_ " gramms total ",!
kill ix quit
}
/// test sample
ClassMethod TestGetData ()
{
set ix = ..% New ()
set ix.BMGLOB = $ name (^ BMI)
set b = .. GetNewTMP (), b1 = .. GetNewTMP ()
do ix.BMEq (b, ix.
do ix.BMEq (b1, ix.GetBitMapName ("C"), "Yellow")
do ix.BMOr (b, b1)
do ix.BSGeLe (b1, ix.GetBitMapName ("L", 1), 45.70 )
do ix.BMAnd (b, b1)
set count = ix.BMCount (b)
write count_ "items selected" ,!
; determine the total weight of the selected
write ix.BSSum (ix.GetBitMapName ("W", 1), b) _ "gramms total" ,!
do ..KillAllTMP ()
kill ix quit
}
}
The attached code contains a complete set of functions for developing a data warehouse using bitmap indexes and bitslices in classes and routines, supporting all the required manipulations with these structures. Also included is a demo method for filling test storage with data, during which the value of the grouping function — the sum — is directly calculated for the subsequent test request:
do ## class (User.BitMapSlice) .Populate (1000000); million records
and a method for executing this test request, in which the value of the same grouping function is calculated using bitslices:
do ## class (User.BitMapSlice) .TestGetData ()
You can compare both the result and the time spent to get it.
For those who are accustomed to the abbreviated COS syntax, I remind you that Caché Studio, starting with version 5.2, allows you to easily convert the selected code fragment from the full form to the short form - Ctrl-Shift-E and vice versa - Ctrl-E .
In conclusion, it is necessary to comment on one subtlety. It is recommended to build intermediate and resulting bitmap indexes for fetching in globals with ^ CacheTemp or ^ mtemp prefixes. Such globals are physically located in the CACHETEMP database and due to the fact that they are not logged , which we have no need for,working with them is more productive than even with local variables. The use of local variables for this purpose is also undesirable for another reason - the ability to run into a 16-megabyte memory limit of the user process.
The emphasis in italics does not apply to newer versions of the Caché DBMS:The code described in this article owes its appearance to information kindly provided by InterSystems at one of the presentations of the DeepSee product, and was subsequently used by MacovaSoft in its Ortofact data warehouse.
