Question re: Filters

[oucil]

First of Konstantin, thank you for writing your own bike :)

I have a question about the "Filter" object, specifically the "position" parameter. I'm not clear what that is. The method describes the type as an int, but I'm not sure what the int is in reference to. Is it a reference to the array position of a corresponding field from the parent select statement?

Appreciate your insights!

Kevin.

[oucil]

I should add the following, because I'm starting to see where it may reference in the index's filter column list, which isn't exactly clear when you're using selectByIndex...

                // Open a connection
        $index_id = $this->_read->getIndexId(
                $this->_database,
                $query->get_table(),
                $query->get_index(),
                $query->get_fields()
        );

        // Create the select query
        $query = $this->_read->selectByIndex(
                $index_id,
                $query->get_type(),
                $query->get_key_values(),
                $query->get_limit(),
                $query->get_offset(),
                $filter_values
        );

Would I be right in assuming that I need to add the filter column list to the getIndexId() method? And that the position param used in the Filters themselves reference the associative array key from that array?

[KonstantinKuklin]

thanks for choosing my bike :)

for example:

        $reader = $this->getReader();

        $indexId = $reader->getIndexId(
            $this->getDatabase(),
            $this->getTableName(),
            'PRIMARY',
            array('key', 'text'),
            true,
            array(0 => 'num')
        );

here I opened some table and PRIMARY index with columns (key, text). The next parameter returnOnlyId ('true') need us to get only int of opened index number. And the last array is filter column (num), later we can use it when we will need to use something like where in SQL.

        $selectQuery = $reader->selectInByIndex(
            $indexId,
            array(42, 100),
            0,
            99,
            array(new Filter(Comparison::EQUAL, 0, 1))
        );

this HS query is the same like in SQL:
select key, text from table where key in (42, 100) and num = 1

I added one filter with comparison type equal(=), also position is 0 it is position of element inside array filterColumnList when you use getIndexId.
I hope, I could answer your question.

I would recommend you to try QueryBuilder, because it is more human friendly :)

same query via querybuilder:

        $selectQueryBuilder = QueryBuilder::select(array('key', 'text'))
            ->fromDataBase($this->getDatabase())
            ->fromTable($this->getTableName())
            ->whereIn('key', array(42, 100))
            ->andWhere('num', Comparison::EQUAL, 1);

another examples you can find in docs or in tests

[oucil]

Thanks for getting back so quick, appreciate you clearing that up! I gradually started to figure that out, I believe part of the problem was I had fixed composer to 0.1 rather than dev-master so I was working from an older copy and the method hints didn't match up to your examples. ;)

Love the look of the querybuilder, very logical and I'm sure a great help to those testing out handlersocket with only a mysql background. I actually wrote part of a sockets library for handlersocket a couple of years ago but it wasn't clean, and I didn't go much beyond basic the basics. The nice thing is that it was abstracted well and the way you've organized your input params falls right in line with what my own library did so your library has been like a drop in replacement for our own, instantly opening up a lot more functionality! Can't tell you how great (and rare) that is!

We're doing a lot of throughput through handlersocket, averaging about 50k reads / s, and bursting to 300k / s every 10 min on a regularly run maintenance script. My intention though is to triple that now that I'll have the ability to offload a little more thanks to the work you've done in your library. If I can glean anything important or give back to your library I will.

Cheers!
Kevin.

[KonstantinKuklin]

thx for your feedback Kevin, now I found that packagist set last stable version to 0.04 tag, but it is not a last stable version in fact. I'll double check it.
It will be great if you`ll find how to improve the library, I'am always open to discuss

[oucil]

Hey Konstantin,

I'm having no luck getting results out of the system, and I'm having a hard time reconciling any of the debug output I've come up with. Maybe you can take a quick peak, you'll probably be able to spot something pretty quickly...

This is my query object that I pass into my nosql/handlersocket adapter...

My Query Object

ECMS_Database_Query_Nosql Object
(
    [_action:ECMS_Database_Query_Nosql:private] => read
    [_index:ECMS_Database_Query_Nosql:private] => sys_domains_domain
    [_table:ECMS_Database_Query_Nosql:private] => sys_domains
    [_keymap:ECMS_Database_Query_Nosql:private] => Array
        (
            [domainName] => ecms.io
        )

    [_filters:ECMS_Database_Query_Nosql:private] => 
    [_fields:ECMS_Database_Query_Nosql:private] => Array
        (
            [0] => cookie_domain
        )

    [_limit:ECMS_Database_Query_Nosql:private] => 100
    [_offset:ECMS_Database_Query_Nosql:private] => 0
    [_type:ECMS_Database_Query_Nosql:private] => =
)

As you can see above, it contains the basics, index name, the _keymap is in the format of index_key => key_value, _fields is the return field list, etc. _filters would contain additional filter

The rest are your own HS objects...

Reader

HS\Reader Object
(
    [countQueries:HS\CommonClient:private] => 3
    [timeQueries:HS\CommonClient:private] => 0
    [stream:HS\CommonClient:private] => Stream\Stream Object
        (
            [stream:Stream\Stream:private] => Resource id #51
            [connection:Stream\Stream:private] => Stream\Connection Object
                (
                    [path:Stream\Connection:private] => ###.###.###.###
                    [protocol:Stream\Connection:private] => tcp
                    [port:Stream\Connection:private] => 9998
                )

            [dataHandler:Stream\Stream:private] => Stream\DataHandler Object
                (
                    [driver:Stream\DataHandler:private] => 
                )

            [readTimeOutSeconds:Stream\Stream:private] => 0
            [readTimeOutMicroSeconds:Stream\Stream:private] => 500000
            [method:Stream\Stream:private] => Stream\ReceiveMethod\StreamGetLineMethod Object
                (
                    [length:Stream\ReceiveMethod\StreamGetLineMethod:private] => 1024
                    [ending:Stream\ReceiveMethod\StreamGetLineMethod:private] => 

                )

        )

    [currentIndexIterator:HS\CommonClient:private] => 1
    [indexList:HS\CommonClient:private] => Array
        (
            [horizonssys_domainssys_domains_domaincookie_domain] => 1
        )

    [queryListNotSent:HS\CommonClient:private] => Array
        (
        )

    [keysList:HS\CommonClient:private] => Array
        (
            [1] => Array
                (
                    [0] => cookie_domain
                )

        )

    [authKey:HS\CommonClient:private] => 
    [debug:HS\CommonClient:private] => 
    [debugResultList] => Array
        (
        )

)

SelectQuery

HS\Query\SelectQuery Object
(
    [comparisonOparation:protected] => =
    [keyList:protected] => Array
        (
            [0] => ecms.io
        )

    [offset:protected] => 100
    [limit:protected] => 0
    [columnList:protected] => Array
        (
            [0] => cookie_domain
        )

    [suffix:protected] => 
    [inKeyList:protected] => 
    [filterList:protected] => Array
        (
        )

    [returnType:protected] => 1
    [resultObject:protected] => HS\Result\SelectResult Object
        (
            [query:protected] => HS\Query\SelectQuery Object
 *RECURSION*
            [code:protected] => 0
            [error:protected] => 
            [data:protected] => Array
                (
                )

            [time:protected] => 0
            [modifyRows:protected] => 
            [openIndexQuery:protected] => 
        )

    [socket:protected] => HS\Reader Object
        (
            [countQueries:HS\CommonClient:private] => 3
            [timeQueries:HS\CommonClient:private] => 0
            [stream:HS\CommonClient:private] => Stream\Stream Object
                (
                    [stream:Stream\Stream:private] => Resource id #51
                    [connection:Stream\Stream:private] => Stream\Connection Object
                        (
                            [path:Stream\Connection:private] => ##.##.##.##
                            [protocol:Stream\Connection:private] => tcp
                            [port:Stream\Connection:private] => 9998
                        )

                    [dataHandler:Stream\Stream:private] => Stream\DataHandler Object
                        (
                            [driver:Stream\DataHandler:private] => 
                        )

                    [readTimeOutSeconds:Stream\Stream:private] => 0
                    [readTimeOutMicroSeconds:Stream\Stream:private] => 500000
                    [method:Stream\Stream:private] => Stream\ReceiveMethod\StreamGetLineMethod Object
                        (
                            [length:Stream\ReceiveMethod\StreamGetLineMethod:private] => 1024
                            [ending:Stream\ReceiveMethod\StreamGetLineMethod:private] => 

                        )

                )

            [currentIndexIterator:HS\CommonClient:private] => 1
            [indexList:HS\CommonClient:private] => Array
                (
                    [horizonssys_domainssys_domains_domaincookie_domain] => 1
                )

            [queryListNotSent:HS\CommonClient:private] => Array
                (
                )

            [keysList:HS\CommonClient:private] => Array
                (
                    [1] => Array
                        (
                            [0] => cookie_domain
                        )

                )

            [authKey:HS\CommonClient:private] => 
            [debug:HS\CommonClient:private] => 
            [debugResultList] => Array
                (
                )

        )

    [indexId:protected] => 1
    [openIndexQuery:protected] => 
    [valueList:protected] => Array
        (
        )

    [queryClassName:protected] => HS\Query\SelectQuery
)

Here's my current setup after looking through some of your test examples. $this->read contains the Reader object, and $query is my object from up above that passes in all the details to the adapter. The methods in use format the parameters appropriately. $filter_keys and $filter_values are empty arrays for this example, which match the defaults cast in your methods.

        // Open a connection
        $index_id = $this->_read->getIndexId(
                $this->_database,
                $query->get_table(),
                $query->get_index(),
                $query->get_fields(),
                TRUE,
                $filter_keys
        );

        // Create the select query
        /* @var $read SelectQuery */
        $read = $this->_read->selectByIndex(
                $index_id,
                $query->get_type(),
                $query->get_key_values(),
                $query->get_limit(),
                $query->get_offset(),
                $filter_values
        );

        // Get the results as an array
        $read->setReturnType(SelectQuery::VECTOR);

        // Run the query and parse the results
        $data = $read->execute()->getResult()->getData();

No matter what I do $data is always an empty array.

Any light you can shed on this would really help!

Thanks,
Kevin.

[KonstantinKuklin]

did u try sent something simple to HS with current configuration without my library? HS worked ok?
database, table, index and column are exists?
how u installed hs? (here example how I did it for tests)

[oucil]

I've been using HS for more than two years in production using our own limited hand crafted library, but wanted to expand it's use, thus the move to your library. Our query object abstracts queries from the adapter so different forms of queries can be dispatched to different adapters depending on the context. For instance, a complex query with joins will automatically go to the Native MySQL adapter, whereas simple Key/Val queries will get routed through HS.

All of the queries I'm testing with work in our own HS library, but are coming up with no results through yours. I know it's a simple matter of me injecting something incorrectly which is why I provided the object dumps above. Maybe you can see something missing, or something incorrectly set?

Thanks!

[KonstantinKuklin]

very strange, do u use HS internal auth on servers?
also pls run this code

        // Open a connection
        $indexQuery = $reader->getIndexId(
            $this->_database,
            $query->get_table(),
            $query->get_index(),
            $query->get_fields(),
            false,
            $filter_keys
        );

        $openIndexQueryText = $indexQuery->getQueryString();


        // Create the select query
        /* @var $read SelectQuery */
        $read = $reader->selectByIndex(
            1,
            $query->get_type(),
            $query->get_key_values(),
            $query->get_limit(),
            $query->get_offset(),
            $filter_values
        );
        $selectQueryText = $read->getQueryString();

I`am interested in $openIndexQueryText and selectQueryText values, dump it here.
Also pls try to send the values from variables to HS server via telnel will it work?

[oucil]

Thanks for the hint on using the query string output, helped me notice my Limit and Offset were reversed, meaning I was limiting to 0 results. :)

Now I'm getting an exception out of the /HS/Result/ResultAbstract.php, though for another query, looks like it might have something to do with one of three things:

1. Something in the XML that's being retrieved,
2. potentially the length of the data, I seem to remember running across something that looked like it might limit the stream data to 1000 or 1024 bytes which would certainly cause problems with some of the results I'd be retrieving, do you know if that's the case? If so is that adjustable, or does your function pull in a loop till it gets an EOL?
3. the error output looks like the newline characters are still encoded for transfer, does your response handler decode responses?

Thanks,
Kevin.

[oucil]

Haven't made much headway on this, but I do see that the "Driver" does indeed perform encoding/decoding, so not sure where the error is coming from. Here's a sampling of the output from the error....

Uncaught exception 'HS\Errors\UnknownError' with message 'gs/>�M�J  </ectag>�M�J  <ecugm>�M�J    <tier>4</tier>�M�J    <settings/>

This is just a snipit but it shows that the error message is cutting in in the middle of my xml result data and a number of �M�J misfires where there should be \n newlines. The tabs still seem to be in place following the newlines.

Is there an opportunity to output the raw data from the response prior to decoding, and then right after decoding but prior to parsing?

[KonstantinKuklin]

1. I think, I need to give a way to change response length
2. New line character "\n" coded by HS protocol, I decode it inside the library with Driver configuration - it is not a problem.

You can't get response raw text after query with library now, but u can generate plain text from the query and bring it to HS via telnet, because UnknownError is a problem from HS side not from library.
I didn't have experience to work with XML via HS, but it must work good

[oucil]

Sounds like responses getting truncated is the likely cause then. Here's how I previously handled getting responses...

        $data = NULL;
        while ($buffer = socket_read($this->_connections['read'],2048)) {
            $data .= $buffer;
            if (strpos($buffer, "\n") !== FALSE) {
                $data = substr($data,0,-1);
                break;
            }
        }

It worked 99% of the time, but there were still some odd issues I hadn't yet sorted out. This will probably help. I'll fork the project and see if I can come up with something based on the stream library you're using (i think). To save a little time, can you point me to file/line where you're performing the actual data read?

Thanks,
K.

[KonstantinKuklin]

Stream library doesn't has any logic, just read/write
u need this

[oucil]

The culprit is actually here where you've hard coded a line length of 1024. I've followed through your process a bit to the stream library here where you call stream_get_line() using that length.

The PHP docs state the following...

Reading ends when length bytes have been read, when the string specified by ending is found (which is not included in the return value), or on EOF (whichever comes first).

... so it's hitting 1024 bytes received and quitting, truncating the response. I'm not sure you'll be able to get past this using the current function, it looks like it's better suited to retrieving things with a roughly consistent size, which is difficult to count on in all but the most basic scenarios.

In the code I posted above, the socket_read() method will run until it reaches a \n, where as in stream_read_line() it will end as soon as ANY of the supplied conditions are met.

That means that I'm going to have to change 1024 to something like 512000 (500Kb), to ensure that any eventuality will be dealt with successfully. I'm not sure whether that will have any performance hit or not, because, it will still hit a \n condition first and quit, but I'm curious to know if it has to reserve enough memory to handle the possibility of 500Kb?!?!

I'm still working on this and will let you know.

Cheers,
K.

[oucil]

Thought a potential solution may be to use a loop like I provided above but using stream_get_contents() instead. Alas, it looks like unless you know the exact length of the data, the system will wait for a specified timeout before giving up, rather than checking for EOL. Still thinking this through. Not sure stream functions which are high level compared to the low level of sockets are going to cut it.