Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
4.0
-
Tomcat6 / Catalina on Ubuntu 12.04 LTS
Description
The SignatureUpdateProcessorFactory used (primarily?) for deduplication does not consider partial update semantics.
The below uses the following solrconfig.xml excerpt:
<updateRequestProcessorChain name="text_hash"> <processor class="solr.processor.SignatureUpdateProcessorFactory"> <bool name="enabled">true</bool> <str name="signatureField">text_hash</str> <bool name="overwriteDupes">false</bool> <str name="fields">text</str> <str name="signatureClass">solr.processor.TextProfileSignature</str> </processor> <processor class="solr.LogUpdateProcessorFactory" /> <processor class="solr.RunUpdateProcessorFactory" /> </updateRequestProcessorChain>
Firstly, the processor treats
{"set": "value"}
as a string and hashes it, instead of the value alone:
$ curl '$URL/update?commit=true' -H 'Content-type:application/json' -d '{"add":{"doc":{"id": "abcde", "text": {"set": "hello world"}}}}' && curl '$URL/select?q=id:abcde' {"responseHeader":{"status":0,"QTime":30}} <?xml version="1.0" encoding="UTF-8"?><response><lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">id:abcde</str></lst></lst><result name="response" numFound="1" start="0"><doc><str name="id">abcde</str><str name="text">hello world</str><str name="text_hash">ad48c7ad60ac22cc</str><long name="_version_">1417247434224959488</long></doc></result> </response> $ $ curl '$URL/update?commit=true' -H 'Content-type:application/json' -d '{"add":{"doc":{"id": "abcde", "text": "hello world"}}}' && curl '$URL/select?q=id:abcde' {"responseHeader":{"status":0,"QTime":27}} <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">id:abcde</str></lst></lst><result name="response" numFound="1" start="0"><doc><str name="id">abcde</str><str name="text">hello world</str><str name="text_hash">b169c743d220da8d</str><long name="_version_">1417248022215000064</long></doc></result> </response>
Note the different text_hash value.
Secondly, when updating a field other than those used to create the signature (which I imagine is a more common use-case), the signature is recalculated from no values:
$ curl '$URL/update?commit=true' -H 'Content-type:application/json' -d '{"add":{"doc":{"id": "abcde", "title": {"set": "new title"}}}}' && curl '$URL/select?q=id:abcde' {"responseHeader":{"status":0,"QTime":39}} <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">id:abcde</str></lst></lst><result name="response" numFound="1" start="0"><doc><str name="id">abcde</str><str name="text">hello world</str><str name="text_hash">0000000000000000</str><str name="title">new title</str><long name="_version_">1417248120480202752</long></doc></result> </response>
Attachments
Attachments
Issue Links
- relates to
-
SOLR-3473 Distributed deduplication broken when using non-uniqueKey for signatureField
- Open